I hope, though I doubt, that Nate Silver’s performance during the stretch drive of the Massachusetts special Senate election will finally lead to him being downgraded from “All-seeing HAL-9000-esque quantitative wizard” to “Just another guy with a computer”. Armed only with the traditional maxims of psephological interpretation, which teach that a late polling break away from the incumbent party is a very unfavourable omen, one could have figured out ten days ago that repulsive Democratic candidate Martha Coakley was in a heap of trouble. Silver, with his revolutionary disregard for everything but the polling numbers, was still arguing as late as Thursday afternoon that Coakley was the clear favourite; he changed his mind at midnight that evening and acknowledged that Scott Brown had a puncher’s chance.
He continued to soothe jumpy Democrats throughout the weekend in the manner of a government radio station denying rumours of a coup d’etat, writing on Sunday night that the election was still a “toss-up”. (He had already cheated on his mechanical bride by citing traditional eyeball analysis from NBC’s Chuck Todd: “If this were any other state we’d say this one was over”). There followed another handful of balm. Tonight, as the national party prepares for the probably loss of its congressional supermajority, he’s still distributing it.
Silver (or “Nate P. Silver”, as I will always think of him) may still be “right”, in the limited sense that a probabilistic prediction about an event that will only happen once can be “right” at all. But even if Coakley does surprise everyone by pulling this election out, the gruesome lack of robustness in Silver’s approach should be evident. We don’t need an advanced proprietary model to tell us what the polls are saying with a five- or six-day lag time built in. I have always understood Silver’s core claim of special expertise to inhere in the ability to give useful information about the future. Boasts like “[the model] correctly predicted the outcome of all 35 Senate races in 2008” are nothing but distracting hype, since 95%+ of Senate elections are easy to call on the morning before they happen. And as Silver certainly knows, a model that delivers probabilistic estimates of outcomes of one-time events has to not only be “right”, but right in such a way that bettors using it would, in the long run, outperform other bettors or prediction markets not equipped with that model. That’s the only appropriate test, and I know of no evidence that he has passed it.