Mr. Overrated

by Colby Cosh

I hope, though I doubt, that Nate Silver’s performance during the stretch drive of the Massachusetts special Senate election will finally lead to him being downgraded from “All-seeing HAL-9000-esque quantitative wizard” to “Just another guy with a computer”. Armed only with the traditional maxims of psephological interpretation, which teach that a late polling break away from the incumbent party is a very unfavourable omen, one could have figured out ten days ago that repulsive Democratic candidate Martha Coakley was in a heap of trouble. Silver, with his revolutionary disregard for everything but the polling numbers, was still arguing as late as Thursday afternoon that Coakley was the clear favourite; he changed his mind at midnight that evening and acknowledged that Scott Brown had a puncher’s chance.

He continued to soothe jumpy Democrats throughout the weekend in the manner of a government radio station denying rumours of a coup d’etat, writing on Sunday night that the election was still a “toss-up”. (He had already cheated on his mechanical bride by citing traditional eyeball analysis from NBC’s Chuck Todd: “If this were any other state we’d say this one was over”). There followed another handful of balm. Tonight, as the national party prepares for the probably loss of its congressional supermajority, he’s still distributing it.

Silver (or “Nate P. Silver”, as I will always think of him) may still be “right”, in the limited sense that a probabilistic prediction about an event that will only happen once can be “right” at all. But even if Coakley does surprise everyone by pulling this election out, the gruesome lack of robustness in Silver’s approach should be evident. We don’t need an advanced proprietary model to tell us what the polls are saying with a five- or six-day lag time built in. I have always understood Silver’s core claim of special expertise to inhere in the ability to give useful information about the future. Boasts like “[the model] correctly predicted the outcome of all 35 Senate races in 2008” are nothing but distracting hype, since 95%+ of Senate elections are easy to call on the morning before they happen. And as Silver certainly knows, a model that delivers probabilistic estimates of outcomes of one-time events has to not only be “right”, but right in such a way that bettors using it would, in the long run, outperform other bettors or prediction markets not equipped with that model. That’s the only appropriate test, and I know of no evidence that he has passed it.




Browse

Mr. Overrated

  1. "…a model that delivers probabilistic estimates of outcomes of one-time events has to not only be “right”, but right in such a way that bettors using it would, in the long run, outperform other bettors or prediction markets not equipped with that model. "

    Actually, I think a number of pollsters and news organizations define "right" in terms of "whatever brings Democrats out to vote by making the election seem close rather than hopeless, or discourages Republican voters by making them view the election as hopeless rather than close." In that sense, Silver has got it exactly "right", and has maintained his righteousness all along by shifting from approach B to approach A as other poll results became publicly available.

    Sort of a "progressive" version of "right", you see.

    • It must be hard, walking around with such a sense of persecution. This right wing victimology just has to stop. Stand up on your own two feet and make something of yourself. Don't mope around, complaining that it's just so tough being right and get on with it, man. Next thing you know you'll want a government program or law to make it all better.

    • Right, because if there's anything the last 15 years have taught us, it's that the mainstream media slavishly supports the Democratic party and progressive causes.

      (In case you couldn't tell through the tinfoil, that was sarcasm)

  2. Has this useless piece of humanity ever admitted his contributions after the murder of four RCMP officers, as he claimed publicly that it was "only a few" pot plants which the perpetrators were growing? This piece of trash does not deserve a national voice in such an esteemed publication as Macleans.

  3. Good lord Cosh, Silver's a stats geek- ever read his work at Baseball Prospectus? It's fantastic stuff. You think he's part of the great liberal conspiracy? To paraphrase Be_rad, this victimology crap sounds just as bad coming from the right as it does from the left.

    • What "victimology crap"? And how is a "stats geek" automatically above reproach? I've read Silver's BPro work; it's not in any way relevant to his political stuff, except possibly insofar as PECOTA is just as overrated as his political model.

    • Yes, good lord, Cosh. The man has done good work in the past. Hence, anything he does is good work. As for your specific and substantive arguments, I will dismiss them with a haughty and irrelevant reference to political ideology.

      QED.

      • Touche on PECOTA. He may seem political to you but I still see him as a stats geek. My take is that he's purely reading tea-leaves and nothing more, and he's damned good at it.

      • Aw hell Dan, I'm trying to become a right-wing blog commentator. It'll take me a few turns before i get real good.

  4. But it's FUN!

  5. Being thunderously, jaw-droppingly wrong is not a career-limiting move in American journalism. Just look at William Kristol or Charles Krauthammer.

  6. I'm still pissed that I carried Zack Greinke for two years because PECOTA failed to account for his various anxiety problems.

  7. I'm just wondering how much play this race is getting up here in Canada, especially if Brown ends up doing the virtually impossible and taking "Ted Kennedy's seat" away from the Democrats.

    Some of you may recall the 24/7 coverage that Kennedy's death and funeral got up here, which included endless references to his legacy of "serving the people" and so on.

    So, will "the people's" rejection of some of that legacy, including Obama's vaunted healthcare proposals, get as much coverage here? Or, better yet, how about the various threats coming from the left that Democrats should pass their monstrosity of a bill regardless of democracy?

    Should be fascinating to watch.

    • I think that you're reading too much into what looks to be a Democratic loss. Coakley is an atrocious candidate – lots of people who might be Dem minded are faced with a hell of a choice between supporting someone who ran (and continues to defend) one of the most awful prosecutions in American history and a Republican.

      • I think you're relying far too much on shameless Democratic talking points.

        A Republican victory in one of the most liberals states in America, in Ted Kennedy's recently vacated seat, in which one of the primary issues is Obama's healthcare plan, and we're not supposed to read anything into this? My God.

        • Kinda the mirror image of Dem. Ben Nelson winning Nebraska in 2000.

          Take a deep breath.

  8. Silver's got a solid track record and has done good work in past elections. No, he can't predict the future, but his models have stood up well to scrutiny, both theoretically and in practice.

    That said, it wouldn't surprise me to see him bomb a bit in this special election – it's like predicting by elections…it's tricky to do based on polling or numbers.

  9. What are we arguing about here? Nate Silver is giving Choakley a 25% chance of winning.

    • Colby's criticism isn't that Silver's final prediction is wrong, rather it is that he was so late in predicting it. There is actually little value in predicting the outcome of something the day before, when the conventional wisdom has already come to the same conclusion. In this race, Silver did keep trying to offer reasons that Coakley was less likely to lose than the conventional wisdom predicted. He also showed that with different assumptions his model could have Coakley ahead (albeit before the last three polls came in). The MA senate race is not exactly a triumph for him, and even if Coakley ends up winning (Intrade has her at 25%), he will have lost his bragging rights by hedging at the last second.

      (still, his enterprise is an interesting and useful one that should not be judged based on a single case)

      • But just over a week ago, *all* MA polls were all over the place. If Nate Silver was wrong last week (I haven't followed his work at all), then he was no more wrong than anybody else.

        Here's some good readin' on the topic: http://www.talkingpointsmemo.com/archives/2010/01

  10. From Silver's "Massachusetts Model Mayhem" post:
    To state the obvious, one's assumptions matter a lot! Any of these are reasonable and defensible sets of assumptions. And I'm sure that you some the more creative among you could come up with other wholly reasonable and defensible sets of assumptions, including some that fall outside the goalposts of the scenarios contained herein.

    I'd have to agree with Colby's assessment of a "gruesome lack of robustness in Silver's approach". Silver shows that by tweaking various sets assumptions, dropping polls that seem like outliers, weighting the most recent polls differently, and increasing the "uncertainly parameter", he can produce six vastly different scenarios, each of them theoretically defensible.

    In other words, it seems more like complicated guesswork and less like reliable prognostication.

    • You are focusing too much on the top-line question (who wins). Though Brown is probably ahead, the race is at best one that leans GOP. There will always be more uncertainty in close races. Indeed, we would be foolish to assign laurels on somebody that accurately predicted say, the outcome of the 2000 election. Any good model of that election would have had the race as a coin-toss.

      Assumptions do matter, especially in Bayesian models like Silver's. However, if they are applied consistently and get good results, then we have a decent predictive model. If they don't lets look at why.
      -maybe Nate over-weights past historical factors?
      -maybe his model is too conservative, and will do badly in late shifts of public opinion (three hundred eight has this problem in my view)
      -maybe his model is calibrated based on general elections, and will do poorly in off years
      -maybe there are intangible factors his model can't capture effectively

      Instead of slagging his model, make a better one (there's a substantial payoff in it for you).

      • I'm no stats wizard, so I'm sure that Silver's proprietary model is much, much better than any that I would be capable of creating. You're probably right that it was the late shift in public opinion towards Brown that threw a wrench in the gears of Silver's model. Maybe after this experience he will tweak his assumptions so that future models will be more sensitive to late polling break aways from the incumbent.

        Still, it seems like no single model can be predictive, and therefore useful, for all elections. For close, fluid contests like this one there is so much uncertainty that I question whether a decent predictive model is even possible.

        • I'm a daily visitor to Silver's site. I go there for his parsing of other polls – in the US there
          are dozens daily – not because I pay much attention to polls. The media makes much of them
          because it's easy copy with no accountability. Who remembers, except for extreme cases, what
          was written about polls a week ago when there's always another one next week.
          And there are a couple of other bloggers on the site – one who writes from and about Europe –
          who are worth reading.

  11. I'm not a big consumer of talking points. My approach involves looking at all of the information, including the fact that Coakley had a huge lead until people actually started paying attention, and concluding that this isn't so much a repudiation of the Democrats as it is a repudiation of her. This fits with my own biases – I'd lean Democratic if I was American but I wouldn't vote for this woman, based on the Amirault stuff. Your approach apparently involves doing nothing more than looking at the polling and declaring that a sea change has taken place.

    We'll see how long Brown is a Senator. I'd be willing to bet that his hold on Kennedy's seat expires at the next election.

    • I'm sorry, but who in the world are you trying to kid? This race has almost everything to do with tipping the balance in the Senate, and the people of Massachusetts are clearly on the verge of telling Obama and the Democrats what they can do with their majority and healthcare.

      I can understand Democratic politicians trying to get away with this kind of stuff in two-minute interviews, but you on here? Wow.

      Is Coakley a bad candidate? Yes. But it hardly explains this shot heard around the world. Indeed, her existence as a candidate is reflective of the kind of arrogance and aloofness that the Democrats now represent only one year after one of the most historic elections in American history. That they would even consider denying him his vote in the Senate on these issues says so much about the lack of hope and change that Obama's presidency has been about.

      I don't think even George W. Bush could screw things up this much and this quickly. But, don't you worry, Obama will continue to be worshipped by his admirers. Hey, they have to keep forcing their religion on us, don't they?

  12. My main issue with Nate Silver and his site in general, is that (despite his lineage), he keeps "discovering" things that are well-researched by political scientists already. His analysis ranges in quality – his model of the primary WAS good and outdid the pollsters because he included past results and state-level factors as well as the polls themselves. His presidential model predicted a clear Obama victory, even in the weeks just after the Republican convention where some polls gave McCain-Palin a slight edge. On the other hand, his misapplication of Binford's law (on the Iran issue) and his overly simple regression of the chance of Vermont's gay marriage ban passing were weak.

    In that sense Silver's approach is more useful than polls. Polls ask the question "who would you vote for if there was an election right now". When an election is distant, however, that question is not always a good representation of how people will actually vote. Incorporating other factors is a good way to ask the question "who WILL win".

    Predictive models should not be judged based on their ability to predict a single event, but on whether they generally outperform other models (or non-quantitative prediction attempts). I think Silver has been especially inaccurate in the past while because his model does a bad job of capturing the midterm/special election electorate. Fewer people vote in midterm/special elections, which tends to favour the Republicans (doubly so, now that they have more motivated supporters).

    One other issue is that it is never clear how much Silver's model explains why a party will win. Whereas predictive models benefit from a kitchen sink approach (lots of variables), that same approach muddles the explanatory power of a model. Thus, if you want to talk about why, perhaps Ray Fair's more parsimonious (but less accurate) approach will be more your speed.

    • I had forgotten that Benford episode–another occasion that had me wondering why, oh why the emperor's legendary new outfit seemed so suspiciously transparent.

  13. Bush, who many on the left consider a moron, got plenty passed with the 60 vote Senate filibuster rule. Obama, who promised bipartisanship and a new era of cooperation, seems incapable of getting any stuff passed now, even with 60 votes. Fascinating.

    It should also be noted that this special election is being conducted with full knowledge that it represents a swing vote, and hugely liberal Mass. is still willing to say NO to the Obama agenda. After only one year. Again, fascinating stuff.

  14. Why are some of you so unwilling to acknowledge the enormous significance of this potential upset? Is it so troubling for you that the people are in the process of stopping Obama's agenda in its tracks just one year into his beloved presidency? lol

    • Well, no. What's troubling is that, even assuming a Democratic loss, a 59-41 majority
      means that nothing happens. Of course not much happened even with the magical 60th
      vote. A system that doesn't permit a majority to govern is even more farcical than our own
      minority government.

  15. I have a lot of respect for both Cosh and Gardner. But this post and Gardner’s comment are smug and mean-spirited.

    Silver is a guy who excels at analysing stats. In his political commentary, he helps make sense of multiple, conflicting, shifting polls. And he’s damned good at it.

    In this case, it looks like his heart overruled his brain to some extent. He spun too many scenarios where Coakley could win, too late in the campaign. He was hoping for the best in an election with so much at stake — that could spell the end of health care reform.

    But Silver’s a blogger, you know. Rumour has it that bloggers are entitled to their biases.

    It looks like sour grapes to me; jealousy over Silver’s prominence.

Your email address will not be published. Required fields are marked *