General

Now is the time to argue about Nate Silver’s method

Jamie Weinman on the gap between trusting your gut and stats

Eric Gay/AP

My colleague Colby Cosh got a lot of favourable notice for this piece about Nate Silver. Well, to paraphrase Patty and Selma from The Simpsons, I believe the best way to write a post is to leech off the popularity of another post. But also, I had something I wanted to say about Silver. Or more about perceptions of Silver.

Arguing about Silver has suddenly become a big thing in the last few weeks. He’s been taking criticism from at least three different directions. First, there are the people who don’t so much have a beef with the man himself as with the idea of him as an oracle. Silver does not claim to be an all-knowing prognosticator. But there are people out there who see him as such. I know a few people who have told me the U.S. presidential race can’t be close because Silver gives Obama a 75 per cent chance of winning, or who simply refer to Silver alone to tell them what’s going to happen. Silver has never set himself up as a prophet, and he can’t be blamed for his adherents, but I do think that the way he expresses his findings is very vulnerable to misinterpretation. It’s true that Silver isn’t literally saying that Romney has almost no chance of winning; he’s talking about probabilities. But they do lend themselves to those talking points, even if that’s not his fault.

The second anti-Silver faction consists of Republicans and conservatives who detect a bias in his work. You can see one such argument here. To some extent, this may be misdirected: the polls themselves, particularly the swing state polls, have tended to show Obama in the lead. But there is an argument that any system with a subjective element — and while Silver has stayed true to his system, it includes subjective decisions about how things are weighted — has an element of bias in it.

From a Republican point of view, Silver may have stood in the way of developing a media narrative of Romney momentum after Denver; as the national polls started shifting toward Romney (many of them have since shifted back again) Silver continued to say that Obama was the favourite based on swing-state polling. I don’t know how harmful that actually was to the Romney campaign, but perceptions of momentum are quite important to some of the people who run campaigns.

Finally, there are the pundits and reporters, many of whom see Silver as an annoyance at best and an enemy at worst. Here’s the article that mentions some of the anti-Silver sentiment brewing among pundits; one of the people from Politico also mocked people who think Silver has some kind of “secret sauce,” when he’s actually just “averaging public polls.” Though, I don’t think he, as opposed to some of his fans, have claimed he has any “secret sauce.”. As many people have already noted, this is developing a lot like arguments over baseball statistics: who has the better perception of the game, the guy who goes out and talks to the players and has inside knowledge of what goes on, or the guy who sits at home with the statistics and plugs them in?

It’s not as simple as that in election forecasting, because analyzing baseball statistics is about analyzing things that have happened, while polling is about things that haven’t happened yet. With predicting the future, it is probably true that inside knowledge can help you see things the stats don’t — just as someone who knows about a baseball star’s drinking or drug problem will do a better job than the sabermetrician of foreseeing his upcoming decline. An example from 2010: Jon Ralston of the Las Vegas Sun, who predicted Harry Reid would be re-elected at a time when Silver gave Reid’s opponent “a better than three-in-four chance,” thanks to polls that were turning in her favour. Ralston didn’t have a lot of evidence to give us, but he did have his reputation as a clued-in, plugged-in observer of Nevada politics, and what he observed was that Reid’s political operation was as strong as ever, and that his opponent wasn’t being carried along as strongly as she should have been by that year’s Republican wave. This is the sort of thing you can probably see that the polls can’t — if you’re intimately familiar with the political workings of a particular area or state.

But most people who forecast elections, of course, have no such familiarity. Even people who live in a state, while their local perspective is almost always more insightful (for example, a local can tell you not only what ads are on television, but what they’re saying on local news and the weirdly political world of sports radio), are going to have limited knowledge of what’s going on. Other people just tell you that someone must be winning, no matter what the polls say, because he had huge turnout at some rally, or the locals seem to be getting really excited about him. And then there’s the most problematic of all these little subgenres: talking to interested parties and asking them if they think they’re winning. Of course they think they’re winning, and can give you all kinds of reasons why. But why on earth would that be more useful to us than looking at an average of the polls?

It always seems counterintuitive and wrong that a guy staying at home with the numbers, never setting foot in a state, could have more insight into the situation than someone who does shoe-leather reporting on the ground. And in one sense that’s true: the number-crunchers would be nothing without the people in the field doing the polls. But in terms of the actual process of figuring out who’s likely to win, this is probably one of those situations Bill James described in response to criticisms of the sabermetric method: told that sabermetricians can’t see the forest for the trees, he pointed out that the trees aren’t in a good position to tell us how tall they are. To go and report on baseball up close, you find out a lot of things, but you still need hard cold statistics to put the season into perspective for you and find out stuff like, well, who’s ahead in the standings. If you ignore the stats and just “trust your gut,” you get something like this piece from Peggy Noonan, a full-blown pundit in good standing, where she argues that the polls don’t matter because she’s hearing a lot of people have signs in their yards:

There is no denying the Republicans have the passion now, the enthusiasm. The Democrats do not. Independents are breaking for Romney. And there’s the thing about the yard signs. In Florida a few weeks ago I saw Romney signs, not Obama ones. From Ohio I hear the same. From tony Northwest Washington, D.C., I hear the same.

Is it possible this whole thing is playing out before our eyes and we’re not really noticing because we’re too busy looking at data on paper instead of what’s in front of us? Maybe that’s the real distortion of the polls this year: They left us discounting the world around us.

This is the kind of predicting that, even if it turns out to be right, is completely useless. It’s useless because it’s not based on anything; it tells us nothing except that humans will pick out the signs and portents that tell them what they want to hear. With Ralston’s prediction about Harry Reid, we could at least look back on it after the election was over and learn something about the way politics works in Nevada. But most gut-feeling punditry, I think, is closer to the Noonan quote: someone is going to win because I feel it in my bones, or a particular candidate has the “momentum.”

This is why I think now is right time to argue about whether Silver’s method makes sense, rather than after the election. There are many reasons why he might wind up calling the election wrong (along with a lot of other poll aggregators, pundits, and so on). There are also flukey reasons why he might be right. It doesn’t exactly matter a lot, since the actual election renders all advance polling completely irrelevant. The question is, though, before the election, when predictions are all we have to go on, which predictions are useful? Which methods shed some light on the state of the race at a particular time? Which posts seem like they might have something useful to tell us about where the polls stood, even if things change?

I think there are some ways in which the Silver method makes the race more confusing, creating the impression that races are less volatile than they really are, and under-stating the chances of surprises like the Harry Reid/Sharron Angle race. And I think it’s important to take the polls in conjunction with some bigger-picture reporting. But that’s not the choice we usually have: the choice we have is between poll aggregation and analysis, and pundits reporting “SHOCK POLL: OBAMA LOSING [name of state]” or telling us how David Axelrod thinks things are going. With a choice like that, no wonder people turn to Nate Silver.

Looking for more?

Get the Best of Maclean's sent straight to your inbox. Sign up for news, commentary and analysis.
  • By signing up, you agree to our terms of use and privacy policy. You may unsubscribe at any time.