How polls keep underestimating the Conservative vote: 338Canada

Philippe J. Fournier: Canadian political polls tend to hit the mark, except when it comes to counting Conservative support in the West. What's going wrong?

Once all the October 2019 federal election votes were counted, the Conservative Party of Canada had won the national popular vote by one point over the Liberal Party, 34 per cent to 33 per cent—a margin of roughly 210,000 votes from coast to coast. The Liberals still managed to prevail in the seat count by winning the vote in Canada’s two largest provinces, while the Conservatives ran up the score in western provinces, especially Alberta and Saskatchewan.

Comparing the election results to the final polls from Canadians firms, we noticed several firms had correctly called the results—meaning the final numbers were mostly within a reasonable margin of error (or uncertainty) of said polls.

Here are the final polls of the 2019 election campaign compared to the actual election results (for more details on the 338 Score and grading system, visit the 338 Canadian Pollster ratings page):

So did Canadian pollsters underestimate the Conservative vote in 2019? The answer to this question deserves some analysis. At first glance, as you can see from the table above, no individual pollster was more than four points off the CPC national result. And, among the five pollsters with an A or A+ mark for this election (namely: Léger, Ipsos, Mainstreet, Nanos, and Abacus Data), none was more than two points off the mark. Therefore, the quick answer is “no”.

However, we also notice that no pollster overestimated the CPC vote either. Consequently, it would also be correct to state that while no individual pollster underestimated the Conservative vote outside a reasonable confidence interval, they collectively underestimated it.

Here is a comparative chart of Canadian pollsters’ final polls and the election results:

[On the graph above, both axes are graded in percentage points off the actual national election results. The Liberal vote is on the x-axis and the Conservative vote on the y-axis.]

Again, it must be stated: the final federal polls in the last election were generally really close to the actual results, which is why seat projection modelling was also fairly accurate. Nevertheless, we also notice that every single data point on the graph is located below the graph’s bullseye—meaning an underestimation of the Conservative vote. Where did this discrepancy occur?

Let us redo the graph above, but using the final poll numbers and election results from the four most populous provinces:



In Quebec and Ontario, poll numbers for the Liberals and Conservatives were, on average, really close to the actual results. The same could be said about the British Columbia numbers. However, we notice that the Alberta results are fairly titled towards the lower part of the graph.

Here are the 2019 final CPC poll numbers in Alberta from the aforementioned polling firms: 63, 59, 61, 64, 57, 57, 60, 58, 54 per cent. A simple average of these numbers gives us 59 per cent. Yet, the actual CPC share of the vote in Alberta in 2019 was 69 per cent.

It would be natural to point to the fact that the regional sub-samples of polls are smaller, hence they have higher uncertainty, and therefore should theoretically show greater variance from the election results. This is a fair argument. However, if smaller sample sizes were the only explanation for the discrepancy, we would see dots all around the bullseye on the graph:

Clearly, that is not the case. On average, polls underestimated the Conservative vote in Alberta by 10 points.

Of course, in any other province, such a miss would have been catastrophic for the seat projections. In Alberta, however, the 338Canada model correctly called the winner in all 34 federal electoral districts: Edmonton-Strathcona went the to the NDP, and the Conservatives won the remaining 33 districts in the province—including 32 districts with more than 50 per cent of the popular vote.

Which brings us to the current numbers and a very simple question: Are Canadian pollsters still underestimating the Conservative vote, specifically in Alberta? Here are the latest CPC numbers in the province from the last five pollsters in the field:

A simple average of the above numbers gives 48 per cent. Therefore, according to the current data, the CPC would have lost about 20 points in Alberta since last fall’s federal election.

I am a data-oriented person and I believe data—especially from experts with proven track records—is very often far more precise than mere impressions (albeit not always). This is not merely a belief of mine: It is a verifiable fact that political polling in this country has hit the mark far more often than not in recent years. Take a look for yourself.

Nevertheless, I must admit that I find this hard to believe.

One plausible hypothesis for these low poll numbers for the CPC in Alberta would be that the CPC leadership race has been eclipsed by the COVID-19 pandemic, which, according to vast amount of polling data, has mostly been handled appropriately by both provincial and federal levels of government in this country. In fact, the latest Léger poll measures satisfaction for the federal government’s handling of the pandemic at above 70 per cent—a figure that has remained mostly constant for many months now.

Nonetheless, given that polls had also underestimated the right-of-centre UCP vote in the 2019 Alberta general election (see complete table here), this may be a systemic issue pollsters in this country may want to study further.