The monthly Labour Force Survey is one of the most important pieces of information that Statistics Canada provides. It’s mandatory, disseminated without cost, and there is no way similar information could be collected by a private sector provider. It comes out mere days after the survey questions are put to respondents, providing an important, timely and up-to-date picture of the performance of Canada’s economy. In contrast, GDP numbers arrive months after the fact and are often subject to large revisions. The Labour Force Survey is exactly what a national statistics agency ought to be doing and, in my view, the end product is of very high quality.
Over the past year, several private sector economists have challenged the method and reliability of the Labour Force Survey. See Bank of Montreal’s Doug Porter in the Globe and Mail, for example. In response, Chief Statistician Wayne Smith has written a blog post and a Globe and Mail article defending the Labour Force Survey against several claims, which he denounces as “myths.” Smith makes several reasonable rebuttals about survey volatility and funding for the survey. However, in my view, Smith trips over one important issue: whether or not the Labour Force Survey is adversely affected by the loss of the long-form census. He goes so far as to claim as “myth” the idea that the Labour Force Survey is affected by the replacement of the long-form census by the more expensive and less reliable National Household Survey. I disagree.
The Labour Force Survey uses the census in several ways. One use for the census is in picking which households get asked to respond to the survey. Rather than randomly picking from the millions of households across Canada, Statistics Canada uses a standard “stratified sampling” approach that targets certain areas in order to improve reliability. Another major use for the census is to form the benchmark weights that are used to make sure the Labour Force Survey is representative.
Smith claims the new Labour Force Survey method depends only on the mandatory short-form part of the 2011 census, not the voluntary 2011 National Household Survey. The Labour Force Survey method was recently updated to incorporate the 2011 short-form census results—presumably, updating both the way that the sample is chosen and the way the sample is weighted.
I say “presumably,” because the most recent complete method document I can find is from 2008. In that document, it is very clear that the census plays a very important role in the method of the Labour Force Survey. In fact, a quick “control F” search of the document reveals the word “census” appears 127 times. Looking through these instances, you can see the long-form census was used in a variety of ways. Sometimes, it was to cross-check an assumption or a decision they made in designing the Labour Force Survey. In other places, it is clear they used the long-form census explicitly to pick which households get surveyed. To give one precise example, page 19 of the method document explicitly references income taken from the 2001 Census—and income is not available on the short-form census.
In short, I think Smith’s “myth” is a miss.
I have complete confidence in Smith’s claim that the 2015 Labour Force Survey method now uses only the short-form census for selecting and weighting households. However, it is equally clear that, until now, the Labour Force Survey relied on the long-form census. This change raises several important questions: Why make a change? Why was the National Household Survey discarded? Is the Labour Force Survey improved by ignoring the National Household Survey? Why and how?
I have a pretty good idea why Statistics Canada discarded its own expensive National Household Survey for designing the Labour Force Survey: The response rate from the National Household Survey is low, and varies strongly within regions. As one example, above is a map made by Dwight Follick that compares the response rate at the “dissemination area” level for Toronto from the 2006 long-form census and the 2011 National Household Survey. Similar maps are available from Follick for Montreal, Ottawa, London, Calgary, Edmonton and Vancouver. The pictures aren’t pretty; the response rate fell dramatically, rendering the results of the National Household Survey much less reliable.
Smith might be correct that the Labour Force Survey method changes aren’t large enough to make much difference. But we will never actually know, since we can’t compare the new results to the previous method, because the 2011 long-form census doesn’t exist, so we can’t check.
I will continue to use and trust the Labour Force Survey, myself. However, the switch to the National Household Survey has degraded some of the tools that Statistics Canada has used until now to make the Labour Force Survey—and other Statistics Canada products—reliable.