The National Household Survey: Pig, meet lipstick -

The National Household Survey: Pig, meet lipstick

The survey won’t answer the crucial questions only the long-form census could, writes Stephen Gordon


(Sean Kilpatrick/CP)

I wrote a lot about the decision to cancel the mandatory long-form census and replace it with with the voluntary National Household Survey (NHS) in 2010-2011 (see here: [1][2][3][4][5][6][7][8][9][10][11][12]), but that was all prologue. The real implications of the new survey are only now being played out with the release of the the numbers –  I can’t bring myself to use the word “data” – collected by the NHS.

Here’s what Statistics Canada has to say about the quality of NHS numbers:

We have never previously conducted a survey on the scale of the voluntary National Household Survey, nor are we aware of any other country that has. The new methodology has been introduced relatively rapidly with limited testing. The effectiveness of our mitigation strategies to offset non-response bias and other quality limiting effects is largely unknown. For these reasons, it is difficult to anticipate the quality level of the final outcome.

The significance of any quality shortcomings depends, to some extent, on the intended use of the data. Given that, and our mitigation strategies, we are confident that the National Household Survey will produce usable and useful data that will meet the needs of many users. It will not, however, provide a level of quality that would have been achieved through a mandatory long-form census.

My reaction to the “usable and useful data that will meet the needs of many users” bit is “like who?” Because I’m far from convinced it will be useful in answering the sort of questions that census data have been used to answer in the past.

Most of the commentary I’ve seen makes the point that that if you’re interested in certain broad features about one variable, the NHS should provide usable information. This is quite likely to be the case: Statistics Canada has many other sources of information that can be used as a check against the NHS: tax files can be used for income, CMHC data for housing, and so on. Where these sources fall down – and where the NHS can’t be used with any confidence – is at the micro level of neighbourhoods and other narrowly-defined criteria.

But the real problem goes beyond that. We’ve always had non-census sources of information for several of the dimensions in previous censuses. (A notable exception is immigration data. The only reliable data we have about how immigrants are faring come from the census. The NHS may be able to piece together numbers based on the 2006 census and available information on inflows and outflows since then, but this is at best a patch job.) But what made the census special was that it captured the correlations between all of these variables. We may have a good idea about, say, the distribution of  income, educational attainment levels and immigration status from non-census sources, but only the census could be used to put them all together to make meaningful inferences about their statistical relationship.

The NHS numbers may provide useful answers for questions where the availability of census data wasn’t crucial. But they won’t be much use in addressing the really interesting questions that only census data could answer.



The National Household Survey: Pig, meet lipstick

  1. “But what made the census special was that it captured the correlations between all of these variables.”

    NatPost Sept 2012:
    An unexpectedly high number of same-sex marriages in places like the oilpatch left census-takers scratching their heads — until they realized many of the “couples” were only splitting the rent. As a result, Statistics Canada said Wednesday, it may have overestimated by as many as 4,500 the number of same-sex married couples in parts of the country.

    XKCD – Correlation:

    Man: I used to think correlation implied causation. Then I took a statistics class and now I don’t.
    Woman: Sounds like the class helped.
    Man: Well, maybe.

    • You do know what realized implies, right?

  2. “Bureau of Statistics”

    CS Lewis – I live in the Managerial Age, in a world of “Admin.” The greatest evil is not now done in those sordid “dens of crime” that Dickens loved to paint. It is not done even in concentration camps and labour camps. In those we see its final result. But it is conceived and ordered (moved, seconded, carried, and minuted) in clean, carpeted, warmed and well-lighted offices, by quiet men with white collars and cut fingernails and smooth-shaven cheeks who do not need to raise their voices. Hence, naturally enough, my symbol for Hell is something like the bureaucracy of a police state or the office of a thoroughly nasty business concern.

  3. Don Draper never used stats.

  4. Actually, the analytic document “Immigration and Ethnocultural Diversity in Canada” says Phillippines were largest source of Cdn immigrants from 2006 to 2011. Accompanying footnote says this conflicts with other data. That’s a pretty high level descrepancy.

  5. when I saw that they were scrapping “the long” form Census I knew immediately it was a big mistake. In any other “sane” world you might infer that the government wishes to commit subterfuge by hiding important statistics to pander to the very wealthy families and interests that really rule the country. Transparency and Accountability indeed. This government better wake up because the writing is on the wall. Unless the Canadian public is as stupid as I am beginning to believe. Mad as h**l, you bet!

  6. The Harper Government has all appearances of being engaged in a widespread effort to suppress facts and evidence: from census data, to access-to-information requests from the media, to public budget documents, to muzzling scientists. In fact, Harper has created a Soviet-style information-control bureaucracy comprised of 1500 communications staff to carry out this farcical agenda. The Conservatives don’t have something to hide: they have everything to hide!

  7. “…but only the census could be used to put them all together to make meaningful inferences about their statistical relationship.”

    Wasn’t that the point of the change?

    It’s the unwritten law in Harperland, haven’t you heard…one must not commit sociology.

    Once they’re finished with uppity scientists and traitorous enviros, they’ll probably be coming for independent minded economists Mr G.

  8. Because the data it generated are unreliable, the NHS is actually worse than useless – it was also a needless expense, another example of the stupid-and-proud-of-it Cons wasting taxpayers’ money.

  9. This is a good one from The Economist:

    “Moreover, in order to keep the sample size constant despite a reduced response rate, the government would have to send out more forms, at an additional cost of C$30m ($29m). Canadians would be paying more money for less accurate information.”

    A senseless census reform

    • And in my northern town the enumerator[is that the term] assured me that they were not telling people, [unless they were asked] that the LF was no longer mandatory. Typical Harper govt schmozzle.

  10. What made the long form census special was that people were compelled to complete it under penalty of law. It’s the legal penalty aspect that offends, particularly when much of the information appears to primarily be used to support commercial activity or satisfy academic interest. I think proponents of the mandatory long-form census would’ve done well to try to replace the threat of penalty with some positive inducement. There’s no doubt the census was useful and important in a number of areas. I’m just not certain that there was a good reason to threaten people with jail or fines for not filling it out.

    • They tried that, actually. Proposed the very solution you’re mentioning. There was zero flexibility on this issue.

      • If the issue with the NHS is problems with non-response rate, though, and the primary value of the long form census was that its mandatory nature led to a high response rate, I don’t see how merely dropping the penalties would’ve changed the current outcome.

        • Incentive based doesn’t address non-response bias, as those who value their privacy over the incentive will still choose not to respond.

          Only by compelling people to file, which requires associated penalties, can this bias be addressed.

          And the reason is simple, and it’s the same reason there’s prison terms if you fail to file your taxes often enough, or don’t pay your speeding tickets or whatever else you do that’s against the law — the reason is that being a member of our society comes with rights, limits, and responsibilities.

          • Civic responsibility is an important concept, and one that is not given sufficient weight. The long form census doesn’t meet a standard of responsibility that justifies criminal sanction for failing to complete it. Legal sanctions are reserved for actions that result in harm to others. Difficulty obtaining an adequate response rate for statisticians is an inconvenience, not a matter of vital public interest. By your standard of responsibility, there should be criminal penalties for failure to exercise 30 minutes per day or brush your teeth (cost to the health care system and economy, plus economic benefit to manufacturers of running shoes & toothbrushes.) Paying taxes is an entirely different matter – despite attempts by some to deny it, taxation is widely accepted as necessary for the functioning of society. Most of the disagreement around taxation is how much to tax, how to collect it, and especially what it pays for.

          • Really? By my standard of responsibility? That’s interesting. Go on and tell me more of these things about myself that I didn’t know. I’m curious.

  11. Considering the fact that virtually anyone who was found guilty of not filling out the long-form census could have faced life imprisonment with no chance of parole for 25 years with a 360 pound cellmate named Big Bernie, who would insist that you do look pretty in pink, it’s no wonder that the Conservatives decided to go this route. We have been saved by the National Household Survey and the Stephen Harper government from a fate far worse than death itself.

  12. i wonder how much the CPC messed everything up even just by demonizing the idea of the census and pointing out the government almost never goes after anyone. That might have been the cherry of malicious on top of the stupid sundae.

  13. “But they won’t be much use in addressing the really interesting questions that only census data could answer.”

    Yes, it’s true that there’s a strong correlation between private information and really interesting information.

    There’s also an interesting dynamic of journalists, academics, and socialist politicians wanting information to be extracted from private citizens for the benefit of journalists, academics, and socialist politicians, while claiming that the extraction of said information from private citizens is actually in the interest of private citizens. Such altruism.

    The census disaster is upon us all. We should all weep. We should yearn for the glory days of the authoritarian regimes who knew everything and anything about their citizens. Those regimes were all such models for the rest of the world. Meanwhile, the freedom-loving regimes of the world have become such scientific backwaters. Oh the horror.

    It’s such a shame that all the less interesting questions in science, like the workings of the human brain or the basic building blocks of matter, those boring questions will have to occupy our scientists, now that they simply cannot be sure about what I earn, where I came from and what education I attained in school.

    Is there someone in the country that is not an academic, journalist or socialist politician whose opinion matters on this subject? Now there’s an interesting question. I’m sure there’s lots of talk on the street about the disaster that has befallen us all.

    • “We should all weep. We should yearn for the glory days of the authoritarian regimes who knew everything and anything about their citizens.”

      Wait a second, weren’t we one of those countries until just this last “census”? What a peculiar way to think of Canada when we had a mandatory census.

      I am not a politician (of any stripe) nor an academic nor a journalist. I am at a bit of a loss to see how aggregated knowledge about our country which can then be used to fine tune everything from policing and fire services accessibility to municipal / provincial / federal programming is so dangerous.

      • That’s because you haven’t spent enough time learning how to tighten a tinfoil hat, while s_c_f is a master of it.

      • You’ve got a lot of straw men there. I was exaggerating of course, in order to prove my point that there is no negative benefit from discontinuing the long-form census, the benefits are positive. And the fact of the matter is, when you take the argument to the extreme, it becomes obvious.
        The part where you and your cohorts are exaggerating is the part where you pretend that we don’t have a census anymore. We do, and it’s mandatory.
        No, the long form census does not “fine tune everything from policing and fire services accessibility to municipal / provincial / federal programming is so dangerous”. There is no fine-tuning required. You census buffs make these ridiculous arguments that the census makes the grass greener and the sky bluer. It doesn’t. This is the part where you’ve bought into the ridiculous propaganda peddled by academics, journalists and socialist politicians.
        We still have a census. We don’t have a long-form mandatory census. The long form census has NOTHING to do with policing or fire. Absolutely nothing. My income, religion and my other personal details has NOTHING to do with policing.

  14. It’s sad to see so many of my fellow Jedi failing to report their status.
    Maybe they no longer fear, which has lessened their anger, and relieved them of hatred.

  15. This is Harper government’s stupid mistake. At least we should know the percentage of each race/ethnic group portion. National Household Survey? with that response rate, I do not think that represents real Canada.

  16. In all of this, I have yet to hear a compelling reason why the government needs anything more than a simple head count. The idea that any government can somehow improve our lives through the magic of “planning” is laughably stupid.

    • I know, right? Schools, hopitals, public transit, social services…put ’em anywhere!

  17. I worked on the National Census one year…and if my results were any wasn’t all that reliable either. Many people were unwilling to fill in more than their name and address…The farm reports were very difficult to obtain…People didn’t want to list and value their machinery. My supervisor told me…just try to get the value of their pick-up. No thanks …I found myself guessing the age of their pickup and checking the papers for values…That’s just a small example of the things everyone was forced to do in order to complete their paperwork.
    I think the statute of limitations has expired on my confession and this is why I have no respect for the longform census.