How to understand that controversial study of Quebec daycare

Stephen Gordon explains what the paper examining the Quebec daycare program reveals, and the ensuing controversy


 
  1
 Ian Barrett/CP

Ian Barrett/CP

Last week a National Bureau of Economic Research working paper was released that analyzed the long-run non-cognitive impacts of Quebec’s daycare program, and almost immediately the paper became a target for critics. Here, Université Laval economist Stephen Gordon explains the paper’s findings and the controversy:

 

What’s going on here? I thought everyone knew that daycare was a good thing!

Well, it can be. There’s certainly compelling evidence to suggest that at-risk children can benefit greatly from high-quality early childcare education. But going from there to concluding that any sort of daycare will benefit any child is a leap of faith that should be checked against the data.

Why are we just hearing about this now? Hasn’t anyone else studied the Quebec daycare model before?

Certainly. For instance, we’re pretty sure that it’s led to an increase in female labour market participation. That may be enough for some people to declare it a success, but it’s not completely unreasonable to suggest that an evaluation of a child care program should also involve an examination of how well it cares for children.

How do you measure that?

One way is to look at cognitive skills, which corresponds to the ability to learn and understand new information. These are generally measured by looking at test score performances. Available evidence suggests that when early childhood education does lead to better cognitive outcomes, these gains erode over time. After a few years, there’s no distinguishing between a child whose performance was improved (or worsened) by daycare and one who did not go to daycare.

Anything else?

Well, there are also non-cognitive skills, and this is the terrain where the debate is being conducted. It’s hard to provide a precise definition of what “non-cognitive” skills are, but they include things like self-discipline, industriousness and emotional stability. Poor non-cognitive skills manifest themselves in disruptive behaviour and can make it difficult to function in society. It also turns out that this is where the durable effects of daycare show up. Most of the long-run benefits to the famous Perry Preschool intervention on the 1960s can be attributed to the increase in non-cognitive skills that it produced, including a reduction in criminal activity later in life.

How do you measure non-cognitive skills?

This study uses data from the Canadian Community Health Survey, the Canadian Health Measures Survey and the National Longitudinal Study of Children and Youth, and looks at the responses to questions on self-assessed health, life satisfaction and mental health. It also looks at criminal behaviour, using data from the Uniform Crime Reporting Survey.

Is this the first time someone has looked at the Quebec daycare model this way?

No, indeed. In this study, the authors – Michael Baker of the University of Toronto, Jon Gruber of the Massachusetts Institute of Technology and Kevin Milligan of the University of British Columbia, henceforth BGM – are revisiting previous work linking the introduction of universal daycare to a worsening of non-cognitive skills. The earlier study may not have made the headlines that this one is making, but it certainly didn’t go unnoticed in the economics policy community. It was published in the Journal of Political Economy — one of the top three journals in economics — in 2008, and it received the Purvis Prize as the best study in Canadian public policy.

Still, this is a controversial finding, to say the least. Did anyone else try to reproduce those results?

Sure; the ones I’ve found are listed at the end. As far as I can tell, no one else has drawn significantly different conclusions from the data.

So how do BGM go about making the statistical link between the introduction of the daycare program and a worsening of non-cognitive skills?

The basic statistical methodology is adapted from that used for medical trials. There’s a “treatment” group that receives the new treatment, and a “control” group that doesn’t. In this case, Quebec is the treatment group and the other provinces are the controls. Changes in non-cognitive outcomes in the treatment group that can’t be explained by changes in other observable factors are ascribed to the treatment.

What are these “observable factors”?

Things like age, parents’ ages and education, family size and composition. They also capture year effects and province effects, as well as year-specific provincial effects.

You said the treatment group was “Quebec.” Does this mean the entire population, including the large number of kids who didn’t go to daycare?

Yes.

Isn’t that a huge problem? If you want to estimate the effects of Quebec’s universal daycare program, shouldn’t you be looking only at the kids who used it? Isn’t it a fatal error to lump both participants and non-participants together like that?

No, not at all. Just the opposite, in fact. In a statistical analysis, you have to treat as random the things that are actually random. The families who made use of the daycare program were not randomly selected; they chose to participate. This means that participants are a self-selected sample, not a random sample. Remember the fuss about the Conservatives’ decision to make response to the long-form census voluntary? This introduced a self-selection bias to the National Household Survey that wasn’t present in previous, mandatory censuses.

Okay, but how does lumping everyone together solve the self-selection problem?

Since thinking about participation in the program as the treatment introduces a self-selection bias, BGM interpret the program itself as the treatment. This is also how the problem is dealt with in clinical trials where subjects’ compliance with the treatment cannot be monitored; they call it intent-to-treat analysis. BGM interpret making available a universal program that families may or may not use the same way that a clinical trial would interpret making available a treatment program that subjects may or may not be following.

We’ve been told that BGM find that the introduction of universal daycare in Quebec has led to a decrease in non-cognitive outcomes. But have they really proven the existence of a cause-and-effect link? Aren’t there other interpretations of the evidence?

Yes, of course; statistical evidence can never‘prove causal links. In principle, there are an infinite number of causal relationships that can fit a finite number of facts. For all we know, these findings could be just part of an alien species’ 10-million-year project to calculate the Ultimate Question of Life, the Universe and Everything and have no other significance beyond that. But some causal relations are more plausible than others.

But there must be a plausible explanation that doesn’t involve concluding that the daycare system worsened non-cognitive outcomes.

Many have pointed out that a much-criticized educational reform occurred around the same time the daycare program was introduced, so maybe BGM are simply picking up the effects of this reform, and not the effects of universal daycare. The mechanics of this link haven’t yet been specified, and it’s not immediately clear to me how the educational reform could have worsened non-cognitive skills while having little effect on cognitive test scores. But the possibility that someone will be able to make that link certainly can’t be ruled out at this point.

Let’s suppose that universal daycare is in fact the problem. What’s the solution? How can non-cognitive outcomes be improved?

This is probably the most important question, and BGM offer no guide to answering it. They’re estimating what economists would call a reduced-form model; they don’t have a structural model that maps inputs such as the curriculum, funding and how daycare providers are trained into non-cognitive outcomes.

So this isn’t the last word on the debate about universal daycare, then.

No, it isn’t. But at least BGM have moved the debate closer to where the data are.

 

Other references on the Quebec daycare program and non-cognitive outcomes:

Kottelenberg, Michael J. and Steven F. Lehrer (2103), “New evidence on the impacts of access to and attending universal childcare in Canada” Canadian Public Policy 39 (2), pp. 263–286

“Overall, our results are in line with BGM’s findings that the introduction of the Quebec Family Policy led to a significant decline in child, parent, and family outcomes. The initial estimates in BGM are robust to inclusion of additional years of data and concerns regarding multiple testing. Estimates of the causal impact of childcare attendance are also negative, with the notable exception of the motor-social development score. On average, attending subsidized childcare leads to a significant increase in this score. Further, our results suggest that the negative impacts reported in BGM are driven by children in families who decided to attend childcare in response to the implementation of the policy.”

Kottelenberg, Michael J. and Steven F. Lehrer (2014), “The gender effects of universal child care in Canada: Much ado about boys?” Mimeo, Queen’s University

“This paper extends earlier research evaluating the developmental consequences of the Quebec Family Policy and documents significant gender differences in the estimated policy impacts. Comparing weighted and unweighted estimates allows us to conclude that for the subsample of girls, many of the negative impacts of access to child care are driven by changes in the composition of the sample who attends child care. In contrast, for the subsample of boys we find that access to child care leads to increases in scores for indexes measuring emotional anxiety, hyperactivity and inattention, separation anxiety and physical aggression that are invariant to changes in sample composition. Once subsidized child care is made available, gender gaps in nearly every behavioral problem and developmental score increase.”

Saguil, Kirsten and Christine Neill (2015), “Universally Subsidized Day Care and its Effects on Youth Crime Rates: Evidence from Quebec”, Mimeo, Wilfrid Laurier University. Paper presented at the meetings of the Canadian Economics Association, June 2014

“We find no statistically significant or economically large decrease in youth crime rates in Quebec relative to the rest of Canada. Indeed, if anything, there is some evidence that crime rates increased. This paper therefore adds to the cautions issued by researchers studying only the short-run effects of the program, that using evidence from “a limited set of experimental preschool interventions” (Baker et al., 2008: 740) with very specific target groups to develop broader programs available to the entire population may be unwise”.

 


 

How to understand that controversial study of Quebec daycare

  1. Similar results were found for a multi-decade longitudinal study of the Swedish childcare system.

Sign in to comment.