Science-reporting smell test of the week

Colby Cosh on how a study linking abortion and depression was grossly misinterpreted

by Colby Cosh

Here’s the lede of a science story from Saturday’s Winnipeg Free Press:

WINNIPEG — Depression and substance abuse plague about half of American women who reported having an abortion, according to a new University of Manitoba study.

The study, published in the current issue of the Canadian Journal of Psychology, suggests there’s an association between mental disorders and abortion…

Eager to investigate this shocking headline claim—the Edmonton Journal, picking up the story, literally gave it the headline “Depression or drug abuse found in half of women who aborted”—I set out to find the study. This presented something of a problem, since there has not been a “Canadian Journal of Psychology” since 1993. I spent a little while rifling through Canadian Psychology and the Canadian Journal of Experimental Psychology until a helpful reader on Twitter clued me in. Yes, you guessed it: it can be found in the Canadian Journal of Psychiatry. First place I should have looked, really.

That’s an understandable mistake. It’s a bit more of a problem that the first sentence of the article—an article that includes a warning from the lead author to the effect that it is “important the study is not misinterpreted”—is totally false. Because of, y’know, misinterpretation.

The paper, entitled “Associations Between Abortion, Mental Disorders, and Suicidal Behaviour in a Nationally Representative Sample”, does what it says on the tin: the data are taken from interviews with a demographically representative subset of the U.S.’s National Comorbidity Survey Replication project. It is hard to know what numbers the reporter added or multiplied or pulled out of a hat to reach the conclusion that “Depression and substance abuse plague about half of American women who reported having an abortion.” (I spoke to the lead author of the study, and she can’t figure it out either.) But a good guess would be that she looked at this section from the article’s main chart—

—and simply added together the estimated lifetime incidence of depression among women who had had an abortion (29.3%) and the lifetime incidence of substance-use disorders (24.6%). It will probably have occurred to you that there might be some overlap there between depression and substance abuse, which go together like poached eggs and hollandaise. You don’t need a Ph.D. to know that the depression group is likely to contain almost all of the women in the substance-abuse group.

And this naïve math (which is hardly attributable to a failure to grasp hyper-advanced statistics) is compounded by the wording of the offending sentence, which doesn’t say that “some percentage of abortion recipients have, at some point before or after getting an abortion, experienced depression or substance abuse or both.” It uses present tense, unjustifiably implying that all the women in question are plagued by both problems now.

This mess is already being picked up, “carelessly” garbled even further, and circulated around the globe by pro-lifers, despite the personal entreaties of the scientist who helped the newspaper with its reporting and the many, many methodological and interpretive caveats in the original study. This kind of thing is exactly why a lot of scientists hate talking to reporters. Nor does it make sincere research into therapeutic abortion any easier. The UM study can’t be used to attribute psychiatric morbidity to abortion, but it could be used by fair-minded pro-lifers (let’s assume for the sake of argument that there were some) to raise questions about abortion’s place in our society and argue for a research program.

Oh, I know: we’re a hundred years away from that kind of discussion being possible. But the inadvertent propagation of urban legends only pushes that day further into the future.

Science-reporting smell test of the week

  1. Thanks for confirming, Cosh.

    PS, my misattribution above was accidental, and by the time I realized it, someone had replied (and I thusly couldn't edit my own post). My bad.

  2. Keep doing this please. Thank you and well done.

  3. I'm feeling lazy. Wherry, does the study or article establish any relationship (associational, correlation, or causal) between abortion and mental health issues in the immediate aftermath (let's say…6-12 months) of abortion?

    • Cosh?

      From reading the abstract, there seems to be a somewhat greater incidence of some mental health disorders amongst women who have had abortions, as compared to the general population. However, the abstract notes that there is some role played by violence, which might skew things a bit (as I read it, violence couldn't be controlled for).

      I'm not sure the general population is the ideal control group, though, as abortions are likely not spread evenly amongst all of us (and I can't get to the main study, so apologies if I'm wrong). I'd like to see such a study control for socioeconomic status, ethnicity, region, etc., and compare the post-abortion health issues with other women from the same social subsets that disproportionately have abortions.

      As I think of it, it would be most interesting to see a comparison between women who've had abortions and those who either a) simply had babies, or b) had unwanted children and may have considerd abortion as a possible option. If those subsets had better outcomes, that might be more telling.

      • The way to put it is "there seems to be a somewhat greater incidence of some mental health disorders amongst women who have had abortions OR WILL AT SOME POINT HAVE THEM". Not only is there no establishment of causal precedence, the table depicted doesn't establish time precedence, either.

        The dataset here is pretty large (in the neighbourhood of 3,300), but probably not large enough to allow for socioeconomic sub-analysis. However, the study does make a suggestion very like the one in your last paragraph–that women who carried unwanted children to term would make for a suitable control group in future, stronger studies of psychiatric comorbidity from abortion.

        • The table might not, but the abstract states:

          " We calculated the percentage of respondents whose mental disorder came after the first abortion."

    • I haven't read the article, but the chart just shows a correlation between abortion and lifetime experience of mental disorders. Without wanting to sound judgmental, I would think that's not very shocking, as people with perfect happy lives usually don't end up having to have abortions and people with totally screwed up lives probably end up having more abortions. Or, to put it another way, it doesn't surprise me that people with pre-existing substance abuse problems have higher abortion rates than people who are totally clean (which, as I read it, is consistent with the phrasing here).

      • Well, the chart shows "relationships" and doesn't define whether it's an associational, correlation, or causal relationship.

        I'm fairly certain that it's not a causal chart (there'd be other data represented, were this the case). But regardless, we'd all do well to heed XKCD on this one.

      • Well, the chart shows "relationships" and doesn't define whether it's an associational, correlation, or causal relationship.

        I'm fairly certain that it's not a causal chart (there'd be other data represented, were this the case). But regardless, we'd all do well to heed XKCD on this one.

  4. This is bizarre, it's one thing for a reporter to copy a few things out-of-context from the abstract of a scientific paper. But it appears the reporter actually chose to read the paper and look at the tables and then grossly misinterpret the study. So that means the reporter is either: A) Completely making it up, or B) Really, really dumb.

    • Yours is the first comment thus far to address the substance of the matter … poor (or, in the worst case interpretation, malicious) journalism. Since we clearly cannot use the study in question to make broad assertions about mental health's connection (if any) to abortion, we should stick to the matter at hand:

      1. How did these articles clear the editorial departments of these news organizations?
      2. When can we expect a front page retraction?

      • Not just that, but are there any repercussions to the individual reporter for producing something like this?

      • On #2, I think the answer is "never." There may be a "correction" in small print on an interior page, but no hope of a front page retraction.

    • It's the age-old question: stupid or evil?

      I find myself wondering that more and more these days.

      • It's entirely possible to be both.

        • And thus begins the complex conversation about whether or not there's a causal link between the article's premise and the author's stupidity, or between the article's premise and the author's capacity for evil, or both, or neither… and we're right back where we started.

  5. It's not causal. As the title of the paper suggests, these are merely associations.

  6. Terrific. Now the results of the study, misquoted by prolifers, will be completely ignored on the grounds that they're misquoting it.

    Which is too bad, because looking at those two columns of percentages affected by psychiatric disorders, it's striking that every single one is higher, sometimes by a factor of 4 (!), for women who've had an abortion.

    • Some of the associations are statistically significant, even when other variables are controlled (in particular the substance abuse associations). As someone who reads data frequently, this tells me more digging is required, because there's something to it.

      • Though, again, the fact that more than half of the increase in substance abuse was seen prior to first abortion (based on the statement made in the abstract that "For all disorders examined, less than one-half of women reported that their mental disorder had begun after the first abortion"), we may just be seeing that women with substance abuse problems are more likely to have abortions than that women who have abortions are more likely to develop substance abuse problems.

        • It's possible, but we can't draw those conclusions based on associational data. Associations mean that two things happen in the same dataset at roughly the same measurement interval.

          • Agree. By saying that we "may" be seeing something, I meant it as "we shouldn't necessarily think x causes y because y may be causing x or maybe z is causing x and y."

    • …. all of which, or course, has nothing to do with the primary abortion debate (i.e. whether abortion should be permitted), although it has a great deal to do with the secondary abortion debate (i.e. whether the public should fund it).

      • Prior to 1988, abortion was legal in Canada … it just wasn't available to women in a manner that measured up to the requirements of the Charter (DOes your dad have a friend on the hospital Therapeutic Abortion Committee? No, too bad.).

        Also, I believe you misstate the current abortion debate. Answers to your 1 and 2 have been fairly well established (Yes, and Yes … though public payment is a moot point in many areas of the country which lack facilities/providers). The current question(s) seems to deal more with how the number of unwanted pregnancies and abortions might be reduced.

    • (See my note above, you're mistaken. Partly my fault.)

      • Nah, my fault – sorry to waste your time,

        • Well, this little thread became uninformative through the deletion-by-user. Sean, could I ask that, absent libellous or thoroughly garbled or misplaced text, the original text should be allowed to stand, even if it requires a subsequent oops-here's-where-I-goofed little note?

          • I tried to delete it minutes after posting, and before I knew Cosh had replied. I never delete posts once responded to, if I can help it.

            For the record, I basically referenced my earlier post, noting that the abstract claimed to be presenting data from post-abortion populations. But that wasn't fully correct (as the messy discussion upstream demonstrates!).

            Goodness knows, if I started deleting all my goofs/errors/etc. there'd barely be record of my presence here. :)

  7. I was restricting my conclusion to women actually questioned in the study, and they're separated according to whether they've had an abortion as I understand it. I'd hesitate to draw conclusions about the general population.

    • There's another column of odds ratios produced after controlling for "lifetime experience of violence", and in that one some of the associations with psychiatric conditions, though not all, lose significance. I tried to show only the part of the chart that could be justified by the need to make my main point about the news coverage.

    • Both abortions and mental disorders in that chart are based on lifetime occurrences, and the abstract makes clear that over half of the mental disorders occurred prior to first abortion. So, any causal arguments are highly attenuated. That said, you could use this to try to develop public policies to help women already suffering from depression, substance abuse, etc., so that they did not find themselves in situations where they felt they needed abortions.

    • Cosh's point is that the chart doesn't indicate when the psychiatric disorders reported occurred relative to the abortion, making the fact that every single one of the report rates is higher among women who have had abortions all but meaningless. This isn't the case, but just looking at the numbers in the table it would be theoretically possible that they represent a group of women among whom every single woman reporting having had a psychiatric problem had also had an abortion at some time in their life, while NONE of the other women who'd had abortions ever had a psychiatric problem develop later in life if they didn't have one before the abortion. There's no way to differentiate in the chart between the woman who got hooked on coke at 14 and then had an abortion at 16, and the woman who had an abortion at 14 and then got hooked on coke at 16, and the woman who got hooked on coke at 14, got off of coke at 16, and then had an abortion at 26, and the woman who had an abortion at 16 and then lived a perfectly happy life for 40 years until getting hooked on coke at the age of 56. All of which makes speculating about causation pretty meaningless.

      Speculating that A might have a causal relationship to B is always dangerous and often inaccurate. However, if you don't even know whether A or B happened first chronologically, it's downright meaningless, isn't it?

      • Yes, which is why I was careful to avoid making any such speculation.

        Women in the study who had had an abortion were far more likely to also have had every kind of psychiatric problem listed. That's the only conclusion I drew. Yet it is interesting and may have relevance to the (secondary) abortion debate. Period.

        • Perhaps, but as was pointed out in discussion above, it would have to be tested against unwanted pregnancies as a control group, not the general population.

        • The entire study will be given no credibility by those involved in the abortion debate.

          I guess my point would be that as the study doesn't really tell us anything about abortion that's meaningful, that makes it essentially IRRELEVANT to those involved in the abortion debate, so it doesn't matter whether or not those involved in that debate give it any credibility. The study can be entirely credible and still have nothing whatsoever to add to the abortion debate. I was looking at reports about that study the other day on how our brains react to race, which seems to me to have been an entirely credible study. I don't think THAT study has anything to add to the abortion debate either, but its relative credibility is irrelevant to that calculation.

          • Fair enough. My point was that if the study has some bearing on the general population (an assumption I am not currently willing to make) and if a correlation between abortion and subsequent neuroses was shown in the study (another assumption I am not currently willing to make) then it might have some bearing on the abortion funding question. Yes, tenuous at best at the moment.

            Anyway, as I said elsewhere on the thread, all this is completely tangential to the primary abortion debate, the one most pro-lifers actually care about, which is whether abortion should be allowed at all. That question has nothing to do with possible health effects and everything to do with whether abortion is merely infanticide by another name.

          • if a correlation between abortion and subsequent neuroses was shown in the study

            Just wanted to add that I don't think it's even metaphysically POSSIBLE for this study to show that. Of course, if ANOTHER study showed that, then that would be an important point, and very relevant to the abortion debate. However, I don't think this study actually collected the data necessary to make that connection, even if one went back through all the data with that explicit intent. It's not that that connection can't be true, or can't be verified, it's that this study simply does not contain data that can test the veracity of that hypothetical connection.

        • Women in the study who had had an abortion were far more likely to also have had every kind of psychiatric problem listed.

          Again though, as there is no distinction between women who had said problems before the abortion and those who didn't have those problems until after, so what? If a study tracks lifelong occurrences of coffee drinking, and also tracks lifelong occurrences of heart attacks, but doesn't track whether the subjects were coffee drinkers who later had heart attacks or heart attack victims who later became coffee drinkers, of what possible relevance would it be to the "Does coffee drinking cause heart attacks" debate? Does it retain the same relevance if you find out that none of the people surveyed had ever had so much as a milligram of coffee before they had their heart attacks? Or that people who drank coffee 15 times a day until they had their first heart attack are treated the same by the study as people who stopped drinking coffee 30 years before their first heart attack, who are themselves treated the same as people who started drinking coffee decades after suffering a heart attack completely unrelated to coffee? The fact that there is a high correlation between two facts in a study is interesting. Maybe even fascinating. It is however, more importantly, COMPLETELY IRRELEVANT to any discussion of the actual relationship between those two facts.

          Discovering that there is a high instance of correlation between chickens and eggs doesn't do anything at all to add to the argument of which came first, so I fail to see how it's useful in that context. To my mind, this is a perfectly credible study that adds ABSOLUTELY NOTHING WHATSOEVER to the abortion debate, because the study itself tells us nothing about abortion.

  8. And goes on say that less than half.did. Which still sounds like higher than average (and hardly surprising).

    • Definitely higher than average in some instances (based on the data here). The substance abuse numbers are the huge standouts, with 24.6% versus 6.9%. If over half of the substance abuse problems were preexisting, you would be looking at a population of whom at least 12.4% had experienced substance abuse issues, more than double the *lifetime* total for the "no lifetime abortions" group. That's a pretty huge difference from the control group and suggests that any causal relationship is substance abuse –> abortions, not abortions –> substance abuse (though, even that is reading into the study in a way that would probably infuriate the authors).

      • We need to be *really* careful about constructing narratives around this sort of data, or even guessing at causality.

        However, it wouldn't surpise me to learn that addicts a) have more unplanned/unwanted pregnancies than non-addicts, and that addicts have fewer support systems (family, social), less stable relationships with fathers, poorer incomes, and greater pessimism about the future than non-addicts: thus making this group more likely to procure abortions.

  9. Yes, they did, but those data are in a separate table. The one above includes both "befores" and "afters". For most of the conditions it turns out to be very close to 50% before, 50% after.

    • From the abstract: "For all disorders examined, less than one-half of women reported that their mental disorder had begun after the first abortion." So, in the worst possible cast 49.9% of mental disorders occurred *after* the first abortion, while 50.1% occurred *before* the first abortion.

  10. (I obviously can't publish the whole study here independently but anyone who wants to have a look at the PDF can drop me a line at colbycosh@gmail.com.)

  11. Damnit, how can lump you into the "crappy writer" category when you keep doing this "responsible journalist" thing.

    Health and science reporting sucks donkeys in this country. Thanks for the windmill tilt on this one, Cosh.

    • I think reporting on subjects that require any kind of specialized knowledge usually aren't very well done.

      And while a decent job, this is pretty low hanging fruit. The "ABORTIONS CAUSE MENTAL ILLNESS" cries from the anti-abortionsists have long seemed pretty questionable just by giving it a good thought. Always nice to see someone provide the paper work, though.

      • Maclean's is sometimes a bit guilty of this too. Many times I've followed the link of a "Need to Know" story about a new study ("Hot Dogs Cause Impotence!" "Americans Laugh More Often!" kind of stuff…) , only to find that the basis of such claims are studies with little or no representative merit (40 Midwestern college students, or some such thing).

        I don't think it's the end of the world or anything, but it's probably not good for general scientific and skeptical thinking.

        • I have to admit, there are sometimes Need To Knows that kinda make me go "This isn't knowledge and I don't need it". :)

          • I agree. "Need-to-Know" entries that included Tiger Woods or anything like hey-who-knew-women-like-sex-too or whatever reeeeally did not live up to the title.

            Suggested new category title: "Hmm Department" or something. Or split it up into a "Headline Digest" for breaking stuff that actually is, you know, news, and a "Now for Something Completely Different" for the fluffy stuff.

    • I'm glad I'm not the only one who feels that way when he reads Cosh. Half the time I read his posts I'm left enraged, but the other half I'm left feeling thankful there's someone writing about whatever the subject of the post is. It's infuriating, but at least it's also challenging and interesting.

  12. Kang:"Abortions for all!"
    Crowd:"Boo!"
    Kang: "Very well, no abortions for anyone!"
    Crowd: "Boo!"
    Kang: "Abortions for some, miniature American flags for others!"
    Crowd:"Yay!"

    Anyhow, I'm curious as the etiquette of this, Cosh. Does one (as in you) directly contact the Winnipeg Free Press about this or is it customary to wait and see if they print a rebuttal/correction ?

      • That letter writer is wrong on his most important point (the sample used does, in fact, contain interviewees who never suffered psychiatric disorders–that should be apparent just from Column 1 above) and completely ignores the mishandling of the data by the newspaper itself. The study is a retrospective one, and that's a weakness, though one he overstates rather hysterically. Not a very helpful contribution, I'm afraid.

    • Honestly, I'm not really concerned about that. More-than-sufficient harm's been done.

  13. Just so I'm clear:

    A study titled “Associations Between Abortion, Mental Disorders, and Suicidal Behaviour in a Nationally Representative Sample” actually finds such associations to exist and, with respect to virtually every disorder and/or behaviour studied, finds them to occur at a considerably higher rate among women who have abortions than women who don't. Some Winnipeg journalist reports this, but through ignorance/malfeasance/incomprehension, overstates the case. Cosh is outraged and bemoans the fact pro-lifers are promulgating the flawed analysis even as we speak and pushing the day an informed and civil discussion about abortion can occur to sometime in the next century.

    I think I've got it and eagerly await Cosh's companion piece on how pro-choice groups' proclamations that there is "consensus" in favour of Canada being utterly devoid of abortion laws and the funding by Canadian taxpayers of illegal foreign abortions aren't particularly conducive to civil debate either.

    So

    • And your point is…..?

      • I think the writer's point is that Cosh's intelligent article and the many comments in this thread, while valid, focus on semantics around the abortion discussion in this country, rather than on the underlying issue, which tends to be generally counter-productive in the long run….it's like putting another coat of wallpaper on a plaster wall that is falling down because we cannot acknowledge the state of the wall.

    • I'm SO MAD that somebody corrected wrong information! Grrr! Unfair!

  14. OK. Thanks!

  15. I think this incident makes a good case for teaching statistics to a wider range of people. Unfortunately, most people do not learn much statistical theory, unless they go to graduate school (undergraduates learn some basics, but only a few get broad details). In fact you could probably produce better citizens by teaching people about statistics, than teaching them about civics. Being able to interpret data is essential to understanding any public policy debate. Without those skills, people are left to trust in one set of authorities or another. The Internet has only heightened the need for better interpretation skills by increasing the amount of data (much of it bad) out there.

    It is not that hard to interpret a regression table and many statistical concepts are reasonably intuitive. New statistical software, like Stata, are also making the application side of things easier and easier to do (this has some pitfalls, but in terms of instruction could be quite helpful). Moreover, there is a most definite need for statisticians in a service-based economy. So huzzah for an R-squared in every pot!

    • The problem isn't stats, it's logic. Even if people know how to do a regression they're not generally going to have access to the raw data on which each study is based. Train people in critical thinking, however, and they understand that studies are not infallible authorities, but rather one opinion among many based on a set of numbers that may or may not show what the author claims they do show….and that reporters, who rarely have training in any rigorous discipline, are even less likely to summarize the results correctly.

    • In fact you could probably produce better citizens by teaching people about statistics, than teaching them about civics.</>

      Won't anyone think of the lotto employees? Government revenue thrives on widespread citizen innumeracy.

    • In fact you could probably produce better citizens by teaching people about statistics, than teaching them about civics.</>

      Won't anyone think of the lotto employees? Government revenue thrives on widespread citizen innumeracy.

  16. I'm just glad there's a few less children being raised by mentally ill and/or drug-addicted mothers.

    • That, friend, is a sick, sick comment given that the mothers haven't been cured and the children haven't been transferred to healthier mothers…the children have just been terminated.

  17. Also of note, the study didn't ask about whether the abortion was elective or medically necessary, and incidences of the various psychological and substance abuse problems were assessed by "layperson interviewers… [using] guidelines from the World Health Organization to assess such mental conditions as major depression, suicide, alcohol abuse and panic attacks".

    • A question to anyone who may know. Does genuinely "medically necessary" even show up on the radar anymore, within the overall total?

  18. I don't think it's metaphysically possible for you to know what is metaphysically possible for this study to show, given that the study is not yet available online.

  19. I agree : THAT study has nothing to add to the debate about abortion and mental health. I wonder why the authors have botherd to do it.

    The best study on this question remains the one of Gilchrist (Gilchrist AC. et al. "Termination of Pregnancy and Psychiatric Morbidity" Brit.J.Psych. 167:243-48, 1995). Because this study DOES compare thousands of women who had an abortion with as many who carried an UNPLANNED pregnancy to term, with a follow-up of 10 years: there was no difference in their mental health.

  20. After having read the article, I would like to post here a sentence out of its Conclusion: "Our study does not support a unidirectional relation between abortion and mental disorders."

    Furthermore, it is evident that there is a strong reporting bias in the 2001-03 US survey used for the study. It is not possible that of women aged 60 and more, at the time of the survey, only 5% had had an abortion in their lifetime (they were in their 20ies before the pill was established as a contraceptive!).

Your email address will not be published. Required fields are marked *