Lessons from a year of ‘Science-ish’

It’s been a year since Science-ish launched with the aim of scrutinizing the news coverage of health topics, and holding politicians, opinion leaders and journalists to account for misusing or misrepresenting science. The modest goal of the blog was, and remains, to help readers wade through the evidence on a given subject and get a sense of what the science actually shows.

So what has Science-ish learned in 12 months of fact-checking the reporting on everything from the health effects of asbestos, to whether the benefits of urban cycling outweigh its harms, how and if cancer screening will save your life, the “cures” for autism, dubious mental health statistics, and just about every health story in between? Here are five key lessons for telling science from science-ish:

1. Mind the gap between research and reporting
In talking to scientists all year about the health headlines and the way journalists use evidence, Science-ish has noted that there can be a great, gaping gulf between research findings on a given topic and what the media report. This is worrying for a number of reasons. For one, many people live by the health advice they read in the media. Do you need more vitamin D? What about the best diet for weight loss? If we journalists get it wrong, public health may suffer. And, let’s face it: many busy policymakers and even doctors rarely get around to reading primary research. They rely on newspapers, magazines, and blogs to inform their decisions. Since a revolution in science journalism is unlikely to happen anytime soon, the best we can do is read with a skeptical gaze. As well, question sources—especially for decisions that matter to your health.

2. Scientists, journalists and policymakers are all equally guilty of bastardizing science
Scientists, journalists and policymakers are equally guilty of misusing and abusing research evidence for their own ends or out of ignorance. Though scientists like to blame journalists for mucking up their work, they can be equally guilty of making trumped-up and inaccurate claims about their own science. Meanwhile politicians might promise “evidence-informed decisions” even though the weight of research on a given policy is, in reality, stacked up against them. Science-ish has documented the full spectrum of evidence abuse, from rotten claims about sugar and salt, to how much doctors in your province take home. This brings us to the next point…

3. Beware the ‘experts’
Looking at the full spectrum of science-ish statements this year—made by such public figures as Dr. Oz and the Ontario Health Minister—it became apparent that it’s essential to question who we trust when it comes to health information. We in the media too often give equal and uncritical voice to both worthy and dubious sources. We put these experts forward and encourage readers to eat up their advice like a probiotic yogurt. But we should take a moment to reflect on such questions as: Why do we believe Playboy model Jenny McCarthy when she talks about childhood vaccines? Is Oprah really the best source for advice on how to manage menopause? Is a naturopathic doctor who peddles the improbable promise to “bust your belly-fat” really going to help you slim down? The answers seem obvious; we forget to ask the questions.

4. Context is everything
Reading the headlines about the latest diet studies can be a frustrating and confusing exercise: one day it looks like coffee will make you live longer, the next day, you hear it’ll kill you quicker. Ditto chocolate, red wine, meat, and a litany of vitamin supplements. The problem here is simple: these studies are often reported in isolation from the body of evidence that exists around them. Health researchers have long known that when you take studies out of context, they can be misleading. For example, though one small study of 100 people may show that coffee reduces the risk of cardiovascular disease, you get closer to the truth by looking at comprehensive reviews of the evidence regarding that question. One secret few other than policy-wonks know: there are academics who dedicate their lives to doing this kind of “secondary research” known as “systematic reviews” or “meta analyses.” Seek them out. They are incredibly useful and informative, and they will save you from the “chocolate-will-set-you-free-then-kill-you” cycle. They also reflect the fact that science is a process: it’s evolving and answers move along incrementally.

5. Not all evidence is equal
A journalist friend once remarked that if a study is peer-reviewed and published, it must be trustworthy. But just because a study is published doesn’t mean it’s worthy or good. There are many ways research can be flawed—from poor design, through the introduction of bias by meddling funders, to ghostwriting. And even when the aims are honest and the methodology impeccable, it’s important to know that different studies serve different purposes and all of them have their limitations.

In observational studies, unlike experimental trials, scientists look at phenomena in a given population, such as people who eat olive oil versus those who don’t. This type of research can show an association between two factors, but not prove that one factor caused the other, or vice versa. This differs from, say, a randomized controlled trial, which is a scientific experiment in which an intervention—a drug or lifestyle change—is introduced in one group but not in another (a control group). Since the only difference between the two, randomly selected groups is the intervention, if they have different outcomes at the end of the trial, researchers can more firmly conclude that this was the result of the intervention.

Likewise, anecdotes are powerful and important—they give life and colour to news stories—but they are a relatively weak form of scientific evidence. A person’s story can change policy or public opinion, and sometimes it should. But it may not be representative of a given population or wider problem. Besides, anecdotes can be cherry-picked.

Science-ish believes in the relationship between knowledge about health information and better health and policy outcomes. One of the best ways to bridge the gap between research and reporting is to be a more careful and discerning consumer of health information. When you spot a science-ish claim, don’t be afraid to hold us fallible journalists, scientists, and politicos to account.

Science-ish is a joint project of Maclean’s, the Medical Post and the McMaster Health Forum. Julia Belluz is the associate editor at the Medical Post. Got a tip? Seen something that’s Science-ish? Message her at [email protected] or on Twitter @juliaoftoronto

*This post was adapted from a talk at the Canadian Association for Health Services and Policy Research meeting entitled “Evidence-Based Health Reporting: Checking the Latest Headlines Against Research Evidence and Holding Leaders to Account for Science-ish Statements” by Steven J. Hoffman (Presenter), Julia Belluz (Presenter), John N. Lavis, Michael Wilson, Donna Ciliska, Maureen Dobbins, Gordon Guyatt, Brian Haynes.