The Oprah effect and why not all scientific evidence is valuable

Some studies are more equal than others

One of the inspirations behind Science-ish was the seemingly endless barrage of complaints by friends in medicine regarding the “Oprah effect” on their offices and hospital wards: patients making important decisions about a lifestyle choice or treatment option based on something they had seen on Queen of daytime talk.

Now, the Oprah Winfrey Show is off the air, but the after-effects of her work on childhood vaccination and menopause will surely haunt doctors’ visits for years to come. Of course, other media—before and after Oprah—have a powerful sway over patient decisions. Every day, newspapers dole out advice on how much alcohol and coffee to consume, how best to manage your diabetes, and the benefits of probiotics. New media play a big role in purveying health knowledge, too. In research into YouTube as a source of information on immunization, the investigators found that about half of the videos posted had anti-immunization messages, and the negative videos were more highly rated and viewed more often than those backed by science.

Understanding one simple concept can help protect you from bad health advice: not all evidence is created equally, and so not all evidence should be given equal weight.

There are “evidence hierarchies” in science, and scores of very smart people around the world working out how to appraise evidence and separate the good-quality stuff from the bad. In this useful (and short) article, “How to read a paper,” Dr. Trisha Greenhalgh, a professor at the London School of Medicine and Dentistry in the U.K., explains why a lot of scientific research is rejected by peer-reviewed journals (“a significant conflict of interest,” “the study was uncontrolled or inadequately controlled,” etc.) and of the published works, how to tell whether a study is relevant or valuable.

According to Greenhalgh, there are three questions to ask when assessing a paper:

1. Why was the study done, and what clinical question were the authors addressing?
2. What type of study was done?
3. Was the [study] design appropriate to the research?

On the question of type, it’s important to differentiate between primary research (such as control studies and clinical trials) and secondary research (meta-analyses and systematic reviews). In the media, you often read about primary research, like this jewel from earlier this week: “Study touts new way to spot babies at risk for obesity.” Greenhalgh points to a useful “evidence hierarchy” that ranks the relative weight of research from highest to lowest:

1. Systematic reviews and meta-analyses
2. Randomised controlled trials with definitive results (confidence intervals that do not overlap the threshold clinically significant effect)
3. Randomised controlled trials with non-definitive results (a point estimate that suggests a clinically significant effect but with confidence intervals overlapping the threshold for this effect)
4. Cohort studies
5. Case-control studies
6. Cross sectional surveys
7. Case reports

The basic principle here is that syntheses of evidence (these are papers that apply a systematic approach to summarizing research on a given topic in its totality) generally tell more about the efficacy of a treatment or intervention than one-off studies (which are more prone to bias). This is why syntheses are regarded as the highest form of evidence. One example that illustrates why this type of research can be more revealing than single studies is the evidence on vitamin supplements. In the book Testing Treatments (which you can download for free), Science-ish patron saint Dr. Ben Goldacre points out that, despite all the single studies that concluded that taking a daily antioxidant vitamin was good for your health, systematic reviews on the subject revealed there was no evidence to support the use of these pills for prolonging life. In fact, when the research was taken as a whole, Vitamin A, beta-carotene, and vitamin E were shown to possibly increase mortality! (Goldacre goes on to note that the systematic review “is quietly one of the most important innovations in medicine over the past 30 years.”)

Not so fast, though: some types of research that rank lower on the hierarchy are still useful in answering some clinical questions, sometimes even more so than the more highly ranked types of evidence. Greenhalgh gives the example of case reports about thalidomide, a drug pregnant women used to combat morning sickness, which was withdrawn in the 1960s because it caused birth defects and nerve-system damage. She writes: “A doctor notices that two newborn babies in his hospital have absent limbs (phocomelia). Both mothers had taken a new drug (thalidomide) in early pregnancy. The doctor wishes to alert his colleagues worldwide to the possibility of drug-related damage as quickly as possible.” Here, the case report about the possible side-effects of the drug conveyed urgent clinical information a trial would have taken years to uncover.

In fact, while randomized-controlled trials rank high on the evidence hierarchy, they can have their flaws too. Some things that make for a weak trial include: too few participants to be clinically relevant, failure to blind participants or assessors, or imperfect randomization. There are many ways external forces can bias a trial, as well. One obvious one: drug industry funding. Dr. Joel Lexchin, of York University, has been studying this subject for years. In his decades of research, a key finding that emerged is that industry-funded studies are four times more likely to have a positive result for the sponsor’s drug than independently funded trials. Also, journals tend to feature trials with positive outcomes or big scientific breakthroughs, which means we don’t often hear about failed drugs or interventions.

You’ll notice a lot of reporting on health is generated on the basis of cohort studies, a type of observational study, which look at a group of people over a long period with the aim of testing a possible correlation between a certain lifestyle choice (diet, exercise) and health outcome. (“Do people who live in Mediterranean countries and consume olive oil live longer than people who do not?”) While these studies rank lower on the evidence hierarchy, this is a very useful type of research: observational studies on tobacco are what gave way to the realization that smoking is linked to lung cancer. But the problem with the reportage on observational studies is that correlation is often reported as causation: “eating olive oil will make you live longer.”

Research also tends to be reported without context, in isolation from other similar studies. “If people lived by the prescriptions of study headlines alone, they would likely be eating lots of chocolate one day and then none the next,” Steven Hoffman, assistant professor at McMaster University (and Science-ish colleague), said. “Individual studies are only helpful to a limit.”

So science is messy and reporting on science is messier still. But Science-ish would like to leave you with a useful how-to for deciphering media stories about health from the Harvard School of Public Health. They lay out a few questions to ask yourself when reading about a reported study:

• Are they simply reporting the results of a single study? If so, where does it fit in with other studies on the topic?
• How large is the study?
• Was the study done in animals or humans?
• Did the study look at real disease endpoints, like heart disease or osteoporosis?
• How was diet assessed?

Science-ish would add two questions: What type of study are you looking at? Who funded the research?

These simple questions can go a long way in critically appraising the unprecedented pile of evidence (studies, research) being generated and reported on. And remember: a lot of research out there is simply junk, despite the import it is given in the 24/7 news cycle. Another of the esteemed academics who advise on this blog, Dr. Brian Haynes, helped to create a free service sponsored by the British Medical Journal called EvidenceUpdates, which collects reviews from clinical journals and critically appraises them for their relevancy and newsworthiness. The aim of the service is to help clinicians and researchers sift through the junk science out there. Of 50,000 articles from 120 premier clinical journals reviewed by EvidenceUpdates each year, only 3,000 (or 6 per cent) measure up. That means 94 per cent of the 50,000 articles are rejected. That’s a whole lot of crap.

*Thank you to Donna Ciliska, Maureen Dobbins, Gordon Guyatt, Brian Haynes, Steven Hoffman, John Lavis, and Michael Wilson at McMaster University for their invaluable guidance on Science-ish and helping me wade through the evidence every week.

Science-ish is a joint project of Maclean’s, The Medical Post, and the McMaster Health Forum. Julia Belluz is the associate editor at The Medical Post. Got a tip? Seen something that’s Science-ish? Message her at julia.belluz@medicalpost.rogers.com or on Twitter @juliaoftoronto




Browse

The Oprah effect and why not all scientific evidence is valuable

  1. “Study touts new way to spot babies at risk for obesity.”

    Why do so many scientists, and others, refuse to consider genes? Look at baby’s parents and you will know if child will be obese or not.  

    One of my many complaints about living in North America in early 21 st century is how self centred we are and rarely, if ever, consider genetics. We live in era where we are convinced that any ‘flaw’ can be overcome with willpower alone but it’s just not true.

    I have been reading up about salt, for instance, and more/less salt in diet can affect your health – some people need more, some need less but many scientists seem to make sweeping generalizations about what people should be doing

    Human beings are unique, there is no average person, but discussion makes it seem we all exactly the same.

    JAMA  ~ A Twin Study Of Human Obesity:

    Height, weight, and body mass index (BMI) were assessed in a sample of 1974 monozygotic and 2097 dizygotic male twin pairs. Concordance rates for different degrees of overweight were twice as high for monozygotic twins as for dizygotic twins. Classic twin methods estimated a high heritability for height, weight, and BMI, both at age 20 years (.80,.78, and.77, respectively) and at a 25-year follow-up (.80,.81, and.84, respectively). 

    Height, weight, and BMI were highly correlated across time, and a path analysis suggested that the major part of that covariation was genetic. These results are similar to those of other twin studies of these measures and suggest that human fatness is under substantial genetic control.

    • There is some suggestion that problems with addiction to substances has a genetic component and obesity does indicate an addiction to food.  However, what is not explained by genetics is the huge increase in incidence of obesity in north america since the popularity of fast food, the tv and computer and the decline of physical jobs.   If “human fatness is under substantial genetic control” than our ancestors should have been as fat as we are but they were not.  Portion sizes have risen dramatically, people eat out regularily and spend most of thier  time in physically inactive pursuits.  This lifestyle has resulted in what is really a modern era epidemic of obesity and complications related to the condition such as diabetes.  Yes, children of fat people tend to be fat but have you noticed that the friends of fat people are fat as well?  Maybe it is because they spend their time enjoying the same pursuits….inactivity & food.  I am not being cruel because I know how difficult it is to be obese in our culture but I also know that just like a smoker, a drinker and a drug addict, an obese person can change their situation.

      • “ If “human fatness is under substantial genetic control” than our ancestors should have been as fat as we are but they were not.”

        Food is cheaper, and more plentiful, than before so maybe people can now afford to fulfill their genetic destiny to be obese? 

        StatsCan ~ Food In Canada:

        “Canadians are not only spending more on food, but they are also buying more calories. Between 1976 and 2007 the number of calories available per person increased 9% from 3,118 to 3,384 kilocalories. Some of this food however is wasted, and it is estimated that in 2007 only 71% of the calories purchased were consumed.”

        • Tony – You are obsessed with obesity and seem to want to come up with reasons why people should just accept that they are obese and not try to change their circumstance.  If they do as you say, they look forward to many complications including diabetes, joint problems, respiratory problems, emotional problems, mobility issues, issues with being able to travel on planes and other public transportation; possible issues with being able to attend movies and concerts.  People like Richard Simmons have proven that people do not have to accept their future as being morbidly over weight.
          You admitted yourself that there are problems with the diets of many – too much emphasis on simple carbohydrates…when the truth is that fats & proteins are what are successful in providing satiation. 

      • “ I am not being cruel because I know how difficult it is to be obese in our culture but I also know that just like a smoker, a drinker and a drug addict, an obese person can change their situation.”

        I am tall, thin person who has never been one pound over weight. In fact,  I have been trying to gain a bit of weight for at least 10 yrs now because I am slightly underweight if you believe in BMI.  Does that mean I can look down on people who are 5 kgs over weight because I am awesome person who doesn’t even have to try to remain thin?

        Also, my missus is short and I am over six feet. I am constantly having to get things out of cupboards for her, I think I am going to start criticizing her for not trying hard enough to grow taller when she was younger.  

        America’s Moral Panic Over Obesity ~ The Atlantic:

        Q: Over the last five years or so, I’ve noticed that public health efforts about obesity are not just amping up the volume, but exploring increasingly coercive methods to induce weight loss: taxes on junk food, lawsuits against fast food companies (which are basically a tax on junk food), and so forth. Does that match your analysis?

        A: It’s the classic pattern of moral panics. As public concern about the damage being done to the fabric of society by the folk devils increases, increasingly intense demands are made on public officials to “do something” about the crisis, usually by eliminating the folk devils. 

        That of course is the strategy for this crisis. If fat people are the problem, then the solution is to get rid of them, by making them thin people. The most amazing aspect of this whole thing, for me, has always been the imperviousness of policy makers, and even more so people who consider themselves serious academics and scientists, to the overwhelming evidence that there’s no way to do this. 

        I mean, there’s no better established empirical proposition in medical science that we don’t know how to make people thinner. But apparently this proposition is too disturbing to consider, even though it’s about as well established as that cigarettes cause lung cancer. So all these proposals about improving public health by making people thinner are completely crazy. They are as nonsensical as anything being proposed by public officials in our culture right now, which is saying something. 
        http://www.theatlantic.com/business/archive/2009/07/americas-moral-panic-over-obesity/22397/

        • 5 kg overweight?? say what….an obese person is defined as being 25% over their ideal body weight….so unless you are “looking down at a child”, it is highly unlikely that someone 5kg overweight is obese.
          Furthermore…I never suggested you “look down on anyone” because they are obese but then I would not look down on anyone that smokes or is addicted to drugs or alcohol either because I believe that those addictions are very difficult (although not impossible) to kick.
          If we have no hope to give people that are obese or are addicted to other substances than we should be just throw up our hands and say wait until 80% of the population is overweight vs. 50%?

    • There is a socio-economic side to obesity as well. Poorer people tend to be more obese because starch and fat is cheaper that vegetable and protein. Poor people have poor children so obesity is passed on through families, but not just by genetics. It is passed on through low standard of living. There are genetic factors for lots of things, but environment is essential as well. We forget that far too often.

      Obesity is not just an individual problem but a societal one. Until we fix the societal issues the rest will continue to elude us.

    • this is ridiculous youre trying to blame everything on genes,whereas you should be blaming people over-eating and not exercising as much as they should. secondly genetic Councilors have much better things to worry about than obesity,you usually get tested for Huntingtons disease,Parkinsons etc because they are REAL DEBILITATING DISEASES for which there are no cures and you wouldnt want to be passing that onto your offspring. Third of all do you even have any idea how much genetic councelling costs,its upwards of 500 dollars so its cheaper to go to a gym than have a certificate that says youre fat because  of a gene (again genetic inheritance is maybe only 10% of the problem).   So you know instead of slagging off online science forums all day how about you join a gym and start a healthy diet,that might be a bit more usefull.

  2. “There are many ways external forces can bias a trial, as well.”

    Our studies will get immeasurably better when we to start to focus on internal forces that bias a trial.

    WSJ ~ When We See What We Want:

    In 1981, Harvard paleontologist Stephen Jay Gould published “The Mismeasure of Man,” a fierce critique of various scientific attempts to measure human intelligence. Mr. Gould began the book with a takedown of “craniometry,” a popular 19th-century technique that attempted to find correlations between skull volume and intellect ….

    In recent years, it’s become clearer that these psychological shortcomings are a serious societal problem. Because we believe we’re impervious to bias—we’re blind to our own blind spots—we assume that our judgment isn’t affected by financial incentives or personal opinions. But we’re wrong.

    This problem has been most convincingly demonstrated in medical clinical trials. A 2005 study of psychiatric drug trials found that when academic researchers were funded by a drug company, they were nearly five times as likely to report that the treatment was effective. (A similar pattern was found with oncology drugs.) What makes this result so disturbing is that all of these studies were randomized, double-blind trials, which are typically regarded as the gold standard of medical evidence. And yet the financial incentives seemed to decisively influence the data.

    What this depressing research demonstrates is that the only way to get objective data is to have institutions that assume objectivity doesn’t exist. It’s not enough to force scientists and doctors to declare conflicts of interest, because our biases seep in anyway. Rather, we need to do a better job of funding truly independent studies and approaching with extra skepticism those that are not. We should also encourage researchers to make their raw data public, as Samuel Morton did, so that others can check it.

  3. The world needs more people like Frances Oldham Kelsey.

  4. Excellent post, Julia.  But I wonder if I only think that it is excellent because it coincides with my prior beliefs.

    Sometimes I think that humans’ least rational belief is that humans are rational.

    As a great poet said, “a man hears what he wants to hear and disregards the rest.”

  5. I think the biggest issue is that reporters have historically not reported what studies have proven but rather have undertaken to project what the report “might” mean for their readership.  In the baby example at least the headline seems technically correct if obvious.  Very large people often start off growing faster than smaller people.  In the old days (that perhaps never were), respected researchers would work hard to rein in reporters… now they are spin-masters with substantive talent.

    In the fat baby study, the authors go on to pontificate about subjects that are not tested in the study. The lead author, Dr. Taveras not only assumes that the early rapid growth is the result of overfeeding and lack of exercise but also recommends treatment.  This type of grasping for the big publicity win has become commonplace among the elite in the research community to the point where the most prestigious journals are also the least reliable.  (Although they are certainly the best read)

  6. Fantastic article Julia. You were able to effectively summarize an entire health research methodology course in 1500 words. While I am a strong proponent of Evidence Based Medicine (EBM), I think there is also substantial value in presenting EB research in a generally understandable and emotionally appealing way (ie. Oprah Effect). However, when that evidence is misrepresented it leads to an inappropriate impact.
    I would like to point people to http://www.mediadoctor.ca , a service that aims to provide an “objective evaluation of the quality of current health reporting in the
    lay press and a mechanism by which to inform journalists and media
    organisations on the quality of their stories with the view to
    improvement.” Perhaps we need a similar evidence and methods based review process for new media (eg. YouTube videos, Facebook groups)?

  7. Schools can do a great job of helping citizens to understand the real world–parents and children alike. Years ago my school age children were taught Map Reading. What a practical idea, I thought. They are adults now, but all their lives they were proficient about finding their way around new places with the help of a map. Since I don’t have an inner gyroscope, I am still easily confused in new locations. Map Reading would have helped.

    I suggest that schools do the same teaching in evaluating Scientific Research, so that citizens could become critical readers of scientific articles.

    That is the only route to laypeople’s understanding of the ‘Map of Science’. Particularly, in the mental illness world with its sloppy psychologica/sociological research, as well as the fraudulent psychology research by Diederik Stape,a respected scientist (?)  whose research fraud lasted for decades, infecting many research findings, without detection. How do we untangle his lies from truth?

    Yet masses of people  accept these flawed theories and most mental health systems have been established on this faux research. Science-literate readers have long suspected deep flaws in the way research is done in psychology , a  practice that has only lately earned a fragile respectability.

    In our modern world, understanding science is as important as the clean air we want to breathe.    .    

            

Sign in to comment.