When science isn’t science-based: In class with Dr. John Ioannidis

Lessons from one of the world’s most influential scientists

by Julia Belluz

Last week at the Harvard School of Public Health, Dr. John Ioannidis – a Stanford professor and Science-ish hero – told a room filled with Harvard doctors (and one journalist) that they can’t trust most of the research findings science has to offer. “In science, we are very eager to make big stories, big claims,” he opened his lecture, with a mischievous grin. “The question is: are those claims accurate?”

According to Ioannidis, the answer – at least most of the time – is an unequivocal ‘no.’

A compact man in his 40s with stooped shoulders and thinning brown hair, Ioannidis has made a career researching research – or “meta-research” – examining not just single studies but many studies across fields as diverse as disease prevention, neuroscience and genomics. His boyish nerdiness and good nature belie the thorn in the side of science that he has become. For the last 20 years, he has amassed an internationally regarded body of research about all the ways science isn’t actually science-based. For this work, he’s considered “one of the most influential scientists alive.”

At a time when scientific knowledge is being produced at an unprecedented rate and global spending on life sciences research alone has topped $240 billion US, the need for people like Ioannidis – who can take a step back and examine trends, gaps, biases, waste and flaws – becomes more urgent than ever. If science continually fails at self-correction, Ioannidis is the closest thing this field has to a one-man self-correction machine.

In the Harvard class, he gave students an overview of his work and all the ways research goes off the rails. Here are some highlights:

1) Why every diet supposedly causes cancer:

In one of his studies – appropriately titled “Is everything we eat associated with cancer?” – Ioannidis and a co-author randomly selected 50 ingredients from recipes in the The Boston Cooking-School Cook Book. They then looked at whether those ingredients were associated with an increased or decreased risk of cancer. At least one study was identified for 40 of the ingredients – from bacon and bread to sherry and sugar – and most of the claims made in the studies contradicted each other or were based on weak evidence. “Most of the ingredients had results on both sides, positive and negative,” he said, making the point that many studies about cancer and nutrition are poorly designed. There were studies to support just about every claim on the popular topic – and many of them are too good to be true. “With one more serving of tomatoes,” he told his class with a smirk, “half the burden of cancer in the world would go away.”

2) Why most published research findings are false:

For Ioannidis, the key reason for this exaggeration and misrepresentation in research can be summed up in one word: bias. “This can be conscious, subconscious, or unconscious,” he said of these deviations from the truth – beyond chance or error – that pervert science. His favourite offender is ‘publication bias,’ which gives a falsely exaggerated impression of the science on a subject because not all studies that get conducted get published and the ones that do tend to have extreme results. It’s like doing a bunch of tests to find out whether your new vacuum works, and even though most tests fail, only reporting the one time the vacuum turned on.

Ioannidis is well known for taking on the entire research enterprise in an essay entitled ‘Why Most Published Research Findings are False.’ In the paper, he described how a combination of uncertainty (no scientific finding is ever final) and publication bias creates a maelstrom of spurious findings that don’t hold up to scrutiny over the long-term.

3) Why you need to be cautious about early studies with big claims:

For another paper on the twists and turns in research, Ioannidis examined the reliability of findings in highly-cited original studies, focusing in particular on those which had been contradicted by later, more rigorous research. These influential studies were not about cold and abstract issues; many focused on the very questions that we all grapple with every day, such as whether to take supplements or not, and whether common medications – like aspirin for blood pressure – really work.

Here, he concluded, “Contradicted and potentially exaggerated findings are not uncommon in the most visible and most influential original clinical research.” In other words, splashy early studies with big effects were often found to be exaggerated or completely wrong. He also found that the original research continued to be cited, sometimes with complete silence on the more recent, contradictory evidence. For example, an early observational study revealed a supposed link between vitamin A supplementation and breast cancer, only to be overturned by a later, much higher-quality randomized controlled trial – yet the debunked observational study remained more highly cited and influential.

In a study, Ioannidis looked at six highly-cited journals between 1979 and 1983, combing for papers in which researchers claimed their basic scientific findings were going to lead to useful treatments. Out of 25,190 studies he identified, 101 made such claims. Yet, the vast majority of these studies were never followed up with randomized controlled trials to test those claims. Of the 27 that did, only five resulted in technologies that were licensed for clinical use in 2003 and only one has been widely used for the purposes for which it was licensed. This means the chances that someone promising a breakthrough and actually delivering one are about as slim as the chances of winning the lottery.

4) How to make science less science-ish:

At the end of the course, Ioannidis shared a few ideas about how to improve the status quo in science. He suggested first that researchers need to learn to live with small effects in their studies. “Having worked in different fields, most of the effects that are of interest are small,” he said. Most effects of a big magnitude – like the link between smoking and lung cancer – have already been recognized. To reduce the signal-to-noise ratio, he said, scientists need to design their studies accounting for the fact that  the effect sizes they are chasing may be tiny.

He also suggested that even if studies aren’t going to be replicated, researchers should at least try repeating their findings by getting an independent investigator to vet their raw data sets. Other fixes for science, which Ioannidis outlined in a new Lancet series on reducing inefficiency in research, include revamping the reward system for research and making data publicly available.

5) Why science, if flawed, is still the best alternative:

At the end of his week-long visit to Harvard, Science-ish asked Ioannidis whether he ever tired of poking holes in science, whether all his work has caused him to lose faith in the scientific process. With wide eyes, he exclaimed, “I remain as enthusiastic about science as ever!” He went on to describe all the benefits of science, why it is “the best thing that can happen to humans”: the value of rational thinking, of evidence over ideology, religious belief and dogma. “We have effective treatments and interventions and useful tests we can apply. We have both theoretical and empirical evidence that science is beneficial to humans and it’s a wonderful construct of thinking. . . Science is beautiful because it’s falsifiable.”

“There’s plenty of room to apply the very same (scientific) tools to the way science is done,” he added. “The question is: can we get there faster and more efficiently without wasting effort?”

Science-ish is a joint project of Maclean’s, the Medical Post and the McMaster Health Forum. Julia Belluz is senior editor at the Medical Post. She is currently on a Knight Science Journalism Fellowship at the Massachusetts Institute of Technology. Reach her at julia.belluz@medicalpost.rogers.com or on Twitter @juliaoftoronto




Browse

When science isn’t science-based: In class with Dr. John Ioannidis

  1. It’s about time…..thank you Dr.Ioannidis!

  2. Last week Dr. John Ioannidis told a room filled with Harvard doctors that they can’t trust most of the research findings science has to offer. “In science, we are very eager to make big stories, big claims,” he opened his lecture, with a mischievous grin. “The question is: are those claims accurate?”
    the answer – at least most of the time – is an unequivocal ‘no.’

    In one of his studies Ioannidis and a co-author randomly selected 50 ingredients from recipes. They then looked at whether those ingredients were associated with an increased or decreased risk of cancer. At least one study was identified for 40 of the ingredients – from bacon and bread to sherry and sugar – and most of the claims made in the studies contradicted each other or were based on weak evidence.

    For Ioannidis, the key reason for this exaggeration and misrepresentation in research can be summed up in one word: bias. “This can be conscious, subconscious, or unconscious,” he said of these deviations from the truth – beyond chance or error – that pervert science. His favourite offender is ‘publication bias,’ which gives a falsely exaggerated impression of the science on a subject because not all studies that get conducted get published and the ones that do tend
    to have extreme results.

    This is absolutely perfect. Now take this entire article, and apply it to climate science.

    • You didn’t read as far as point 5

      “At the end of his week-long visit to Harvard, Science-ish asked Ioannidis whether he ever tired of poking holes in science, whether all his work has caused him to lose faith in the scientific process. With wide eyes, he exclaimed, “I remain as enthusiastic about science as ever!” He went on to describe all the benefits of science, why it is “the best thing that can happen to humans”: the value of rational thinking, of evidence over ideology, religious belief and dogma. “We have effective treatments and interventions and useful tests we can apply. We have both theoretical and empirical evidence that science is beneficial to humans and it’s a wonderful construct of thinking. . . Science is beautiful because it’s falsifiable.””

      • the value of rational thinking, of evidence over ideology, religious belief and dogma.

        I most certainly did.

        • Yup. The conservative christians that form the core demographic of climate change denial have already proven those dogmatic atheist scientists wrong about evolution – thanks be to God.

          • Lenny, it’s hazardous to use sarcasm or irony when writing. People don’t get it. Another hole in the cultural fabric.

    • First a quibble, I hate it when medical doctors (and Julia) refer to medical science as if it is typical of all science. Meta-research is largely irrelevant in the real sciences (physics, chemistry etc) although that doesn’t mean mistakes aren’t made.

      Now to John’s point, apply this article to climate change. A somewhat fair observation, since climate change like the medical sciences involve wickedly difficult problems with strong and interacting feedback mechanisms to any stimulus. So many of the claims associated with the details of climate change are overstated or false, sure. John’s stance is that based on those mistakes we should ignore the consensus until the models are perfect. Should we ignore all of the conclusions of medical science as well?

      • We must go with best evidence until disproven, not merely disputed.

        • And the best evidence is that CO2 cannot possibly be the Great Climate Boogeyman it has been made out to be.

          • Why don’t you ask Dr. John…

          • In what esteemed publication might we find this evidence? The blog of the guy who reads the weather on the radio?

          • The Hadley Centre, NOAA, NASA, and the UAH all provide the evidence.

            Here’s the HadCRUT4 dataset, for example, showing how the same temperatures that the AGW movement predicted would rise rapidly have in fact stagnated over more than a decade instead.

            http://www.cru.uea.ac.uk/cru/data/temperature/HadCRUT4.pdf

          • I realize you picked HADCRUT because you’re a dishonest hack, and 20% of the globe is omitted from HADCRUT leading it to underestimate warming, but it indeed shows about .15C/decade of warming for the past 30 years.

          • I used HadCRUT because that is the dataset used by the IPCC when referencing observations.

            The others show the same thing; trivial if any temperature increase for over a decade.

            data.giss.nasa.gov/gistemp/graphs/

            http://www.ncdc.noaa.gov/sotc/service/global/global-land-ocean-mntp-anom/201101-201112.png

            http://www.drroyspencer.com/latest-global-temperatures/

            In contrast the predictions for scenarios of ever increasing CO2 (A2, below) make the claim of ever increasing temperatures.
            http://www.ipcc.ch/graphics/ar4-wg1/jpg/fig-10-4.jpg

          • They all show .15C per decade of warming or more for the past 30 years, which doesn’t contradict the IPCC projection graph.

          • They all show stagnation for well over a decade, which does not match the projections.

            There is no dispute that the globe WAS warming, but the fact that it is no longer doing so despite continued CO2 increases means that CO2 increases are not resulting in warming in the 21st century, and were therefore not predominantly responsible for doing so in the late 20th century.

            BTW, the prediction of scenario A2 is for over 0.20 degrees per decade, and increasing later.

            The current observations show temperature increase rates below even the ‘constant composition’ scenario.

          • And nowhere in the literature will you find projections for single decades. But you know that.
            Is it fun to “refute” projections of your own invention?

          • The projections are continuous.

            And they are different from the observations for almost two decades now.

            Considering how the AGW fear mongery was started in the 1980s based on little more than one decade of temperature increase, it is remarkable that nearly two decades of no increase should leave the fanatics of the movement unmoved.

          • So what do you wish to conclude from those two links?

            For clarity, here’s the result of taking into account the PDO 60 year cycle:

            http://wattsupwiththat.files.wordpress.com/2009/03/akasofu_ipcc.jpg

            One can note a considerable difference between what can be rationally expected as a temperature change on the one hand, and on the other hand the model predictions.

            The remaining influences, which Akasofu leaves as a simple linear fit, would be whatever CO2 contributes, plus the upward leg of a roughly 200 year cycle between successive Grand Solar Minima and the upward leg of a millennial scale cycle with nearly 1000 years between the Minoan, Roman, Mediaeval and Modern climatic maxima.

            There’s far less room for CO2 than the assumptions made for the modelling.

          • My conclusion is that judging from the swiftness of your response, you have neither clicked nor read them.

          • Which decadesare you referring to?

            And yup, it was based on little more than, uh, physics.

          • Physics says that the climate sensitivity to CO2 is about 1.2 degrees per doubling.

            http://climatephys.org/2012/06/28/climate-sensitivity-and-the-linearized-response/

            The fear mongery is based on sensitivity estimates three to four times greater than that. The IPCC Fourth Report suggests 3, 4, or even 5 degrees per doubling.

            That’s not justified by the actual physics.

            The fear mongery is being revealed as without foundation as each year passes without any sign of the rapid warming of the late 20th century, the same warming rates that the observations refute.

          • Warming rates haven’t be refuted by ten years or whatever cherry-picked period you’re using, because rates aren’t accurately determined by such short periods. I mean, how many ways can you be told this?
            Let’s apply your logic to another problem:
            Do you believe that if temperatures do not uniformly rise throughout the spring, that increasing day length has failed to predict rising temperatures?

          • LOL…

            Wrong analogy.

            You’re just desperately hoping that the warming will return, because otherwise all the fear and panic will be ridiculed.

            It’s not the rate or the trend that is important, but the stark difference between the predictions of the theorizings and the reality.

            It’s just not happening for you, and your beloved warming is just not supporting your paradigm.

            And even if the warming had continued, it would not confirm that CO2 had anything at all to do with it, much less that CO2 be the dominant driver.

          • Sure it is. Now tell us why it’s the wrong analogy.

            There is no “stark difference” between the predictions and the +/- .15C per decade of warming that’s occurred since the first IPCC was published.

            Of course the warming will continue it’s upward climb – jaggedly enough for you to keep playing your silly game.

          • It’s hardly +/-15 degrees since the first report, though the rate might have approached -15 in the late 1940s or at certain time periods in the late 1800s.

            And the only reason you believe temperatures will resume increasing is because you cling to the belief that CO2 concentrations be an important or even dominant factor.

            But that’s just the problem, isn’t it?

            After another lustrum or perhaps two of no warming, the “CO2 causes all the warming” paradigm is going to collapse even further.

          • Indeed we’ve warmed, +/- .15C per decade since 1990, according to the instrumental temperature record, and there’s no reason to believe your “evidence” demonstrates it’s stopped, any more than when the same “evidence” failed you in 1978, 1988 and 1997.

            Now tell me why my analogy above is wrong.

          • Where is your -0.15?

            For 1990-2000 it was closer to +0.20

            For the last decade it’s trivially different from 0.

            And the times when the evidence fails the models are from 1880-1910 and 1940-1970 when cooling took place while CO2 increased, and even 1910-1940 when warming was almost twice as rapid as the models ‘predict’ based on the popular paradigm’s ideas of sensitivity to CO2.

            http://www.cru.uea.ac.uk/cru/data/temperature/HadCRUT4.pdf

            The development of the temperatures corresponds much better with Akasofu’s description of a 60 year cyclicity overlaid on the recovery from the Dalton minimum, itself the upward leg of a roughly two century cycle for which we are at or near a peak.

            http://wattsupwiththat.files.wordpress.com/2009/03/akasofu_ipcc.jpg

            If you still cling to a belief in the IPCC projections, you;re going to be sadly disappointed.

          • Temperatures increase in the springtime as we move toward summer, for well known reasons and following a pattern known to have occurred many times before.

            But global temperatures show, if anything, patterns that will lead to cooling rather than warming in the next century.

            So your analogy just doesn’t correspond to the reality.

          • The assumption inherent in that blog would be that the 1970-2000 warming did not result from a warming state of the PDO.

          • Uh no, it clearly doesn’t. Unless you want to keep pretending that those lines represent annual temperature predictions and ignoring the matching shaded areas, in which case I’ll keep reminding you that you’re lying.

          • Those ARE the predictions.

            The shaded areas do not encompass the observations anyway, which makes it difficult to ascertain exactly what you’re trying to argue here.

            And if zero warming over decades is the prediction, then there never was an AGW problem in the first place.

          • For laughs, tell us what anomaly the IPCC “predicts” for 2013.

          • Nope? Can’t tell us what the prediction for 2013 was?
            Didn’t think so.

          • NASA’s GIStemp just just fakes in data for that same 20% of the globe.

            Neither assumption is fully compatible with completeness, but making no assumption (as HadCRUT does) has advantages over making guesses (as GIStemp does).

      • ” John’s stance is that based on those mistakes we should ignore the consensus until the models are perfect”

        That’s ridiculous.

        First of all, science is not based on consensus. Second of all, there is no consensus in climate science, there is a lot of uncertainty and disagreement. Thirdly, the models are essentially useless, we don’t need to ignore them til they’re perfect, we need to ignore them until they’re at least slightly useful, which they are not. Finally, we need to go where the evidence leads, and at this time the evidence is saying that climate estimates were wildly over-estimated, not just by the models, but by bias that pervades climate science.
        We should not ignore climate science, we should actually pay attention to what it is telling us, which is that the climate sensitivity of CO2 is vastly overstated and that previous predictions are vastly wrong.

        • you are vastly overstating your case

          • You are vastly understating what SCF is – and what is, is wrong.

          • …what he is…

        • If only climate deniers could publish their comprehensive and detailed analysis in peer-reviewed scientific journals, then this whole publication bias thing would be corrected. Imagine the awe when non-scientific and non-statistically sound models are used to explain ocean acidification, dendrochronology records, sea level increases, polar ice cap melting, and temperatures levels decade-after-decade exceeding previous peaks.

          • There is no such thing as a ‘climate denier’.

            Everyone is well aware that there is a climate.

  3. And the reason we should believe HIM is………?

  4. A recently published meta-data study has found that Dr. Ioannidis has substantially overstated his case. Consider:
    “the overall rate of false discoveries among reported results is 14%,
    contrary to previous claims. We also found that there is not a
    significant increase in the estimated rate of reported false discovery
    results over time”
    http://www.ncbi.nlm.nih.gov/pubmed/24068246

    There are lots of reasons why biomedical research, especially as it relates to pharmaceuticals, is not credible. British doctor, Ben Goldacre, who writes for The Economist magazine, recently published an entire book on the subject. However, to make the categorical statement that most scientific results are false is simply wrong. Science by its very nature is contrarian and as anyone who as worked in a lab or research facility knows, a great deal of effort and thought goes into refuting and undermining the opinions of competitors.

  5. It’s almost impossible not to have a bias. That fact played a large role in the ‘Star Trek’ series. Bias isn’t a bad thing. It just is. But we don’t want (and I have been trained in Science) to have only scientists running the show. Science presents it’s findings, and others, some who are specialists in Science and others who are not, make decisions. On a personal level, I don’t want it to be a requirement that the only people who run the country, the provinces, municipalities or other facets of government – like First Nations -,have a PhD. or any other educational degree. Some of the brightest people I know had very low levels of formal education. I like science, but a lot of science is BS.

  6. The article shows:
    “spending on life sciences research alone has topped $240 billion US”
    Which, helps explain why the “climate scientists” are so disheartened that the climate is not realizing a catastrophic increase in temperature due to increased CO2.
    Every time you hear someone say, “The science is settled’…link them to this Maclean Article.
    Nothing is ever settled in the mind of someone who actually adheres to the scientific method.
    For a group of people who have spent the last 30 years warning us about climate change…you would think they would be greatful that they have been proven wrong about their theories of warming caused by CO2. One would think, these “scientists” would see this as good news….but given the money at stake, I guess we shouldn’t be surprised.

  7. A couple of science issues important here are of course, “climate science” and also forensic science. The mistakes and errors are all over the place and it’s important they not be if we’re going to spend hundreds of billions of dollars and change our society — or put someone in jail. Big issues. Small minds forcing “consensus” thought on all of us.

  8. “Out of 25,190 studies he identified, 101 made such claims [that their findings would lead to treatments]. Yet, the vast majority of these studies were never followed up with randomized
    controlled trials to test those claims. Of the 27 that did, only five resulted in technologies that were licensed for clinical use in 2003 and only one has been widely used for the purposes for which it was licensed.This means the chances that someone promising a breakthrough and actually delivering one are about as slim as the chances of winning the lottery.”

    I am very excited to find that my chances of winning the lottery are 1/27 (approx. 4%) – or 1/101 (1%), or even 1/25,190 (0.0003%). Those are far better odds than I’d previously expected! 1/13,983,816 (0.00000007%) – hyperbole much?

    But honestly, this paragraph is pretty silly. 101 research groups thought that their research would have direct human medical applications – more than 1/4 of those were actually picked up by [companies, most likely] who paid additional money for those claims to be tested in the real thing – the human. Amazingly, almost 1/5 of those tested in humans actually ran the gamut of safety and efficacy testing (and the large amounts of money required to do this testing) to get approved for use in humans. Finally, 1/5 of those [is] used widely – which is to say, there aren’t other technologies or approaches that are better and more effective than this idea that was put forward sometime between 1979 and 1983. If you aren’t familiar with biomedical research, maybe these numbers sound preposterous – personally, these numbers sound pretty good, considering all the hurdles you have to jump through to go from “[this drug/intervention] works in cells and might help people with [this condition]” to “[this drug/intervention] is safe, does what we say it does, and is better than most/any other treatment already out there for treating [this condition].”

    There are valid concerns about clinical trials (human and animal) and how to appropriately interpret data from this type of research. Researchers in these fields should be (usually are) well aware of that, and it is valuable to bring this to the public’s attention as well. It would be particularly beneficial if members of the media (issuers of press releases, as well as writers and editors) also took notice of this. However, the issues described in this article have very little bearing on most scientific research (eg. math, computer sciences, physics, chemistry, even hardcore biochemistry). In fact, aside from the caution about overinterpreting early or small human and animal studies, it’s not terribly applicable to a lot of the basic biomedical research either. So no, the vast majority of the scientific research “out there” is not wrong – but the stuff you hear about in the news may not be as amazing (or terrible!) as they make it sound. How hard is it to encourage people to take sensationalist news stories with a grain of salt?

  9. And the conclusion is that “science” can never be certain about anything.

  10. Most of the problem could simply be solved by not forming beliefs based on single epidemiological studies. Statistical correlations derived from surveys and reporting are only good for a hypothesis. Get a meta-analysis of epidemiological studies in nutrition, look at ALL of the experimental evidence and then form your conclusion. Anything else is just madness and it’s less a problem with science as it is with people interpretation science poorly.

    Though publication bias is a problem I’ll give him that.

Your email address will not be published. Required fields are marked *