A Science-ish Q&A: Dr. Ben Goldacre

The ‘Bad Science’ columnist on quacks, scaremongering journalists, and the importance of good research

by Julia Belluz

Photograph by Rhys Stacker

With his “Bad Science” column in the Guardian newspaper and a best-selling book of the same title, U.K. physician Ben Goldacre has been leading the international charge in quack-busting, unpicking dubious scientific claims made by everyone from politicians to alternative-medicine practitioners and nutritionists. But Dr. Goldacre doesn’t scrutinize only the most obvious quacks among us. As he told an audience of health professionals, policy-makers, and researchers at the Evidence2011 evidence-based medicine conference in London, “We’re on a quack continuum and our work here today is unpicking the details of evidence to make sure we stay at the saintly end of that continuum rather than the dodgy one.”

As of this fall, Dr. Goldacre was on a break from the bedside to work as a research fellow on clinical trials and publication bias at the London School of Hygiene and Tropical Medicine. (He’s also the Science-ish patron saint.) Julia Belluz sat down with him in London to learn about how other doctors can undertake similar quack-busting work, about his forthcoming book on the pharmaceutical industry, and why understanding the mechanics of bad science is the best way to arrive at good science.

Q: In a presentation here, you said we can put all evidence on a “quack continuum.” Can you explain what that is?

A: I write about misuses of evidence in plenty of different spheres: scaremongering journalists, obvious quacks and naturopaths, and flaws in the way that evidence is used in mainstream academia, medicine and in (government) policy. One of the things I always found interesting is the same tricks are used to distort medicine in all of those domains.

There are tricks used to distort evidence in medical academia that are more sophisticated, and they are more sophisticated because they are used to bamboozle and confuse a more sophisticated and more evidence-savvy audience-but they are nonetheless in the same basic category as the distortion of evidence from outright quacks.

Q: So what are some examples of obvious quackery compared with more covert quackery?

A: One example is how you can make your treatment look better by comparing it with something that’s really rubbish. We laugh at acupuncturists who do trials where acupuncture comes out as being brilliantly effective when you compare it with no treatment at all. But we have the same problem in mainstream medicine. It’s common to compare your drug with placebo, which is basically the same as comparing your drug against nothing. And that’s justifiable if there’s no currently available treatment for the disease your tablet is there to address. But if there is a currently available treatment, we don’t care if your new treatment is better than nothing; we care whether it’s better than the currently best available treatment.

And yet, there have been several studies on this, the most recent published a couple of months ago, which showed that about one-third of all treatments approved by the U.S. Food and Drug Administration have evidence only that they are better than placebo, even when you’re looking at tablets where we already have a currently available treatment. That’s one problem, as dramatic in its prevalence in the world of mainstream academia as it is in the world of quackery.

Q: Your focus has shifted over the years-from the low-hanging fruit of alternative medicine to the more intricate and complex dealings of pharmaceutical companies. Why the change?

A: I think the trajectory is probably from easy aspects of research methodology to more complicated aspects of research methodology. There are some obvious things—particularly in the U.K.—such as quacks and scaremongering journalists who were getting away with making extraordinary howlers and being treated with the utmost respect and credulousness. The first things I wrote about were the basics of how to do a trial. How do you know if something works? How do you know if something is good or bad for you?

So something like homeopathy, where you’re getting a dummy sugar pill that has no medicine in it, is a perfect teaching tool for evidence-based medicine because when homeopaths ran this trial showing that their dummy sugar pill works better than placebo, that’s exactly the problem you’re trying to avoid in real evidence-based medicine. You’re trying to avoid seeing a positive treatment effect where there clearly is none.

By going through the ways a trial can be flawed by design—by not being properly randomized, by not being properly blinded—you can use homeopathy as a brilliant teaching tool for how crap studies can get. It’s also a very good teaching tool for more complicated topics, such as cherry-picking results.

So once you cover the basics of how trials work, you can move on to how trials can be badly designed, how trial outcomes can be selectively reported, and all the fascinating areas of how people can set out claiming that they’re measuring one outcome as their primary outcome and then suddenly a completely different outcome gets reported as the primary outcome when the paper is published.

Q: You’ve worked to explain evidence in that systematic way because you’ve said you always tried to get away from arguing from a position of authority. Why do you find authority so offensive?

A: The thing that interests me is not whether something is wrong but whether something is interestingly wrong, whether there is an aspect of research methodology that can be explained using somebody getting something wrong and being an idiot as a kind of emotional hook for making that quirk of research methodology relevant and interesting to peoples’ lives.

Because of that, I have never felt comfortable charging in and saying, “You know, here are some drugs that don’t work.” I’m not really interested in the answers of research, I’m interested in the methods and the structures of it. How do you know if something is good for you or bad for you? Unless you explain all the evidence, all you’re left with is an authority play.

Q: Who or what is your next target?

A: I’ve already written a lot about problems in the information architecture of academic medicine, the most extreme end of that being publication bias. So I’m writing a book about how the pharmaceutical industry distorts its evidence, and more relevantly, how doctors, academics, regulators and governments have acquiesced in the face of that, and how we’ve failed to address some very obvious problems. (This book, The Drug Pushers, is to be published in Canada by the end of the year.)

Q: Any findings from your upcoming book that you can share?

A: One of the things that is so interesting about writing in this area is that the outcomes that you have, the information that you have, is always three to five years behind the curve because it takes time for a drug to be widely adopted, (to) kill people if necessary, and for that signal to be detected with the very imperfect, post-marketing pharmaceutical company vigilance strategies.

Also, (it takes time) to try to get clues from outside an organization that there was bad behaviour within an organization which could have exacerbated the harm.

And then you have a long process of going to court. And finally, only in a small select number of cases, (finding) some internal documentation (suggesting distortion of evidence). Because that’s five years behind the curve, you always have people saying, “That’s an isolated incident,” or, more likely, “That’s an old problem.”

One thing I’ve done in the book is document how in the past people have said, “Oh that’s an old problem which we have fixed now.” Each time people say it’s fixed, it’s not. It keeps happening even now.

Q: Do you have examples of apparent solutions to real, live problems with the pharmaceutical evidence base?

A: Two years ago, I was on a BBC program up against a chap who previously worked for Merck in the U.K. I was explaining the problems around publication bias. And he said this problem of negative trials going missing in action had been fixed because you now have to register your study.

That sounds really good, but there’s a paper from 2009 which goes through every single trial published in the top 10 journals in 2008, looking at whether the trials were properly registered before they started and on completion. About one-third of them weren’t. You’ve got journal editors saying we’re not going to toe the line and publish unregistered trials anymore. But when you look at it, demonstrably, journal editors failed in their role as gatekeepers. So the history of the distortion of evidence in medicine is littered with these failed solutions.

Q: You’ve been quite outspoken as a physician, raising your voice when you see misreported studies or politicians perpetuating bad science, at a time when many doctors are afraid to speak out. What advice would you give to your colleagues who want to stand up about the distortion of science and evidence?

A: Firstly, nobody should feel under pressure. There’s no obligation to stand up and communicate. But if you want to, it’s very easy and more people could and should do it.

I got my (Guardian) column by ringing up a switchboard number on the letters page of the newspaper. A lot of times, editors are very pleased to hear from people who know about epidemiology or evidence-based medicine or medical statistics or medicine.

You also don’t really need to worry about whether you can write or not. This is one of the great untold secrets of journalism: a lot of copy by people who self-identify as professional writers is complete rubbish and it gets knocked into shape by very good editors on magazine and news desks.

Or you can set up a blog. People can be snotty about blogs but really, 1,000 blogs getting 400 views each is 400,000 views in total. And that compares very favourably with the mainstream media.

So a vast army of nerds, each working on their issues, catching a small corner of the world interested in those issues, in total are every bit as powerful a resource as media outlets.

Q: Has your finger-pointing ever got you into trouble?

A: Sir Iain Chalmers (a physician and one of the founders of the Cochrane Collaboration, a non-profit group that produces systematic reviews on health-care interventions), who has been very outspoken for a long time about flaws in evidence-based medicine, describes what he has as “terminal candour.” Toward the end of his career, he said he can risk saying literally whatever he wants.

I have been doing this since I was 29. I’m 37 now. Nothing bad has happened to me. You get homeopaths and anti-vaccination campaigners and conspiracy theory bullies who bizarrely assert that I am somehow a servant of Big Pharma when in reality, if you’ve read my stuff, you couldn’t find a bigger critic.

I hope I never missed out on research funding or missed out on a job just from standing up and communicating about what the real evidence shows. The adverse-risk outcomes that people fear from writing sensible stuff about evidence-even when it involves being critical-aren’t as bad as people say and I think are outweighed by the benefits.

Q: Any final words of advice to mobilize an army of nerdy, would-be quack-busters?

A: Doctors need to grow a bit of oomph about setting out evidence clearly in the way I think their patients would expect them to.

It’s quite common in a one-to-one medical consultation for there to be a conflict between the doctor and patient in what they want. For example, patients come wanting benzodiazepine to get to sleep. The doctors won’t want to prescribe that because they think it’s not in the long-term interest of their patients—they think it will cause more harm than good.

We have an obligation to stand up not just to patients but administrators and legislators. Not in a pompous, childish, warfare way. But to stand up and set out the facts clearly and not let issues of values and evidence get confused, as they so often do.

This article first appeared in The Medical Post. To register for the website, click here

Science-ish is a joint project of Maclean’s, The Medical Post, and the McMaster Health Forum. Julia Belluz is the associate editor at The Medical Post. Do you have a burning question about science or a health claim you’ve seen this year that seems dubious? Message Julia at julia.belluz@medicalpost.rogers.com or on Twitter @juliaoftoronto by December 13 to participate in a year-in-review Science-ish column.




Browse

A Science-ish Q&A: Dr. Ben Goldacre

  1. There is also a problem with study funding bias, in that the Rx companies that fund research are uninterested in solutions that aren’t proprietary such as diet and exercise and who look for products to replace the recommended treatment.

    See for instance any of the commericals showing unrealistically active, fit people who “need” a drug when “diet and exercise aren’t enough.” The vast majority of North Americans with dyslipidemia could address their cholesterol problems with diet and exercise, but drug companies and doctors pretend that’s already been tried and sell them pills instead.

    • I would disagree with your statement that physicians “pretend’ that patients have tried diet and exercise to control cholesterol or for that matter hypertension or type II diabetes and then “sell them pills instead”.  Physicians have to tell patients to undertake diet and exercise to try to improve their health…otherwise the physicians can be sued for malpractice.  However, very few patients are willing to change their lifestyle completely and therefore the medication is the option left available to the physician who has to treat the condition.  It is a interesting fact that illnesses in developing countries are related largely to communicable disease while in our developed world it can be almost wholely attributed to a unhealthy lifestyle.  I’d venture to say, if we all gave up our bad habits, the pharmaceutical companies would be far worse off.

  2. It’s good that Mr. Goldacre is pointing out some of the obvious flaws in the medical research world. Of course, these flaws are well-known to anyone who has read the work of Richard Smith, Richard Hortan, Marcia Angel, etc (former editors-in-chief of BMJ, the Lancet, and NEJM, respectively).

    Of course, he is wrong about homeopathy. Not that homeopathy is without flaws – far from it. It is an imperfectly understood and developed science. There are all kinds of people making claims for homeopathy which are unfounded. But it does work, in its imperfect way, and would be much stronger if it were more thoroughly developed through research, instead of opposed a priori on philosophical and professional (economic) grounds

    Goldacre shows his lack of objectivity when he says certain homeopathic trials show effects when there “clearly is” no effect. A real scientist is neutral.

    Are there flaws with some of the positive homeopathic trials? probably, though some are of extremely high rigor.

    But, flaws can be found in any study ever done. Period. So, if you are looking to dispute the efficacy of any therapy, you can simply point out a few flaws and say “the studies are insufficient”.

    So, if you are, as Goldacre, looking for a certain result, you can always find some data to suport it.

    The fact is, the majority of homeopathic trials have significant methodological flaws, since the homeopathic process is very difficult to properly assess in a controlled setting. However, this leads to false negatives far more often than false positives.

    The facts are as follows: Of all randomized, controlled homeopathic trials having sufficient methodology and statistical power so as to draw firm conclusions, positive studies out-number negative studies by 6 to 1.

    Another point nearly always conveniently ignored by homeopathic critics is that basic science research of very high quality overwhelmingly demonstrates biological effects of homeopathic remedies:

    http://www.ncbi.nlm.nih.gov/pubmed/17544864

    http://www.ncbi.nlm.nih.gov/pubmed/19945674

    I don’t think Mr. Goldacre is shilling for the pharmaceutical industry, he’s just overly-invested in a scientific paradigm which is set to expire soon – and it has nothing to do with the lack of ethics in the medical research world.

    • Heh! “imperfectly understood and developed science”. In reality it’s perfectly well understood to be quite possibly the most deranged pseudoscience in existence, and it’s rightly “opposed a priori” on those – scientific – grounds, as well as on ethical grounds. 

      A “real scientist” knows that a clinical trial of homeopathy is *intrinsically* and irredeemably methodologically flawed (not to mention unethical). Homeopathy CTs are futile, mountain-of-established-science-evidence-defying cargo cult science, and Ben Goldacre is doing scientific inference absolutely correctly and objectively by pointing out that (significant) positive results from such nonsense trials are, inevitably, evidence of error rather than evidence of homeopathic phenomena.

      A “real scientist” also knows the difference between high quality basic research and the appalling rubbish which can be found in a pseudoscience journal such as Homeopathy or CTIM.

      • A real scientist does not judge the validity of consistent results based on current theory.  A real scientist upgrades current theory so as to explain consistent results.  Else we’d still be thinking the sun revolves around the earth.  It amazes me how many people claiming the mantle of science don’t accept its ruling principle.

        • “A real scientist does not judge the validity of consistent results based on current theory.”

          Oh yes (s)he does! It’s a fundamental principle of inference in empirical science and, as I said, it’s the reason that the rational explanation for any anomalous results coming from empirical homeopathic pseudoscience is error. Furthermore, a real theoretical scientist also first makes sure that his/her new ideas and hypotheses are consistent with the established-by-a-mountain-of-evidence current theory. (S)he knows that if they’re not, they’re already proven wrong and no empirical testing is necessary. Sam Hahnemann (the founding homepathy ‘theorist’) had an excuse for seriously contemplating 200 years ago what is now known to be absurd nonsense. His modern counterparts are just ignorant and deluded crackpots.

          • Karen W is correct. Don’t know where you get your preconceived notions from, but they have absolutely nothing to do with real scientific method and make no sense at all.

          • I’m sure my “preconceived notions” are of mysterious origin to you and don’t make any sense to either you or Karen, and that you both sincerely do believe you know what “real scientific method” is. Laplace’s principle and its consequences for scientific inference (and empirical practise) are not particularly well known even among scientists precisely because science is so rarely done so perversely and with such wilful ignorance and incompetence that they apply.

      • Pseudoskeptics attacking Homeopathy generally begin by claiming there are no studies… then when the studies are presented they opine that they’re unwilling to accept them. Fortunately real patients in the real world don’t rely on Opinion-based Medicine from social media.
        Conventional medicine is not a science. It does not attempt to disprove it’s own hypotheses. It’s “science-ish” only in the sense that meta-analyses of studies employs mathematical statistics, even though on a practical basis this may mean a benefit to one person in 10,000 to be considered statistically significant.
        An MD can read the results of a controlled study, however this does not make her a “scientist”, or even an expert in a particular field of medicine. Moreoever, medical associations dictate treatment protocols. Failure to follow them can result in disciplinary action and/or lawsuits.
        Basic materials science and nano-particle studies indicate that Homeopathic remedies are not “nothing”. Homeopaths regularly use medicinal substances that are not highly diluted in any event. Treatment has always been directed towards the individual — it is not based on some nebulous statistical average.
         

  3. I agree that we need to start questioning experts much more than we do. Msm has mostly stopped questioning authority, for instance, and they seem to be in thrall to technocrats, academics and other charlatans.

    Wall St Journal ~ When We See What We Want:

    The larger lesson of the Gould-Morton affair is that bias is everywhere, that many of our studies are shot through with unconscious errors and subtle prejudices. To Paul Simon, we see what we want to see and disregard the rest.

    In recent years, it’s become clearer that these psychological shortcomings are a serious societal problem. Because we believe we’re impervious to bias—we’re blind to our own blind spots—we assume that our judgment isn’t affected by financial incentives or personal opinions. But we’re wrong.

    This problem has been most convincingly demonstrated in medical clinical trials. A 2005 study of psychiatric drug trials found that when academic researchers were funded by a drug company, they were nearly five times as likely to report that the treatment was effective. (A similar pattern was found with oncology drugs.) What makes this result so disturbing is that all of these studies were randomized, double-blind trials, which are typically regarded as the gold standard of medical evidence. And yet the financial incentives seemed to decisively influence the data.

    http://online.wsj.com/article/SB10001424052702303936704576397771567839728.html

    • I have a few comments to make about the article you quoted.  One is regarding the assertion that physicians who benefited from trips and lunches provided by the pharmaceutical companies, found favor with the medications the pharmaceutical company was selling.  There is an easy fix for this….ethical standards should be in place to stop physicians who work in programs that are affliated with universities in Canada from accepting any kind of “perks” from any pharmaceutical companies. 
      This article is from the US and so I am not sure how studies are conducted there.  I can say that in Canada, university ethics boards decide which studies are undertaken.  Pharmaceutical companies do share in paying part of the costs as do the universities and the health regions.  Participants get their medication for free and their parking paid.
      I know people are very suspicious of “big pharma” but I just want to say something about some medications, especially medications to treat conditions like schizophrenia.  There are quite a few of them out there and they continue to develop more and more, which is a good thing because some of the current ones have terrible side effects and so the search continue for medications that are effective but less troublesome in their side effect profile.  Also, some people are treatment resistant and therefore they do not respond to the drugs currently on the market.  Therefore, it is a good thing that they continue to search for alterntive medications that may work for this portion of the population.

  4. Reply to Heathcare Insider -you say:

    “Also, some people are treatment resistant and therefore they do not respond to the drugs currently on the market.”

    Isn’t the phrase “treatment resistant” simply just a way to pass blame to patients for lousy medications that have never  been properly shown to help? 

    • We are talking about the patient’s ILLNESS being treatment resistant and are in no way “passing blame” toward the patient.  Having said that, there are many patients whose illness responds to medications that those with treatment resistant illness don’t so you cannot really say that the medication is lousy.  As Dr. Goldacre said, the studies done should be measured against the older “gold standard” medications to see if these medications work as well or better and then establish whether they have a better side effect profile. 

  5. Sounds like his book The Drug Pushers, is similar to ”Selling Sickness: How the World’s Biggest Pharmaceutical Companies Are Turning Us All Into Patients by Ray Moynihan and Alan Cassels”

    “Serious questions are raised that our profession should be asking itself about the reluctance to establish stronger ethical boundaries with the pharmaceutical industry, regulation of financial relationships with sellers, acceptance of financial incentives, and frank conflict of interest. Material is drawn from several countries’ experiences and it reveals the cost of shrinking government support for research and evaluation; it leaves drug development to an industry that aims to fund only what might increase sales.The information in this book confirmed many things I had suspected.

    Physicians reading this book should recognize big pharma’s influence on the provision and cost of services and increased testing and office visits. The overlap from consumer advertising in the United States to the Canadian system and the number of patients sent to physicians’ offices specifically to request products they have been “sold” are further evidence of the industry’s sway.”
     
    http://sellingsickness.blogspot.com/ 
     
     
    Just look at the mess we are in with OxyContin abusers.   There should have been more control and a huge warning on how addictive the stuff is.  I find doctors are handing out antidepressants like candy as they always seem to have a stash of ‘free’ samples.   Much of the stress, anxiety, depression people experience is situational and tends to get treated as clinical. 

    Look forward to reading Dr. Goldacre’s book.   Love his avatar for his Bad Science column, lol

    • Are physicians offering you anti-depressants, Leo?  If you look at the brain and how the most commonly prescribed anti-depressants work, you will see that most work on neuro-transmitter sites as re-uptake inhibitors.  They don’t actually give you anything, they just stop the neurotransmitters from flushing away because people who suffer from depression don’t make as much of these neurotransmitters as people who do not suffer from this biological illness.  Let’s not pretend that stress and anxiety do not precipitate other physical illnesses as well such as heart disease,  gastritis and bowel afflictions.   Certainly, some mood disorders are caused by situational crisis but to suggest that clinical depression isn’t real or that antidepressants are a happy pill is not accurate.  If you have a normal level of the neurotransmitters that regulate mood, your response to the antidepressant would be nil.
      As for samples, physicians have samples of every kind of medication including those to treat hypertension and cardiac problems.  They give these samples to poor working people who can’t afford the newest meds and don’t have medication plans through work.

    • Um, this article has nothing to do with leprosy.

      • Egg. On. Face.  I have made the correction. Many thanks.

        Despite this error, it is most definitely an interesting application of homoepathic prophylaxis.

Your email address will not be published. Required fields are marked *