RCTs: the next big thing in politics? - Macleans.ca

RCTs: the next big thing in politics?


The British physician Ben Goldacre, whose 2008 book Bad Science pushed him into the global front rank of science popularizers, has collaborated with a UK government thingie called the “Behavioural Insights Team” on an important paper whose title alone captures a significant amount of its importance. It’s called “Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials”. Goldacre refers to the paper as an attempt at writing a “Ladybird Book” on public-policy RCTs. I am not quite sure how to translate this for the New World audience: “Public Policy RCTs for Dummies” would be in the ballpark, though possibly in the left-field bleachers.

The randomized controlled trial is a model for knowledge-acquisition, a particular kind of scientific activity, that has progressed slowly upward in generality and power throughout modern times. It started in late 19th-century psychology and agriculture, took over medicine after the Second World War, and has recently (as Goldacre et al. point out) taken root in web design. Its history is a bit like the histories of Darwinian evolution or Turing machines or the equal-temperament keyboard: it’s an idea, created to address very particular situations or questions, that seems more and more fundamental the more people play with it.

Nobody would now think of introducing an important health intervention, whether it be a drug, a surgical procedure, or a piece of equipment, without subjecting it to an RCT. Heck, the Pepsi Challenge is a crude form of randomized controlled trial. The basic idea isn’t that complicated. So the question the Behavioural Insights people have been asking is, how come we almost never do this with public-policy interventions? Bureaucrats are deeply devoted to testing new ideas and programs by means of the “pilot project”, but rarely do they make sure such projects have the key elements of an RCT: (1) truly random assignment of the intervention being considered, (2) test criteria selected and announced in advance; (3) the existence of a matching control group that receives a different intervention or none at all; and (4) safeguards designed to limit the possibility of experimenter manipulation of results, whether conscious or unconscious.

One can imagine Sir Humphrey Appleby flinching at all this brute objectivity designed to deliver cold yes/no decision guidelines to the ruling class. Items 1, 2, and 4 supra deprive him of a lot of the power to get the result he wants in advance. And that is probably one reason why the RCT model has not been terribly influential in policy circles. Yet RCTs, as the UK paper notes, were embraced quickly in medicine despite being, in that setting, very expensive and sometimes fraught with ethical intricacies. When it comes to a question such as whether it is worthwhile for courts to follow up on parking tickets with text messages, RCTs can be absurdly cheap, and nobody could possibly find some other good argument for not doing one.

It’s easy to overstate the novelty of the public-policy RCT: the famous and still much-discussed RAND Health Insurance Experiment, for example, began more than 40 years ago. And entire fields of public policy are impossible to capture in an RCT lens. It would be nice if we could settle the macroeconomists’ intergenerational wrangling over the appropriate target for monetary policy by means of a controlled experiment, for example; but it is hard to see how you could pull that off. There are, however, fields of statecraft that are very open to RCTs, and that are in desperate need of the epistemic discipline they impose. The best example might be foreign aid, and the RCT model has caught fire amazingly quickly in that realm, thanks to scholars like Esther Duflo and Abhijit Banerjee.

Pepsi Challenge-ing economic-development interventions in the Third World has already won Duflo a Macarthur genius grant and the John Bates Clark Medal. There’s pretty much no top honour left for her to nab except the quasi-Nobel they give to economists, and she is a lock for that if she lives long enough. It also seems inevitable that some careers will be made preaching the RCT gospel here, in Canada; all that is needed is for a few good policy professionals to take up the torch. And look: Goldacre’s just handed it to you! He is, I think, a tremendously key figure in this development. For the past decade or so he has been doing for RCT methodology what pulp science fiction writers of the ’40s and ’50s did for physics, taking up the cause of Bradford Hill and Archie Cochrane and our own David Sackett to make lay converts for gold-standard evidence-gathering.

Maybe it’s even possible RCTs will turn up as an element in electoral politics, as a way of finessing ideological questions for post-Cold War voters who (I notice) don’t like labels and won’t take them. There have been a lot of jokes floating around about a post-ideological “Economists Party” devoted to accepted principles of free trade, public choice, and mainstream welfare economics. Perhaps it should be an “Evidence Party” instead?


RCTs: the next big thing in politics?

  1. Never in a million years will our bureaucracy introduce evidence based trials because our public servants are mostly social scientists who peddle lamarckisms. Government is efficient at stealing people’s money and then transferring it to others – civil servants are witless when it comes to creating useful public services. Government would mostly disappear if proper trials were held before anything was implemented because social sciences are useless when used to create public policy.

    Would an Economist Party really be an evidence based party? Economists are astrologists who try to sound clever with lots of dubious stats. Any party that has connections with social sciences can not accurately be described as evidence based.

    • This is trolling.

    • TonyAdams is right. It takes an extremely disciplined, rigourous, and capable mind to apply evidence-based science successfully. That’s why good scientists are so exceedingly rare. Having worked in the federal bureaucracy for many years, and witnessed numerous numbers monkeys (statisticians) who thought that evidene-based science required nothing more than a solid grasp of statistical modelling and sampling theory, I can assure you that there never will be widespread success with RCTs in government.

      Many bureaucrats will embrace the concept, (hell, they’ll love it) but they’ll bastardize it and dilute it beyond belief to suit their own agenda. Not because they’re bad people, but because they’re average people. In other words, human. If good science was easy, we’d all be doing it. Alas, human nature guarantees otherwise. Science requires, at the very least, an understanding of one’s own limitations and fallibility. The vast majority of us, if we indeed possessed that self-awareness, would admit to ourselves, “I’m no scientist.”

  2. How about a RCT of the success of austerity economics?

    Wait! No. It wouldn’t even have to be random. It could be comprehensive.

    Sorry. Am I off-topic?

    • WTF is ‘austerity economics’?

    • You could say that about any facet of macroeconomics. Applying rigourous scientific methodology to macroeconomics is not even possible, which is precisely why everything is so screwed up right now, and likely on a semi-permanent basis. We create complex models, apply all the necessary “scientific and mathematical rigour”, then stand there with that “who farted?” look on our faces when reality doesn’t match our “evidence”. Rinse and repeat. We’ll be doing it a hundred years from now.

      • Don’t agree, Cranky. The mapping between austerity and economic stability is simple. How are those economies practising austerity, at any cost, doing compared to those who have taken a balanced approach?

        Scandinavia is doing fine. How come?

        Austerity has clearly failed. You can obfuscate, sure. But it is clear to anyone watching, even idly and without ideological bias, that those economies who have rigorously compensated for the redistribution of resources and effort that unfettered capitalism inevitably implements are producing healthier economies..

        It is not complicated. It is plain RCT. How are the austere economies doing?

        Not so good. The evidence is there. Check it out.

        • Yeah, so, thing is, you don’t really have any idea what a “randomized controlled trial” is.

          • I agree that I show no evidence of understanding. It was a stupid post. Please see my post of a few minutes ago.

        • It is not complicated. It is plain RCT. How are the austere economies doing?

          So a comparative study of different national economies qualifies as a randomized controlled study?

          An RTC, among other things, requires a control group. No such thing exists at the macro level in economics. (Indeed, it’s difficult enough to do at the micro level). Therefore there a true RTC macroeconomic experiment is not even possible. Thus my objection to macroeconomists masquerading as true scientists with real hard evidence.

          • You’re right Cranky. I withdraw my comment. Please see my posting of a moment ago.

        • Cranky and Colby,

          I did go off the rails in this topic. My ‘example’ was not initially intended as RCT, it was a completely off-topic indefensible rant. My later defence was intended as a defence of my off-topic assertion. I thought I had been saying that we don’t need RCT to see the truth of my assertion, but I see that I babbled on enough to make a complete ass of myself.

          I have no idea what I meant by “it’s plain RCT”. It is clearly not – for all the reasons that Cranky provides. I have a ‘good enough’ understanding of RCT to recognize that. I understand that too many whiskies is no excuse but I will have to fall back on that.
          All in all, I feel a bit of a dope this morning.

          • No problem bobbyriled. I had my own rant above that was only tangentially related to the topic, as was pointed out by the author. I never let the facts get in the way of a good rant. Cheers mate.

  3. I would vote for the Evidence Party in a heartbeat. The CPC’s derisive attitude toward evidence is what drives my loathing of the current crop. Their crusade against evidential decision-making is best demonstrated by their gutting of the long form census, replacing it with a more expensive, essentially worthless survey.

  4. The randomized double-blind concept works well in medicine and has some merit, but let’s not get carried away. Much of this sounds like creeping scientism. The belief that we can apply scientific methods in the social disciplines to make them more “rigourous” has already completely destroyed economics, creating entirely false and fraudulent fields of study (macroeconomics anyone?). Instead of going with what we have proved to be true through much trial and error and repetition (science), we run statistical formulae and then pretend to know something we don’t (scientism). This same pattern is repeated across numerous disciplines, with economics being the biggest – but by no means only – offender.

    Now they want to apply it across the board to all government policy? The fact is, the results of social policy can NEVER be measured completely objectively. Statistical models, however complicated, always contain within them the bias of their creators. As do the interpretations of those results. You can come up with criteria ahead of time, but you can’t completely control for how the results are interpreted. And of course we all know that double-blind, controlled drug trials are NEVER manipulated and NEVER inaccurate, right?
    I am not anti-science (though I’m sure I’ll be accused of precisely that). But believing that one can apply completely objective scientific methods to all areas of social policy – based on statistical models no less – is not science at all. For anyone unconvinced, read The Black Swan by Nassim Taleb. He makes a brilliant mockery of our belief that we can accurately model and predict the broad effects of government policy. Scientism is to science what ‘truthiness’ is to the truth. A farcical and destructive imitation of the real thing.

    • The article above is about randomized controlled trials. Which could be summed up loosely as “organized trial and error, with repetition”. But you don’t otherwise mention them, instead going off on a riff about science in general and macroeconomics in particular (hint: there’s a reason I singled it out as an example of a field that is not easily open to RCT evidence).

    • I was more concerned about how the RTC concept will be misused and abused by bureaucrats and politicians than directly criticizing the idea itself. I should have made that clearer.

  5. They were big when I was a kid in the late 1960s and 1970s, but they seldom came up with the desired answer, so enthusiasm drained away.