What do we really know about e-health? Not much.

There’s still much to be learned about its true impact on cost and quality of healthcare

by Julia Belluz


The statement: “Our research raises real concerns about whether health information technology is going to be the answer to reducing costs.”—New York Times, Mar. 5, 2012

In a new study published in Health Affairs a group of American researchers looked at the records of patient visits to a sample of office-based physicians to find out whether doctors with electronic access to imaging results ordered more tests than doctors without them. Surprise, surprise: the researchers found that the digital doctors were 40 to 70 per cent more likely to order imaging, and even blood tests, than doctors without access.

Although the survey did not collect data on payments for those tests, the investigators stated in a follow-up about the study, “It’s hard to imagine how a 40 to 70 per cent increase in testing could fail to increase imaging costs.”

Media outlets in the U.S. and Canada ran with the story and liberally interpreted the findings. “Doctors order more X-rays, not fewer, with computer access,” said the Washington Post headline. The New York Times screeched: “Digital Records May Not Cut Health Costs, Study Cautions.” And the CBC’s Metro Morning asked: “E-Health Costs More?”

But perhaps observers could have been more critical: Science-ish found there were several problems with this study—and research on health IT in general.

First, let’s look at the Health Affairs article. It was a cross-sectional observational study. Before you stop reading, allow Science-ish to explain: this just means the authors looked at two, non-randomized groups—doctors with electronic access and doctors without it—at one point in time. Why does this make for a weak study in this case? As Dr. Stephen Soumerai, a Harvard professor who examines health IT, pointed out, “If you’re looking at people who have digital access versus people who don’t, that’s a selection bias. Having digital access means you have other technologies, so of course you would use them more.”

Plus, with observational studies, you can’t prove causation; there could be other factors that lead to a certain outcome for one group. For example, docs who bought into e-health may be more cautious or trained differently than those who did not. Dr. Soumerai suggested a cleaner way to study the issue would have been to compare two randomly assigned groups of physicians over time. Researchers could introduce an intervention—say, electronic health records—to one group, and leave the control group in the luddite world of no digital access, then see how costs change over time. “You hope that in the comparison group, there’s no change, and in the study group, there is change,” he added.

Cost aside, the quality question was not examined in the article, either. So did the patients getting more tests need them? Did electronic access save lives? We don’t know. 

Still, despite the limitations of the study, the authors were right about one thing: we know very little about e-health and its true impact on cost and quality. “Researchers—mostly at a few flagship hospitals with cutting-edge academic computing groups that employ customized health information technology—have demonstrated that such technology can reduce total ordering of radiologic and other diagnostic tests…” they stated. But few studies—besides their poorly designed one—have looked at whether these improvements bear out in other settings, like the thousands of standalone physician practices that use “off-the-shelf” electronic health records.

This dearth of evidence hasn’t stopped policy makers in Canada and the U.S. from arguing for the adoption of health IT as some sort of cure-all for health system woes—and a pathway to cost savings. As Dr. Robyn Tamblyn, a McGill University professor who has studied electronic health records in Canada put it, “Despite the vast sums of money invested worldwide on IT in health care, there simply was no accompanying investment in rigorous evaluation, except New York State.” She went on: “The key question at this point is why? Why were there no funds invested in evaluation? Why after all the hoopla on saving health care with IT do we know so little about what works, what doesn’t, where we should put more investment and where we should pull back?”

In fact, systematic reviews (or syntheses of research on these issues) have shown, at best, mixed results when it comes to the impact of health IT on both cost and quality of care. In a systematic review on the effects of computer programs designed to improve physician decision making, the authors found that about two-thirds of the studies showed the programs improved doctors’ performances—but these studies with positive outcomes were often biased because the physicians had some stake in the technology.

Other studies point to the gaping holes in the evidence around health IT. This overview of the literature on the impact of electronic health, published in PLoS Medicine, found that “despite the wide support for eHealth technologies and the frequently made claims by policy makers when constructing business cases to raise funds for large-scale eHealth projects, there is as yet relatively little empirical evidence to substantiate many of the claims made about eHealth technologies.” Here as well, the authors point out, the tools that have been tested and were shown to be useful suffer from that generalizability problem: would they work in another setting? That’s the same question this oft-cited Annals of Internal Medicine review of the literature on the impact of health IT raises.

Before Science-ish hung up the phone with Dr. Soumerai at Harvard, he noted that some make the argument that because technology is advancing so quickly, we have to hop on board or else we’ll miss the fast moving train. Health care will be left in the dark ages. Of course no one wants that. “Many people say, ‘we can’t wait for the researchers to study these things.’ We (researchers) say if you can’t wait for us, build in a strong evaluation component.”

Science-ish is a joint project of Maclean’s, The Medical Post, and the McMaster Health Forum. Julia Belluz is the associate editor at The Medical Post. Got a tip? Seen something that’s Science-ish? Message her at julia.belluz@medicalpost.rogers.com or on Twitter @juliaoftoronto


What do we really know about e-health? Not much.

  1. The true benefit of an e-records system will only be seen if it links everyone together so that physicians will be able to access health records of everyone just by entering their healthcare number.  However, that will cause privacy issues because already there are concerns that health records from hospitals have become too easily accessible to family physician’s offices.  Perhaps that can be solved by patients signing consent forms about which doctors can access their records (with the understanding that once they make an appointment with a physician they give consent).  Further, there will have to be better record keeping and entering into the e-records by all physicians.  I am not sure that a coordinated e-record will help save money on tests but it will show if the tests were done recently and it certainly will save time that a nurse would otherwise spend calling around trying to track down paper records at different sites where the patient was seen.  In a nursing shortage, this time can certainly be better spent.

    • Another great thing about a coordinated health e-record is that if you are a community nurse and your patient is a no-show for an appointment, you can check the e-records.  They will tell you if your patients is in any emergency room or has been admitted to hospital or much worse, is deceased.
      It also provides nurses in the ER with the community programs the patient is involved with so they can have the community nurse come and see the patient if they are waiting alone in the ER.

  2. Very interesting.  I agree with the author that the selection bias is very problematic in this case, I think it’s unquestionable that the doctors who have made the effort to obtain medical imaging would also have made the effort to make use of other medical technologies as well.

  3. I have heard from people who ought to know that experts within the federal government have also conlcuded that, while electronic health records may improve health outcomes, there is no evidence showing that they will reduce costs. As might be expeceted, these experts have been muzzled.

    • Improving health outcomes should be a big enough benefit to deem ehealth records worthy.  I am not sure why people though they would save so much money….especially when most of the systems are  not even coordinated.  Yes, a coordinated system would ensure that people could not “doctor shop” and that tests would not be repeated or prescriptions foolishly given out to people with addictions but if the e-records are not coordinated so that each health site has access and are not complete, so that all care given at each health site is detailed, how could they be expected to generate real cost savings?

    •  The provinces are responsible for health care.  The federal govt has no interest.

  4. Ontario’s eHealth program has cost us – How much? $3 billion + so far and nothing to show. How much more could be done with that kind of money actually going into something that will benefit people – other than these so called consultants.

    Does the program that Sick Kids use work? – if so make some minor changes and use it province-wide and save the money.

    Just think we could have a lower the provincial deficit by $3 billion dollars.

Your email address will not be published. Required fields are marked *