The statement: “Our research raises real concerns about whether health information technology is going to be the answer to reducing costs.”—New York Times, Mar. 5, 2012
In a new study published in Health Affairs a group of American researchers looked at the records of patient visits to a sample of office-based physicians to find out whether doctors with electronic access to imaging results ordered more tests than doctors without them. Surprise, surprise: the researchers found that the digital doctors were 40 to 70 per cent more likely to order imaging, and even blood tests, than doctors without access.
Although the survey did not collect data on payments for those tests, the investigators stated in a follow-up about the study, “It’s hard to imagine how a 40 to 70 per cent increase in testing could fail to increase imaging costs.”
Media outlets in the U.S. and Canada ran with the story and liberally interpreted the findings. “Doctors order more X-rays, not fewer, with computer access,” said the Washington Post headline. The New York Times screeched: “Digital Records May Not Cut Health Costs, Study Cautions.” And the CBC’s Metro Morning asked: “E-Health Costs More?”
But perhaps observers could have been more critical: Science-ish found there were several problems with this study—and research on health IT in general.
First, let’s look at the Health Affairs article. It was a cross-sectional observational study. Before you stop reading, allow Science-ish to explain: this just means the authors looked at two, non-randomized groups—doctors with electronic access and doctors without it—at one point in time. Why does this make for a weak study in this case? As Dr. Stephen Soumerai, a Harvard professor who examines health IT, pointed out, “If you’re looking at people who have digital access versus people who don’t, that’s a selection bias. Having digital access means you have other technologies, so of course you would use them more.”
Plus, with observational studies, you can’t prove causation; there could be other factors that lead to a certain outcome for one group. For example, docs who bought into e-health may be more cautious or trained differently than those who did not. Dr. Soumerai suggested a cleaner way to study the issue would have been to compare two randomly assigned groups of physicians over time. Researchers could introduce an intervention—say, electronic health records—to one group, and leave the control group in the luddite world of no digital access, then see how costs change over time. “You hope that in the comparison group, there’s no change, and in the study group, there is change,” he added.
Cost aside, the quality question was not examined in the article, either. So did the patients getting more tests need them? Did electronic access save lives? We don’t know.
Still, despite the limitations of the study, the authors were right about one thing: we know very little about e-health and its true impact on cost and quality. “Researchers—mostly at a few flagship hospitals with cutting-edge academic computing groups that employ customized health information technology—have demonstrated that such technology can reduce total ordering of radiologic and other diagnostic tests…” they stated. But few studies—besides their poorly designed one—have looked at whether these improvements bear out in other settings, like the thousands of standalone physician practices that use “off-the-shelf” electronic health records.
This dearth of evidence hasn’t stopped policy makers in Canada and the U.S. from arguing for the adoption of health IT as some sort of cure-all for health system woes—and a pathway to cost savings. As Dr. Robyn Tamblyn, a McGill University professor who has studied electronic health records in Canada put it, “Despite the vast sums of money invested worldwide on IT in health care, there simply was no accompanying investment in rigorous evaluation, except New York State.” She went on: “The key question at this point is why? Why were there no funds invested in evaluation? Why after all the hoopla on saving health care with IT do we know so little about what works, what doesn’t, where we should put more investment and where we should pull back?”
In fact, systematic reviews (or syntheses of research on these issues) have shown, at best, mixed results when it comes to the impact of health IT on both cost and quality of care. In a systematic review on the effects of computer programs designed to improve physician decision making, the authors found that about two-thirds of the studies showed the programs improved doctors’ performances—but these studies with positive outcomes were often biased because the physicians had some stake in the technology.
Other studies point to the gaping holes in the evidence around health IT. This overview of the literature on the impact of electronic health, published in PLoS Medicine, found that “despite the wide support for eHealth technologies and the frequently made claims by policy makers when constructing business cases to raise funds for large-scale eHealth projects, there is as yet relatively little empirical evidence to substantiate many of the claims made about eHealth technologies.” Here as well, the authors point out, the tools that have been tested and were shown to be useful suffer from that generalizability problem: would they work in another setting? That’s the same question this oft-cited Annals of Internal Medicine review of the literature on the impact of health IT raises.
Before Science-ish hung up the phone with Dr. Soumerai at Harvard, he noted that some make the argument that because technology is advancing so quickly, we have to hop on board or else we’ll miss the fast moving train. Health care will be left in the dark ages. Of course no one wants that. “Many people say, ‘we can’t wait for the researchers to study these things.’ We (researchers) say if you can’t wait for us, build in a strong evaluation component.”
Science-ish is a joint project of Maclean’s, The Medical Post, and the McMaster Health Forum. Julia Belluz is the associate editor at The Medical Post. Got a tip? Seen something that’s Science-ish? Message her at firstname.lastname@example.org or on Twitter @juliaoftoronto