The Real-World Data Dilemma — Part 2: Medicine’s new black box

Medical decisions are increasingly shaped by massive datasets extracted from patient records. But who verifies how those data are processed before researchers ever see them?

Massive datasets containing the health records of millions of patients now underpin thousands of studies on drug safety, vaccines and treatment effectiveness.

Regulators are increasingly relying on them too.

Last December, then-FDA Commissioner Marty Makary announced plans to expand the agency’s use of “real-world evidence” in regulatory decisions.

“We’re removing unnecessary barriers that have prevented us from using powerful real-world evidence to get life-changing treatments to patients faster,” Makary said.

Dr Marty Makary, former FDA commissioner

There are valid reasons for the shift. Electronic health records covering millions of patients can reveal rare side effects and long-term outcomes that clinical trials may be too small or too short to detect.

But there is a problem. Much of this research relies on enormous, heavily processed datasets that researchers and independent reviewers cannot readily check against the original hospital records.

The danger became clear during the Surgisphere scandal in 2020, when the database underpinning a major Covid-19 study that influenced health policy turned out to be fake.

Six years later, the same fundamental problem remains.

If neither researchers nor independent reviewers can trace the data back to the original medical records, how do we know they are reliable?