{"id":7714252,"date":"2025-10-07T21:39:41","date_gmt":"2025-10-07T21:39:41","guid":{"rendered":"https:\/\/orionhealth.com\/?p=7714252"},"modified":"2026-01-29T19:37:02","modified_gmt":"2026-01-29T19:37:02","slug":"the-ai-credibility-crisis-why-healthcare-software-lacks-peer-review","status":"publish","type":"post","link":"https:\/\/orionhealth.com\/nz\/blog\/the-ai-credibility-crisis-why-healthcare-software-lacks-peer-review\/","title":{"rendered":"The AI Credibility Crisis: Why Healthcare Software Lacks Peer Review"},"content":{"rendered":"

Walk through any health tech conference and you\u2019ll hear it: AI is transforming healthcare.<\/em> Algorithms can read scans, detect sepsis, and even predict who is likely to miss their appointments. The rhetoric is intoxicating.<\/p>\n\n\n\n

Yet behind the buzzwords lies an uncomfortable truth: the majority of AI systems being piloted or sold into hospitals have never been peer-reviewed, independently validated, or clinically tested in the real world. A credibility gap exists, disguised as innovation.<\/p>\n\n\n\n

The Evidence Vacuum in Healthcare AI<\/strong><\/h2>\n\n\n\n

In traditional medicine, no drug, device, or diagnostic tool enters practice without rigorous evidence, controlled trials, peer review, and post-market surveillance. But healthcare AI operates in an ethical and scientific grey zone.<\/p>\n\n\n\n

A systematic review identified only 44 peer-reviewed studies<\/strong> on AI in healthcare as of mid-2023, highlighting that, while academic validation is increasing, it remains sparse.<\/p>\n\n\n\n

\"\"
What Can We Realistically Expect from AI to Deliver in Practice?<\/strong>
TMF mapped the expanding universe of AI use cases in healthcare \u2014 from speculative \u201con the horizon\u201d ideas to \u201csafe bets\u201d backed by strong evidence.<\/em>
Source: The Medical Futurist Institute<\/a><\/figcaption><\/figure>\n\n\n\n

Only a handful of applications, such as AI-assisted radiology analysis and clinical documentation, have reached the \u201csafe bet\u201d category. Most others, from predictive analytics to digital twins, remain speculative or risky, underscoring a systemic validation deficit.<\/p>\n\n\n\n

Regulatory Approval \u2260 Clinical Validation<\/strong><\/h2>\n\n\n\n

Regulatory approval often gets mistaken for evidence of benefit. The U.S. FDA has cleared more than a thousand AI-enabled devices, but most approvals were based on \u201csubstantial equivalence\u201d<\/em> \u2014 proving a product works like<\/em> something else, not necessarily better<\/em>.<\/p>\n\n\n\n

In other words, AI may be entering healthcare without proving it improves patient outcomes.<\/p>\n\n\n\n

And as the World Economic Forum\u2019s Future-proofing AI in Health<\/em> report bluntly states: \u201cWithout deliberate investment in high-quality evidence, we risk overlooking the most valuable use cases and funding the most dangerous ones.\u201d<\/p>\n\n\n\n

The Trust Deficit: When AI Fails to Earn Confidence<\/strong><\/h2>\n\n\n\n

This crisis of validation feeds an even more profound crisis of trust. The KPMG and University of Queensland Global Trust in AI Study found that while 85% of people recognise AI\u2019s potential, only half believe the benefits outweigh the risks.<\/p>\n\n\n\n

Healthcare remains the most trusted AI domain, but only when clinicians retain oversight. A 2025 JAMA Network study revealed that two-thirds of Americans don\u2019t trust their health systems to use AI responsibly, especially among those who\u2019ve experienced bias in care.<\/p>\n\n\n\n

Trust isn\u2019t just ethical, it\u2019s operational. Without peer review and transparency, AI becomes a black box, forcing clinicians and patients to take its recommendations on faith.<\/p>\n\n\n\n

\"Top

Regulatory actions that would increase trust in AI and the likelihood clinicians would adopt AI tools
Source: American Medical Association \u2013 Physician Sentiment Report<\/a><\/figcaption><\/figure>\n\n\n\n

Why the Evidence Gap Persists<\/strong><\/h2>\n\n\n\n

So why does the AI evidence vacuum remain so wide? There are three main culprits<\/strong>:<\/p>\n\n\n\n

    \n
  1. Commercial secrecy<\/strong> \u2013 Vendors often hide training data and validation methods behind proprietary walls. Transparency clashes with the venture capital mantra of \u201cmove fast and sell first.\u201d<\/em><\/li>\n\n\n\n
  2. Academic inertia<\/strong> \u2013 Clinical researchers are still adapting trial designs to evaluate algorithms that are continually evolving.<\/li>\n\n\n\n
  3. Policy lag<\/strong> \u2013 Most investment flows into product development, not the data and governance frameworks<\/em> required for safe deployment.<\/li>\n<\/ol>\n\n\n\n

    The result? A perverse incentive loop where startups overpromise, hospitals pilot untested tools for publicity, and investors reward speed over rigour. In this environment, peer review becomes an inconvenience rather than a necessity.<\/p>\n\n\n\n

    \"\"
    7 Assessment Clusters of the AI for IMPACTS Framework<\/strong>
    The IMPACTS framework evaluates AI across seven domains \u2014 from interoperability and governance to cost, safety, and trust \u2014 offering a roadmap for responsible implementation.<\/em>
    Source:     The Medical Futurist Institute<\/a><\/figcaption><\/figure>\n\n\n\n

    Rebuilding Trust Through Evidence<\/strong><\/h2>\n\n\n\n

    To restore confidence, peer review must become healthcare AI\u2019s immune system, filtering out false positives before they infect clinical practice. Three key actions can help rebuild this foundation:<\/p>\n\n\n\n

      \n
    1. Mandate transparency.<\/strong>\n