Reality check: how AI performs in real-world clinical settings?

Reality check for AI in clinics

Regardless of whether people support or disavow the AI hype, many AI technologies have already received FDA approval, been deployed in clinics or directly to individuals, and begun to reshape healthcare. How well do these AI technologies perform in real-world clinical practice? It’s time to do a reality check. ⏱️

A recent study in Nature's npj Digital Medicine demonstrated that an AI algorithm significantly improved junior clinicians' performance when diagnosing pediatric intussusception using ultrasound. With AI’s assistance, junior clinicians’ performances become comparable to those of senior clinicians.

It is a typical “AI validation” paper where the authors claim that AI performance is either superior to or comparable with that of clinicians. However, we know the performance of AI algorithms often drifts in real-world settings.

One study checked the performance of two AI-powered skin cancer diagnosis algorithms in real-world clinical settings. One algorithm is equivalent to senior clinicians’ diagnoses and superior to junior clinicians, and the other algorithm is inferior to senior clinicians and superior to junior clinicians. Both are worse than the original assessment. (The result was published in the Lancet Digital Health.) This study suggests that AI's real-world performance might not live up to the claims of the original "validation study," and we should exercise caution when using them.

Nevertheless, even if they may not outperform senior clinicians, the improvement over junior clinicians can be valuable, especially for individuals with low socioeconomic status seeking access to high-quality healthcare comparable to that provided by senior and expert clinicians. For example, a study recently published on npj digital medicine reported a 40% higher clinic productivity when using an autonomous AI system, LumineticsCore, to diagnose diabetic retinopathy and diabetic macular edema. AI systems are usually easier to deploy, and the increase in clinic productivity by AI can also improve patients' access to clinicians. (The result came from a double-masked and cluster-randomized clinical trial in Bangladesh. The LumineticsCore system received FDA approval in 2018 and has been on the market for several years.)

FDA-approved AI/ML devices

The FDA maintains a list of AI/ML-enabled medical devices approved for marketing in the United States (up to October 05, 2022). Most of these AI technologies are image-based. With the recent success of large language models like ChatGPT, I believe we’ll soon see a new wave of NLP-based (and even multimodal) AI technologies used in clinics!

P.S. I asked ChatGPT to analyze the FDA’s list. It gave me accurate results and insights within one minute! Here are some figures copied directly from ChatGPT’s output. Amazing! 🎉

Pharma Proteomics Project

For those who are interested in genetics, omics, bioinformatics, or early drug discovery, the UK Biobank Pharma Proteomics Project published three Nature papers last week. This collaborative project, involving the UK Biobank and thirteen biopharmaceutical companies, aims to characterize the plasma proteomic profiles of 54,219 UK Biobank participants. (Job seekers may find the author lists useful. Many of them work on data science and AI.)

1. Plasma proteomic associations with genetics and health in the UK Biobank

2. Large-scale plasma proteomics comparisons through genetics and disease associations

3. Rare variant associations with plasma protein levels in the UK Biobank

Business news about AI + drug discovery

• AI drug hunter Exscientia chops down 'rapidly emerging pipeline' to focus on 2 main oncology programs (Fierce Biotech), while just last month, Merck KGaA Enters Deal With BenevolentAI, Exscientia for AI-Based Drug Discovery (MarketWatch)

• Iambic ups rhythm in AI-enabled race to clinic, revealing $100M round and Nvidia pact (Fierce Biotech)

##########

If you find the newsletter helpful, please consider:

• 🔊 Sharing the newsletter with other people

• 👍 Upvoting on Product Hunt

• 📧 Sending any feedback, suggestions, and questions by directly replying to this email or writing reviews on Product Hunt

• 🙏 Supporting us with a cup of coffee.

Thanks, and see you next time!