What happened
A peer-reviewed scoping review published October 12 examined where and how clinical large language models have been evaluated outside English. Researchers searched PubMed, Scopus and Embase for work from January 2022 through February 2026 and included 806 studies covering 62 non-English languages. They found that evaluation is unevenly distributed across languages and regions, leaving important gaps in the evidence available for deployment decisions.
Why it matters
Before this review, a broad claim that a model worked in multiple languages could obscure how narrowly it had been tested. The mapped evidence gives buyers and clinicians a more useful question: has this model been evaluated for the language and clinical task in which it would actually be used? Uneven coverage and missing reporting make that question harder to answer, particularly where published evaluation is sparse. They warrant more relevant testing, not a conclusion that any named model is either safe or unsafe.
Evidence
Just 17 of the 806 studies, or 2.1%, came from the African and South-East Asian WHO regions, which together contain approximately 3.15 billion people. That contrast shows how little of the published evaluation came from those regions; a study count does not directly measure whether a deployed tool is safe. Only 202 studies, or 25.1%, compared English with non-English performance. Because the models, prompts and datasets differed, those comparisons cannot be pooled into one reliable measure of a language-related performance gap.
The authors also checked how clearly studies described the language-specific details needed to interpret their results. In a 100-study full-text sample, 55% reported none of four such items, including matters such as translation and cultural adaptation. They propose LINGUA as a candidate reporting framework but say it needs prospective validation. The search ended in February 2026, and the review maps published evaluations rather than measuring patient outcomes after a particular model was deployed. (nature.com)
What remains unknown
The review cannot establish whether a particular model is safe for a given language, disease or care setting. It also cannot show that better reporting will improve clinical performance or safety; the proposed framework still needs to be tested. These limits matter because identifying a gap in evaluation is not the same as demonstrating that every tool affected by it fails.
Who may be affected
Patients and clinicians using medical AI in non-English languages, and health systems, developers and evaluators deciding what evidence supports its use.
AXIONOVA connection
The review gives health systems a concrete way to scrutinize medical-AI claims: English performance, or performance on translated examinations, cannot stand in for testing in the language and clinical workflow where a tool will be used. Prospective testing of the authors’ proposed LINGUA reporting framework is the next step for assessing whether it helps make evaluations more interpretable.
What ALETHA can explain
ALETHA can explain how a scoping review differs from a pooled accuracy study, why performance in English cannot automatically be generalized to another language, and why missing validation is an evidence gap rather than proof of failure.
Action posture
Informational: The study informs evaluation standards but is not an individual medical or official safety instruction.