AI Science Tracker
EN

Aggregate Accuracy Hides Diagnosis-Specific Failure in Medical Vision-Language Models

Not stated in the reviewed material · 11 Nov 2026 · medicine

Discovery

Aggregate Accuracy Hides Diagnosis-Specific Failure in Medical Vision-Language Models

Source check: Source checked. Not independent scientific confirmation.

Finding: On 6,048 balanced ChestX-ray14 cases, HuatuoGPT-Vision and Qwen2.5-VL detected none of 432 pneumothorax-positive cases; comparison models detected 38% and 67%. This model-specific failure also appeared at larger model scale. The failure replicated on an independent SIIM-ACR set with radiologist-drawn masks: the same models detected 0/432 confirmed pneumothoraces, including 0/144 largest lesions. Retrospective diagnostic evaluation only; no clinical benefit established.

Technology: Medical vision-language models directly answered per-finding questions from chest radiographs; the study evaluated their diagnosis-specific recall, confidence and internal representations.

Publication: Preprint · Test context: retrospective or benchmark. Catalog categories, not an evidence grade.

AI contribution
Medical vision-language models directly answered per-finding questions from chest radiographs; the study evaluated their diagnosis-specific recall, confidence and internal representations.
Test and result
On 6,048 balanced ChestX-ray14 cases, HuatuoGPT-Vision and Qwen2.5-VL detected none of 432 pneumothorax-positive cases; comparison models detected 38% and 67%. This model-specific failure also appeared at larger model scale.
Limitation / still unknown
The failure replicated on an independent SIIM-ACR set with radiologist-drawn masks: the same models detected 0/432 confirmed pneumothoraces, including 0/144 largest lesions. Retrospective diagnostic evaluation only; no clinical benefit established.
Evidence sources and annotation context

Evidence context (descriptions follow the cited sources; test type is a catalog classification):

Evidence sources: https://arxiv.org/html/2611.14975v1

Source checked · editorial details

The source page was checked against this entry. Independent confirmation is not assessed by this label.

Source type: paper. Paper: not linked in this catalog; this does not mean no paper exists.

Review date: 22 Nov 2026. Review scope: Read the primary preprint Methods and Results; its v1 record dates the result to 11 November 2026. The study directly evaluates medical VLMs on a specific diagnostic question: two models missed every pneumothorax in a balanced ChestX-ray14 audit while other models detected a substantial share; the failure replicated on an independent SIIM-ACR set with radiologist masks. Exact title/result comparison found no matching entry in either supplied catalog. This is retrospective diagnostic validation, not evidence of clinical benefit; ChestX-ray14 labels and binary prompts limit interpretation..

No follow-up recorded; this is not a claim that no follow-up exists. Follow-up review scope and date are not recorded. No source-supported relationships recorded.

Editorial assessment: Impact 3/5 · Validation 2/5 · Novelty 3/5. Impact and novelty are editorial judgments; validation follows the annotated test type where one exists. Not an evidence grade. Scoring criteria.

Open in the full catalog for filters, charts and related discoveries. Browse medicine discoveries.

Related catalog entries

Links use documented relationships where available, plus shared catalog context. A shared topic or model family does not establish a scientific dependency.