Deployed delirium model retains discrimination at a second hospital
Source check: Source checked. Not independent scientific confirmation.
Finding: A frozen random-forest system combining clinical data and an NLP risk score was evaluated during deployment at a second Mount Sinai hospital. Retrospective workflow analysis reported AUC 0.91 and positive predictive value 25.9%; targeted assessments had a higher delirium yield than a historical period. Selective verification and changing workflows prevent a causal patient-benefit interpretation.
Technology: Random forest with structured clinical features and an NLP-derived risk score
Publication: Peer-reviewed paper · Test context: retrospective or benchmark. Catalog categories, not an evidence grade.
- AI contribution
- A trained random forest fused structured/semistructured clinical information with an NLP-derived note score to prioritize live clinical delirium assessments; the study measured its within-system transportability and operational assessment yield.
- Test and result
- Frozen model deployed without local retraining at Mount Sinai Morningside in November 2024. Retrospective analysis compared 9,020 admissions there with 21,186 contemporaneous development-hospital admissions and 2,889 historical predeployment admissions. At the second hospital, reported AUC was 0.91 and positive predictive value 25.9%. Only 2,649 of 9,020 admissions received CAM assessments; the rest relied on diagnosis codes. Mean monthly yield among CAM assessments rose from 9.23% to 19.59%.
- Limitation / still unknown
- Same-health-system transportability, not independent-health-system validation. Model-guided CAM assessment creates incomplete verification and incorporation biases; remaining reference labels use diagnosis codes. Site thresholds differ, calibration was not the primary endpoint and false positives were frequent. Historical2018–2019 versus2024–2025 comparisons are confounded and measure yield per assessment, not incidence or improved outcomes. Linked invited commentary emphasizes these interpretation limits.
Evidence sources and annotation context
Evidence context (descriptions follow the cited sources; test type is a catalog classification):
Evidence sources: https://pmc.ncbi.nlm.nih.gov/articles/PMC13735856/
Source checked · editorial details
The source page was checked against this entry. Independent confirmation is not assessed by this label.
Source type: paper. Paper: not linked in this catalog; this does not mean no paper exists.
Review date: 21 Nov 2026. Review scope: Read actual primary model operation, frozen deployment, cohort/reference and workflow Methods, performance and yield Results, limitations, plus publisher-linked invited-commentary context. Publication date follows original publisher and PMC Article Information November 4; indexed November 2 is retained separately without a first-ever-version claim. Exact primary identity/date and fresh primary PubMed correction/retraction relationships checked..
No follow-up recorded; this is not a claim that no follow-up exists. Follow-up review scope and date are not recorded. No source-supported relationships recorded.
Editorial assessment: Impact 2/5 · Validation 2/5 · Novelty 2/5. Impact and novelty are editorial judgments; validation follows the annotated test type where one exists. Not an evidence grade. Scoring criteria.