AI Science Tracker
EN

When the Text Is Silent: Memory Shortcuts and the Validity of LLM Annotation under Evidence Insufficiency

Not stated in the abstract reviewed · 1 Nov 2026 · computer science

Discovery

When the Text Is Silent: Memory Shortcuts and the Validity of LLM Annotation under Evidence Insufficiency

Source check: Source checked. Not independent scientific confirmation.

Finding: In controlled counterfactual texts about public figures and matched control names, three commercial LLMs produced labels unsupported by the text when evidence was absent. These memory shortcuts were often factually correct about a namesake but methodologically invalid as text-grounded annotations.

Technology: Three commercial large language models

Publication: Preprint · Test context: retrospective or benchmark. Catalog categories, not an evidence grade.

AI contribution
Three commercial large language models
Test and result
Across 995 public figures and four matched control names per figure, three commercial LLMs were evaluated on 121,594 prompts each. When text evidence was absent, one model increased remembered occupations, ages and countries by 17.0, 14.9 and 6.8 percentage points relative to synthetic names.
Limitation / still unknown
The results use selected models, entities and constructed prompts; they do not quantify all annotation settings.
Evidence sources and annotation context

Evidence context (descriptions follow the cited sources; test type is a catalog classification):

Evidence sources: https://arxiv.org/abs/2611.02273

Source checked · editorial details

The source page was checked against this entry. Independent confirmation is not assessed by this label.

Source type: paper. Paper: linked separately.

Review date: 22 Nov 2026. Review scope: Primary result-bearing material for the stated method, result and evaluation, with bibliographic version information..

No follow-up recorded; this is not a claim that no follow-up exists. Follow-up review scope and date are not recorded. No source-supported relationships recorded.

Editorial assessment: Impact 3/5 · Validation 2/5 · Novelty 3/5. Impact and novelty are editorial judgments; validation follows the annotated test type where one exists. Not an evidence grade. Scoring criteria.

Open in the full catalog for filters, charts and related discoveries. Browse computer science discoveries.