AI Science Tracker
EN

A confidence probe rose on repeated wrong answers in Qwen2.5

Hanoi University of Science and Technology · 26 Oct 2026 · computer science

Discovery

A confidence probe rose on repeated wrong answers in Qwen2.5

Source check: Source checked. Not independent scientific confirmation.

Finding: A probe trained to read answer correctness from first-turn activations gave higher scores when Qwen2.5-7B repeated an incorrect answer. On questions held out from probe training, scores rose by about 0.32 after either a re-ask or agreement. Other confidence measures disagreed, and the pattern did not reproduce consistently across tested models.

Technology: Qwen2.5-7B and other language models; fitted linear activation probe

Publication: Preprint · Test context: retrospective or benchmark. Catalog categories, not an evidence grade.

AI contribution
The fitted linear readout made it possible to compare answer-related signals from learned model activations with confidence estimates from the same dialogue.
Test and result
A frozen first-turn probe was compared against three other confidence measures. The post hoc check refitted the probe in five folds grouped by question and compared it with probes fit to shuffled labels.
Limitation / still unknown
The strongest result concerns short-answer tasks in Qwen2.5-7B. A cleaned Llama-3 analysis provided support; four other models failed to reproduce the pattern consistently. Probe scores do not directly measure belief.
Evidence sources and annotation context

Evidence context (descriptions follow the cited sources; test type is a catalog classification):

Evidence sources: https://arxiv.org/abs/2610.35694v1 · https://arxiv.org/html/2610.35694v1

Source checked · editorial details

The source page was checked against this entry. Independent confirmation is not assessed by this label.

Source type: paper. Paper: linked separately.

Review date: 21 Nov 2026. Review scope: §§3 and 5–8, held-out grouped analysis, and front matter.

No follow-up recorded; this is not a claim that no follow-up exists. Follow-up review scope and date are not recorded. No source-supported relationships recorded.

Editorial assessment: Impact 2/5 · Validation 2/5 · Novelty 2/5. Impact and novelty are editorial judgments; validation follows the annotated test type where one exists. Not an evidence grade. Scoring criteria.

Open in the full catalog for filters, charts and related discoveries. Browse computer science discoveries.