AI Science Tracker
EN

Judge-labeled invalid coding traces were longer in two related models

Shanghai Jiao Tong University · 26 Oct 2026 · computer science

Discovery

Judge-labeled invalid coding traces were longer in two related models

Source check: Source checked. Not independent scientific confirmation.

Finding: Code-generation reasoning traces labeled invalid by a model-judge ensemble averaged 17,163 tokens for DeepSeek-R1-0528, compared with 7,102 for valid-labeled traces. Its distilled 8B variant showed the same direction. These labels concern reasoning coherence and are explicitly distinct from whether the final code passes functional tests.

Technology: DeepSeek-R1-0528; DeepSeek-R1-0528-Qwen3-8B

Family DeepSeek: profile · models page

Publication: Preprint · Test context: retrospective or benchmark. Catalog categories, not an evidence grade.

AI contribution
DeepSeek R1 models generated traces; DeepSeek-V3.2, Gemini-3-Flash and GLM-5.1 judged their validity, while learned semantic annotators classified reasoning actions.
Test and result
Researchers collected traces under matched prompting and decoding. Three model judges each rated trace validity three times; confidence-weighted votes labeled high-confidence cases and two human annotators adjudicated the remainder. The reported length contrast groups these trace-validity labels, not final code correctness.
Limitation / still unknown
Validity judgments include coherence and unnecessary detours, which may themselves relate to trace length; the contrast does not establish a causal effect of long reasoning or a functional-code failure rate. The models share a lineage, selected complete traces exclude incomplete or unparsable responses, and verbal traces need not faithfully reflect internal reasoning.
Evidence sources and annotation context

Evidence context (descriptions follow the cited sources; test type is a catalog classification):

Evidence sources: https://arxiv.org/abs/2610.34943v1 · https://arxiv.org/html/2610.34943v1

Source checked · editorial details

The source page was checked against this entry. Independent confirmation is not assessed by this label.

Source type: paper. Paper: linked separately.

Review date: 21 Nov 2026. Review scope: v1 task/model setup, trace taxonomy and §§4.1–4.2 differences by outcome.

No follow-up recorded; this is not a claim that no follow-up exists. Follow-up review scope and date are not recorded. No source-supported relationships recorded.

Editorial assessment: Impact 2/5 · Validation 2/5 · Novelty 2/5. Impact and novelty are editorial judgments; validation follows the annotated test type where one exists. Not an evidence grade. Scoring criteria.

Open in the full catalog for filters, charts and related discoveries. Browse computer science discoveries.

Related catalog entries

Links use documented relationships where available, plus shared catalog context. A shared topic or model family does not establish a scientific dependency.