AI Science Tracker
EN

Rank compression followed delayed generalization in tested Transformers

Lossfunk · 26 Oct 2026 · computer science

Discovery

Lossfunk

Rank compression followed delayed generalization in tested Transformers

Source check: Source checked. Not independent scientific confirmation.

Finding: In 29 one-layer Transformer runs that developed delayed generalization on modular arithmetic, the steepest effective-rank drop followed the rise in test accuracy by about 900 training steps. In a separate intervention, restricting embedding rank prevented delayed generalization in 15 of 20 runs.

Technology: One-layer Transformers; rank-constrained training

Publication: Preprint · Test context: direct test. Catalog categories, not an evidence grade.

AI contribution
Trained modular-arithmetic Transformers and matched factorized controls serve as instruments for evaluating weight rank and generalization timing.
Test and result
The paper used a predefined test-accuracy threshold and compared rank restrictions with factorized full-rank models across seeds.
Limitation / still unknown
Results concern small arithmetic models. Timing and intervention findings do not exclude other roles for representation compression in other training regimes.
Evidence sources and annotation context

Evidence context (descriptions follow the cited sources; test type is a catalog classification):

Evidence sources: https://arxiv.org/abs/2610.35112v1 · https://arxiv.org/html/2610.35112v1

Source checked · editorial details

The source page was checked against this entry. Independent confirmation is not assessed by this label.

Source type: paper. Paper: linked separately.

Review date: 21 Nov 2026. Review scope: v1 one-layer Transformer results, rank intervention and Appendix E.

No follow-up recorded; this is not a claim that no follow-up exists. Follow-up review scope and date are not recorded. No source-supported relationships recorded.

Editorial assessment: Impact 2/5 · Validation 3/5 · Novelty 3/5. Impact and novelty are editorial judgments; validation follows the annotated test type where one exists. Not an evidence grade. Scoring criteria.

Open in the full catalog for filters, charts and related discoveries. Browse computer science discoveries.