Rank compression followed delayed generalization in tested Transformers
Source check: Source checked. Not independent scientific confirmation.
Finding: In 29 one-layer Transformer runs that developed delayed generalization on modular arithmetic, the steepest effective-rank drop followed the rise in test accuracy by about 900 training steps. In a separate intervention, restricting embedding rank prevented delayed generalization in 15 of 20 runs.
Technology: One-layer Transformers; rank-constrained training
Publication: Preprint · Test context: direct test. Catalog categories, not an evidence grade.
- AI contribution
- Trained modular-arithmetic Transformers and matched factorized controls serve as instruments for evaluating weight rank and generalization timing.
- Test and result
- The paper used a predefined test-accuracy threshold and compared rank restrictions with factorized full-rank models across seeds.
- Limitation / still unknown
- Results concern small arithmetic models. Timing and intervention findings do not exclude other roles for representation compression in other training regimes.
Evidence sources and annotation context
Evidence context (descriptions follow the cited sources; test type is a catalog classification):
Evidence sources: https://arxiv.org/abs/2610.35112v1 · https://arxiv.org/html/2610.35112v1
Source checked · editorial details
The source page was checked against this entry. Independent confirmation is not assessed by this label.
Source type: paper. Paper: linked separately.
Review date: 21 Nov 2026. Review scope: v1 one-layer Transformer results, rank intervention and Appendix E.
No follow-up recorded; this is not a claim that no follow-up exists. Follow-up review scope and date are not recorded. No source-supported relationships recorded.
Editorial assessment: Impact 2/5 · Validation 3/5 · Novelty 3/5. Impact and novelty are editorial judgments; validation follows the annotated test type where one exists. Not an evidence grade. Scoring criteria.