Coding agents broke verified site behavior after completing requested changes
Source check: Source checked. Not independent scientific confirmation.
Finding: In 107 of 586 benchmark runs that completed the requested website change and had comparable baseline checks, the coding agent broke an already verified browser state. In 78 of those 586 runs, a continuity check still failed at the end.
Technology: Coding agents evaluated with ProductWebBench
Publication: Preprint · Test context: retrospective or benchmark. Catalog categories, not an evidence grade.
- AI contribution
- Thirteen language-model systems implement requested code changes; browser checks on baseline and altered repositories expose regressions.
- Test and result
- The study replayed checks against untouched repositories and states that each run had previously passed. The 586-run subset had evidence from both comparisons.
- Limitation / still unknown
- The rates are specific to the completing runs with both baselines available, and do not estimate production software regression rates.
Evidence sources and annotation context
Evidence context (descriptions follow the cited sources; test type is a catalog classification):
Evidence sources: https://arxiv.org/abs/2610.31118 · https://arxiv.org/html/2610.31118v1
Source checked · editorial details
The source page was checked against this entry. Independent confirmation is not assessed by this label.
Source type: paper. Paper: linked separately.
Review date: 21 Nov 2026. Review scope: arXiv v1 controlled task construction, §§4–5, Table 4 and Appendix C provenance comparison.
No follow-up recorded; this is not a claim that no follow-up exists. Follow-up review scope and date are not recorded. No source-supported relationships recorded.
Editorial assessment: Impact 2/5 · Validation 2/5 · Novelty 2/5. Impact and novelty are editorial judgments; validation follows the annotated test type where one exists. Not an evidence grade. Scoring criteria.