AI Science Tracker
EN

Coding agents broke verified site behavior after completing requested changes

Beihang University; IQuest Research; Huawei Technologies · 22 Oct 2026 · computer science

Discovery

Coding agents broke verified site behavior after completing requested changes

Source check: Source checked. Not independent scientific confirmation.

Finding: In 107 of 586 benchmark runs that completed the requested website change and had comparable baseline checks, the coding agent broke an already verified browser state. In 78 of those 586 runs, a continuity check still failed at the end.

Technology: Coding agents evaluated with ProductWebBench

Publication: Preprint · Test context: retrospective or benchmark. Catalog categories, not an evidence grade.

AI contribution
Thirteen language-model systems implement requested code changes; browser checks on baseline and altered repositories expose regressions.
Test and result
The study replayed checks against untouched repositories and states that each run had previously passed. The 586-run subset had evidence from both comparisons.
Limitation / still unknown
The rates are specific to the completing runs with both baselines available, and do not estimate production software regression rates.
Evidence sources and annotation context

Evidence context (descriptions follow the cited sources; test type is a catalog classification):

Evidence sources: https://arxiv.org/abs/2610.31118 · https://arxiv.org/html/2610.31118v1

Source checked · editorial details

The source page was checked against this entry. Independent confirmation is not assessed by this label.

Source type: paper. Paper: linked separately.

Review date: 21 Nov 2026. Review scope: arXiv v1 controlled task construction, §§4–5, Table 4 and Appendix C provenance comparison.

No follow-up recorded; this is not a claim that no follow-up exists. Follow-up review scope and date are not recorded. No source-supported relationships recorded.

Editorial assessment: Impact 2/5 · Validation 2/5 · Novelty 2/5. Impact and novelty are editorial judgments; validation follows the annotated test type where one exists. Not an evidence grade. Scoring criteria.

Open in the full catalog for filters, charts and related discoveries. Browse computer science discoveries.

Related catalog entries

Links use documented relationships where available, plus shared catalog context. A shared topic or model family does not establish a scientific dependency.