Sales agents applied policies more reliably in static probes than interactive tasks
Source check: Source checked. Not independent scientific confirmation.
Finding: In controlled sales scenarios based on the same policies as static questions, eight language-model backbones showed 89.9–99.1% compliance on the static probes and 3.0–17.3% on interactive tasks. Scoring the actions also required a verified safe ending, so the gap reflects several differences between conditions.
Technology: Qwen3-8B and other language-agent backbones; paired static and interactive evaluation
Publication: Preprint · Test context: direct test. Catalog categories, not an evidence grade.
- AI contribution
- Eight language-model backbones reason about sales policies and choose actions in matched scripted interactions.
- Test and result
- The confirmatory study included 21,888 runs over 19 semantic cores and 12 configurations. The comparison of ranges covers the eight backbone configurations.
- Limitation / still unknown
- The scripted Chinese-language setting and stricter action criteria limit interpretation. The gap does not estimate an effect of sales pressure or real deployment failure rates. The semantic violation scorer had no released human-calibration accuracy estimate, and shared provider-family judging was not ruled out; its hard-violation flags materially influence action scores.
Evidence sources and annotation context
Evidence context (descriptions follow the cited sources; test type is a catalog classification):
Evidence sources: https://arxiv.org/abs/2610.34185v1 · https://arxiv.org/html/2610.34185v1
Source checked · editorial details
The source page was checked against this entry. Independent confirmation is not assessed by this label.
Source type: paper. Paper: linked separately.
Review date: 21 Nov 2026. Review scope: v1 Methods, confirmatory benchmark experiment and stated measurement limitations.
No follow-up recorded; this is not a claim that no follow-up exists. Follow-up review scope and date are not recorded. No source-supported relationships recorded.
Editorial assessment: Impact 2/5 · Validation 3/5 · Novelty 2/5. Impact and novelty are editorial judgments; validation follows the annotated test type where one exists. Not an evidence grade. Scoring criteria.