A perfect Hausa safety proxy masked low-quality responses in one model assessment
Source check: Source checked. Not independent scientific confirmation.
Finding: In a 500-prompt Hausa safety evaluation, Gemini classified every base Qwen2.5-7B-Instruct response as safe—the paper’s refusal-rate proxy—but rated 96.2% low quality. Model variants changed benign-request over-refusal and judged linguistic quality by different amounts.
Technology: Qwen2.5-7B-Instruct variants; Gemini evaluator
Family Gemini: profile · models page
Publication: Preprint · Test context: retrospective or benchmark. Catalog categories, not an evidence grade.
- AI contribution
- The tested learned models produced Hausa responses; the learned judge supplied separate safety and language-quality ratings rather than using refusal counts alone.
- Test and result
- Five model variants answered 500 harmful and 200 benign Hausa prompts. One Gemini judge rated safety, helpfulness and linguistic quality using a structured rubric.
- Limitation / still unknown
- Many prompts were translated, and the ratings did not receive independent native-speaker validation. The results concern this benchmark and its one automated judge. The harmful-prompt judge returns SAFE/UNSAFE; its reported refusal-rate proxy does not independently authenticate a coherent Hausa refusal.
Evidence sources and annotation context
Evidence context (descriptions follow the cited sources; test type is a catalog classification):
Evidence sources: https://arxiv.org/abs/2610.37245v1 · https://arxiv.org/html/2610.37245v1
Source checked · editorial details
The source page was checked against this entry. Independent confirmation is not assessed by this label.
Source type: paper. Paper: linked separately.
Review date: 21 Nov 2026. Review scope: §§4–6, evaluation conditions, Table 3 and single-judge limits.
No follow-up recorded; this is not a claim that no follow-up exists. Follow-up review scope and date are not recorded. No source-supported relationships recorded.
Editorial assessment: Impact 2/5 · Validation 2/5 · Novelty 2/5. Impact and novelty are editorial judgments; validation follows the annotated test type where one exists. Not an evidence grade. Scoring criteria.