AI Science Tracker
EN

A perfect Hausa safety proxy masked low-quality responses in one model assessment

African Institute for Mathematical Sciences; Bayero University Kano; University of Hamburg; Imperial College London · 27 Oct 2026 · computer science

Discovery

A perfect Hausa safety proxy masked low-quality responses in one model assessment

Source check: Source checked. Not independent scientific confirmation.

Finding: In a 500-prompt Hausa safety evaluation, Gemini classified every base Qwen2.5-7B-Instruct response as safe—the paper’s refusal-rate proxy—but rated 96.2% low quality. Model variants changed benign-request over-refusal and judged linguistic quality by different amounts.

Technology: Qwen2.5-7B-Instruct variants; Gemini evaluator

Family Gemini: profile · models page

Publication: Preprint · Test context: retrospective or benchmark. Catalog categories, not an evidence grade.

AI contribution
The tested learned models produced Hausa responses; the learned judge supplied separate safety and language-quality ratings rather than using refusal counts alone.
Test and result
Five model variants answered 500 harmful and 200 benign Hausa prompts. One Gemini judge rated safety, helpfulness and linguistic quality using a structured rubric.
Limitation / still unknown
Many prompts were translated, and the ratings did not receive independent native-speaker validation. The results concern this benchmark and its one automated judge. The harmful-prompt judge returns SAFE/UNSAFE; its reported refusal-rate proxy does not independently authenticate a coherent Hausa refusal.
Evidence sources and annotation context

Evidence context (descriptions follow the cited sources; test type is a catalog classification):

Evidence sources: https://arxiv.org/abs/2610.37245v1 · https://arxiv.org/html/2610.37245v1

Source checked · editorial details

The source page was checked against this entry. Independent confirmation is not assessed by this label.

Source type: paper. Paper: linked separately.

Review date: 21 Nov 2026. Review scope: §§4–6, evaluation conditions, Table 3 and single-judge limits.

No follow-up recorded; this is not a claim that no follow-up exists. Follow-up review scope and date are not recorded. No source-supported relationships recorded.

Editorial assessment: Impact 2/5 · Validation 2/5 · Novelty 2/5. Impact and novelty are editorial judgments; validation follows the annotated test type where one exists. Not an evidence grade. Scoring criteria.

Open in the full catalog for filters, charts and related discoveries. Browse computer science discoveries.

Related catalog entries

Links use documented relationships where available, plus shared catalog context. A shared topic or model family does not establish a scientific dependency.