AfriSafe-Eval

Tebogo — the core finding here is worth taking seriously. Writing the prompts natively instead of translating them, and then seeing a ~37-point isiXhosa/isiZulu gap survive across four models, is the kind of result that's genuinely hard to dismiss. The honest weak point is the measurement layer underneath it.

Your harmful-compliance numbers are only as trustworthy as the classifier producing them, and that classifier was validated on 40 responses with what looks like a single annotator and no per-language breakdown. So the first thing I'd want is a second labeler on a bigger sample and a Cohen's kappa (or Gwet's AC1, given the class imbalance) — without an agreement number, reviewers can't tell signal from labeling noise. Report per-class specificity and the confusion matrix too, not just aggregate accuracy; a rule-based labeler that only reads the first ~250 characters will miss late refusals and corrective framing, and there's no reason to assume it misses them evenly across five languages.

Two smaller things. You sampled each model once, so there are no confidence intervals on any rate. And the four-model set was picked off free Huawei credits — worth stating plainly so nobody reads it as a designed comparison.

The benchmark created is solid and the results - differences with English - are striking. I have a concern about the main finding - that cross lingual safety gaps in LLMs are not well explained by resource availability on its own. Is representation in CC a great proxy for the dataset frontier models are trained on? Both languages have such low representation in CC that adding even small amounts of (ideally high quality) data for either language from any other source can greatly affect total representation. Would be more conclusive if this could be run on open source models that make all their training data public (Pythia? OLMo 2?) + some analysis on the training data to support this.

Interesting findings on divergence in harmful response rate between isiXhosa (61.2%) and isiZulu (23.8%)!

One reason this could happen is if there are differences in prompts introduced through translation: isiXhosa-vs-isiZulu could differ because the isiXhosa prompts are simply more concrete, more fluent, or push harder, not because the models are less safe in isiXhosa.

Consider doing control experiments (e.g. having a human rate prompts based on difficulty or harm) to isolate whether there is authorship variance.

Well done. I especially liked that the prompts were natively written rather than translated, and that the human labeled validation sample was included, which adds credibility compared with fully automated evaluations. The isiXhosa versus isiZulu finding is particularly interesting and novel.