AfroJailbreak-ZW: Evaluating Jailbreak Resistance in Shona
Malvin T. Machingura
A pilot study testing whether ChatGPT and Gemini are easier to jailbreak in Shona than in English. In a small test (n=5–6 per prompt type), both models complied with harmful requests more often in Shona and Shona-English code-switched prompts than in English, suggesting AI safety protections built mainly for English may not hold for Zimbabwean languages
The project has chosen a good idea of identifying and resolving jailbreak in low-resource languages. It introduces an additional metric LVI aiming to make ASR more accurate by considering the relevance and comprehension factor. However, the LLMs in test(ChatGPT/Gemini) don't claim to support Shona. So, the baseline of testing isnt accurate. Also, the methodology uses only 6 queries for testing, which is very small and can highly misguide the accuracy/orientation of the whole project.
This is a valuable and timely pilot, offering the first jailbreak‑resistance evaluation for Shona and introducing a Language Vulnerability Index that adjusts attack success rates for relevance/comprehension, which is highly pertinent for Global South safety where low‑resource languages and code‑switching are common.
The methodology—three parallel prompt sets (English, Shona, code‑switched), multiple attack categories, GPT vs Gemini comparison, and an open harness that can be extended to other Southern African languages—is well thought out for hackathon scale, but quantitative strength is limited by very small sample sizes (n=5–6), single‑rater labelling, and lack of independent back‑translation or inter‑rater reliability.
The paper is clearly written, with honest discussion of limitations and dual‑use risks, and would be strengthened by slightly more detail on how LVI is computed, a clearer plan for scaling the dataset and judge pool, and an explicit roadmap for operationalising this benchmark with local institutions and providers.
3 / 2 / 3
Useful first step for Shona AI safety. The main issue is that the sample size is very small, so the results should be treated as early evidence.
Cite this work
@misc {
title={
(HckPrj) AfroJailbreak-ZW: Evaluating Jailbreak Resistance in Shona
},
author={
Malvin T. Machingura
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


