AI Safety Observatory for Africa
Moegamat Samsodien
Problem: LLM safety systems are built and tested almost exclusively in English. Harmful prompts in African languages routinely bypass guardrails that correctly block identical English content — a blind spot affecting 2,000+ languages and over a billion people.
What it does: An open-source platform evaluating LLM safety across African languages including Swahili, isiZulu, Afrikaans, Amharic, ChiShona, and Yorùbá. It runs adversarial prompts across multiple models (Gemini, GPT-4o, Claude, Llama) and measures whether harmful content is correctly refused in each language.
Key components:
30-prompt benchmark spanning jailbreak, hate speech, fraud, medical misinformation, and election interference — written natively, not machine-translated
Real-time risk scoring, safety classification, and language detection pipeline
Filterable audit dashboard with per-language and per-model breakdowns
Safety leaderboard surfacing guardrail gaps across languages
Core contribution: A reproducible measurement system that makes the African language LLM safety gap legible to researchers, policymakers, and model developers.
I think the lack of specific job descriptions being evaluated here is an omission. Not that I think the findings don't expose a very obvious here, but as LLMs evaluate candidates by mapping the semantic overlap between the CV and the job description - I wonder if a descriptions in those industries including things like "experience with UK regulatory compliance" or a job location as an example influences things.
To take this further into a reusable benchmark, other folks should be able to pull in the job descriptions and run regression tests on new models
Other methodology notes:
- 45 samples are a great start, but you can scale test a little more with programmatically generated identical CVs for a larger dataset fairly quickly
- Evaluations on open weight (llama3 etc) in addition to closed-mixed models (gemini, chatgpt) could be useful as their training data mixtures are better known and may expose specific gaps
Strengths: Addresses an important gap by centralizing AI safety knowledge and initiatives across Africa. The regional focus is timely and socially impactful, and the presentation clearly communicates the vision. Areas for Improvement: Strengthen the technical implementation by providing more detail on data ingestion, validation, update mechanisms, and scalability. Include measurable evaluation metrics (coverage, accuracy, user engagement) and comparisons with existing AI observatories. A clearer roadmap for governance and long-term sustainability would further strengthen the project.
The real-time monitoring dashboard introduced by this project is helpful as it creates a way to make AI safety risks more trackable.
One area that could be strengthened is the evaluation of refusal mechanisms. Testing on very few examples feels too limited to draw strong conclusions. A larger dataset, paired with a clearer scoring rubric would lead to more interesting and reliable results.
I’m also curious about over-refusals, where the system may have refused requests that should have been allowed. Evaluating both under-refusal and over-refusal would give a good picture of how well these systems work on African languages.
Cite this work
@misc {
title={
(HckPrj) AI Safety Observatory for Africa
},
author={
Moegamat Samsodien
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


