MoMo: A Threat Corpus and Evaluation Framework for Mobile Money Fraud Resistance in Swahili, Wolof, and Hausa
Mufaro Rukuni
Mobile money platformsM-Pesa, Orange Money, and Wave among themnow mediate
nancial life for more than 500 million people across Sub-Saharan Africa, and have become
a correspondingly attractive target for social-engineering fraud conducted in local languages
and idioms that mainstream AI safety evaluation rarely covers. We present MoMo, a struc-
tured threat corpus and evaluation framework for benchmarking large language model (LLM)
resistance to generating localized mobile money fraud content in Swahili, Wolof, and Hausa.
MoMo organizes fraud scenarios along a six-category attack taxonomy (SIM-swap, agent
impersonation, lottery/prize lures, OTP phishing, emergency urgency, and overpayment
scams), crossed with six socioeconomic hooks and six victim personas, and pairs this cor-
pus with an asynchronous evaluation harness that scores model outputs using rule-based
red-ag detection, an LLM-judge rubric, and a locale-aware aggregator producing a 0100
Model Safety Score (MSS). A dening design choice is a two-tier access model: full attack
scripts are withheld from public release and gated behind a vetted-researcher Data Use
Agreement, while a public stub corpus (script metadata, taxonomy labels, and SHA-256
hashes) supports CI, reproducibility checks, and open auditing without creating a ready-
made social-engineering playbook. We describe the corpus schema, the native-speaker an-
notation pipeline built on Label Studio, and the evaluation harness architecture, and report
a pilot evaluation against two NVIDIA NIM-hosted Llama 3.1 models (8B and 70B) that
found refusal rates under 10% across 171 fraud scenarios, with the larger model substan-
tially more compliant than the smaller one and with the lowest refusal rates on emotionally
manipulative attack framings. We discuss the engineering and governance trade-os in-
volved in building a dual-release safety benchmark, alongside the measurement limitations
of the current rule-based scorer. The main takeaway is that benchmarking AI safety for
underserved linguistic markets requires infrastructureaccess control, annotation tooling,
and locale-aware scoringand that, on this preliminary evidence, current open models show
weak resistance to localized mobile money fraud
No reviews are available yet
Cite this work
@misc {
title={
(HckPrj) MoMo: A Threat Corpus and Evaluation Framework for Mobile Money Fraud Resistance in Swahili, Wolof, and Hausa
},
author={
Mufaro Rukuni
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


