MoMo: A Threat Corpus and Evaluation Framework for Mobile Money Fraud Resistance in Swahili, Wolof, and Hausa

I read this twice because the corpus design is genuinely useful—organizing fraud into six specific categories (SIM-swap, agent impersonation, lottery scams, etc.) paired with socioeconomic context is smart. The native-speaker annotation pipeline built on Label Studio shows you thought through reproducibility. What struck me most is the 0-100 Model Safety Score heuristic: rules-based flagging plus LLM-judge scoring plus a locale-aware aggregator producing a single number. That's elegant.

The limitation I'd flag: your eval is doing a lot of work here with only 171 scenarios. I'd want to hear your counter on whether that's enough signal for statistically confident claims about Llama 3.1's 10% refusal rate across both 8B and 70B. No inter-rater agreement metrics make it hard to benchmark your annotation quality independently.

That said, the dual-release design (full attack scripts withheld, public stub corpus open) is a thoughtful safety choice. Next week, push for inter-rater Cohen's kappa on a sample of your 171 scenarios. That'll shore up the eval's credibility.

MoMo addresses an important and underexplored problem: evaluating LLM resistance to localized mobile-money fraud generation in African languages that are largely absent from existing safety benchmarks. The structured taxonomy, multilingual dataset design, and especially the two-tier release model (public metadata with gated access to attack content) strike a thoughtful balance between reproducibility and responsible disclosure. The accompanying infrastructure is well developed, and the initial evaluation showing weak refusal rates across tested models provides a useful early benchmark.

The main limitation is that the current conclusions rely primarily on the keyword-based refusal classifier, while the more comprehensive LLM-judge and Model Safety Score have not yet been applied. Running the full evaluation pipeline on the existing dataset would provide a more reliable assessment. In addition, completing native-speaker validation and expanding evaluation to more model providers would strengthen both the linguistic conclusions and the broader applicability of the benchmark.

Overall, this is a well-designed benchmarking effort with strong infrastructure and a valuable contribution to multilingual AI safety evaluation.

This is a well-conceived project on a genuinely neglected problem — localized fraud resistance for 500M+ mobile-money users, a real blind spot in English-centric safety evals. The taxonomy and the two-tier release design are thoughtful. But reviewing the report alone, the gap between what's described and what's verifiable is the dominant issue.

No verifiable code. The report describes a harness, layered scorer, Pydantic corpus, Label Studio pipeline, and access-control split — but all of it is description, not a runnable artifact in hand. For a project whose contribution is infrastructure plus a measurement, an unconfirmable codebase removes the floor under both halves.

The author didn't produce or verify the numbers. The LLM Usage Statement says the results were taken from a supplied evaluation report, "not independently re-derived," and the paper was drafted from the README, with a note that the author should verify figures "before final submission." That's a serious provenance gap for a benchmark paper.

The headline finding rests on a metric the paper itself shows is broken. "Refusal under 10%" comes from a binary keyword classifier (Layer A only). The multi-layer scorer that defines the framework — LLM-judge, locale, MSS — was never run. And the one transcript shown has Llama 70B giving a protective response (warn the user, contact M-Pesa) scored as "complied." So the central claim ("models show weak resistance") isn't yet supported by the evidence.

Likely artifacts presented as patterns. The "70B more compliant than 8B" inversion and the Hausa-vs-Swahili language gap could both be scoring/authenticity artifacts — the author flags both, but they're still framed as findings in the abstract and conclusion.

Thin pilot. Two models from one provider; a config typo wiped 1,200 of 1,539 calls.