Signalshield Africa
Trivesh Sivnarain, Daniel Mokone
SignalShield Africa is an explainable, human-in-the-loop scam-triage tool for suspicious SMS, WhatsApp, email and voice-note-style messages. Rather than trying to prove whether a message was written by AI, it identifies scam signals such as impersonation, risky requests and pressure tactics, then recommends a safer next step. We tested a rule-based prototype on 160 fictional South African-style scenarios; on a harder 80-message challenge split, it achieved 85.9% high-risk recall with a 12.5% false-positive rate, outperforming a basic keyword baseline.
- From the technical POV, would have liked to see the criteria used for clustering laid out in the paper - which led to the 72.9% score
- Ofc you mention that the rule-based version is for prototyping only, with further exploration into how new rules can be added without degrading performance or causing logical conflicts in rules (i.e how would rule based version scale?)
- "Non-English messages could then be processed using a locally hosted translation component or a
multilingual classifier" --> note that there is a real risk of error rates with translating things as localized slang often breaks standard translation tools, you could explore running it instead through localized models to avoid the intermediate step increasing the error probability of mistranslation or losing the nuance of urgency
One-line summary: A human-in-the-loop, rule-based scam-triage framework for AI-amplified messaging fraud in South Africa, built around a structured "deception fingerprint" (claimed authority, requested action, manipulation tactic, recommended route) plus campaign-grouping, and evaluated on a 160-scenario synthetic benchmark with a held-back 80-message challenge split against a keyword baseline.
Constructive critique:
This is the most rigorously executed and most honest submission in the batch, and it is not close. It is the first entry that actually ran a controlled experiment with a real comparator: a full metric table on the challenge split (85.9% high-risk recall and 12.5% false-positive rate for the fingerprint, against 23.4% and 31.2% for the keyword baseline), plus campaign-grouping F1, category accuracy, and routing agreement. It does genuine error analysis, telling us the nine false negatives avoided explicit code or payment cues and the two false positives were educational messages that read like scams. It is reproducible in a way the others are not: public repo, evaluation script, per-message outputs, and a SHA-256 hash of the benchmark. And the dual-use handling is thoughtful, deliberately withholding full templates and thresholds. This is the eval discipline Hamel Husain and Shreya Shankar describe, actually practiced rather than gestured at.
The intellectual honesty deserves specific praise because it is rare. The paper states plainly that the challenge split is not an independent test set, since the same team wrote both the rules and the scenarios in the same weekend, and that the results show "controlled feasibility, not real-world generalisation." It is explicit that the multilingual, edge-deployment, and contextual-model components were not built. After several submissions that dressed projections or single unverified numbers as findings, this calibration is exactly what you want from a researcher.
The honesty also exposes the two real limits, which are worth pushing on. First, the comparator is weak. Beating literal keyword matching on data you wrote yourself is close to a foregone conclusion, so the headline result mostly confirms that structured features beat raw keywords. The genuinely informative comparison would be against an LLM-as-judge baseline (Gemini or Claude doing zero-shot scam triage), which is the realistic alternative a bank or telecom would actually consider, and which would tell us whether the hand-crafted fingerprint earns its keep. Second, the benchmark is small, synthetic, and self-built, so the next step is a truly independent test set, ideally real de-identified scam reports labelled by someone other than the rule authors. Those two moves would turn a clean feasibility study into a result that actually moves the field. One more honest note on scope: the technique itself is structured rules rather than a new method, and the regional value (local languages, code-switching) is the part that remains proposed, so the demonstrated novelty is moderate even though the craft is high.
Track + flags:
On-topic and an ideal fit for the sub-track (monitoring and tooling for emerging harms, Africa). Dual-use handled responsibly, LLM use disclosed, references real and relevant, reproducible artifact provided. No flags. If anything, this is a model for how to present hackathon work honestly.
This is a relevant research direction, especially given the increasing use of generative AI in phishing scams.
One area that is not fully resolved is the project’s human-in-the-loop framing. The abstract presents SignalShield as a human-in-the-loop framework, but the implemented proof of concept appears to be deterministic and rule-based with no actual human review process. The project would be stronger if it more clearly separated proposed future framework from what was actually implemented and tested.
The paper acknowledges that the current evaluation is English-first proof of concept. I would be interested to see future work build on this by testing the system in multilingual and code-switched settings.
Cite this work
@misc {
title={
(HckPrj) Signalshield Africa
},
author={
Trivesh Sivnarain, Daniel Mokone
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}
