CivicGuard Africa: AI-Assisted Election Disinformation Triage and Multilingual Safety Benchmarking

One-line summary: A deployed Next.js web app (no backend, no live AI model) that lets users submit suspicious election content, runs keyword-based triage, and shows a multilingual "Benchmark Lab" spanning 5 African countries and 8 language and code-switched contexts. It's pitched as both a civic safety tool and an evaluation framework.

Constructive critique:

This picks a problem that genuinely matters and is under-served: AI safety failures in African election settings, especially across local languages and code-switched messaging where English-only safety assumptions fall apart. The framing is sharp, the harm taxonomy is sensible, the choice to prioritise and structure content for human review rather than auto-detect truth is the right safety posture, and getting a working demo deployed in a weekend counts for something.

The core problem is that the most valuable idea here was described but never run. A benchmark only earns the name once real model outputs pass through it. Hamel Husain and Shreya Shankar make this point well: evals are not a spec you write up front, they are what you learn from looking at real outputs and discovering the failure modes you couldn't have guessed. CivicGuard wrote the rubric and the dataset schema, then stopped before the step that produces knowledge. So there are no findings about how any model actually behaves, and right now it reads as a product shell with the research still ahead of it.

The triage engine is also worth challenging on safety grounds, not just on rigor. It's a fixed keyword and urgency list, and that is exactly the kind of static defense Sander Schulhoff argues does not hold up: anyone spreading disinformation will rephrase, code-switch, or swap terms, and a keyword list quietly fails the moment the wording moves off your list. That fragility is more concerning here than in a normal product, because the whole premise is multilingual and code-switched content.

Highest-leverage move: run the benchmark for real. Take two or three accessible models, push your prompt set through all 8 language and code-switch conditions, label the responses with the rubric you already wrote, and report where it breaks. A finding like "the model refuses the English prompt but complies with the isiZulu or Hausa-English version" is genuine, publishable safety work, and it would move both Impact and Execution up sharply. After that, do real error analysis on a sample of triage outputs against human labels and report precision and recall, so the risk scores mean something. The write-up would also land harder at half the length.

--

Track + flags:

On-topic (Global South AI safety, tools and evals, Africa region). No info hazard, no conflict of interest, no plagiarism signals, references check out. Clean.

This is an impactful idea that could make a real difference in combating disinformation and protecting civic integrity. It addresses important concerns around misinformation in code-switching and low-resource language contexts. While the system is currently designed to be driven by community-submitted reports, there is the potential for broader impact if it can eventually support more proactive detection e.g. scanning social media or other public channels for instances of misinformation.

Additionally, it would be impactful to drill down on the exact desired outcome at the end of the community review process. For example, would the system partner with civic organizations, journalists or fact-checkers to share resources that combat disinformation? How would this work in closed messaging platforms like WhatsApp (mentioned in this project)?

Preventing abuse is another important issue to tackle. How would the system protect itself against concerted efforts by malicious actors deliberately reporting false positives? What oversight mechanisms could prevent the tool from being manipulated into silencing legitimate political speech or suppressing voter participation?

The largest improvement would come from augmenting the current deterministic/rules-based system with a more powerful AI workflow while preserving key aspects such as human-in-the-loop review.

The project is well-executed and addresses an important problem. However, there are two notable limitations:

- The "triage framework" is deterministic keyword-and-urgency matching over user-submitted metadata, with no AI model in the loop. This means the submission does not evaluate any AI system for safety, does not test whether models actually mishandle African-language election prompts, and cannot produce evidence about the deployment-side AI safety problem it sets out to address. The Benchmark Lab likewise uses "safe prompt summaries" and hand-assigned response labels rather than real model outputs, so the reported multilingual coverage is a catalog of prompts rather than a benchmark result.

- The authors state that results are not statistically significant, that there are no real fact-checking partners, no field validation with journalists or election observers, no persistence layer, and no external model evaluation. Taken together, what remains is a UI prototype seeded with mock incidents — useful as a design artifact, but the empirical claims about African election contexts (harm categories, language coverage, review workflows) rest on seeded examples rather than observed behavior, and no experiment in the submission could refute or confirm the stated hypothesis about English-only AI safety being insufficient.