AfriVish-Bench

The project can be commended for identifying a real and significant threat and providing sufficient evidence to highlight the magnitude of the threat. I appreciated the explanation for the African context and the bullets describing the ways in which these attacks may take advantage of African citizens. The research gap was clear and the challenge / solution for it also well thought out. 2.3. highlighted a novel form of an existing challenge enabled by AI technologies. The author is also well aware of the limits to their plan to address the challenge and presents what appear to be workable solutions. I say this because I do not have the technical competence to adequately assess whether they are indeed workable. Throughout the author also referenced existing evidence highlighting that the question had been well considered. I was also impressed by the ethical safeguard considerations and the consideration of applicable legislation in Zimbabwe and RSA. The balance between technical detail and explainability was also thoughtfully considered, making it a well executed project.

One-line summary: A research proposal for the first African-context benchmark to detect AI voice-agent vishing (robocall to WhatsApp to mobile-money drain), built around four detection tasks, a planned 1,300-sample synthetic audio dataset, a live adversarial voice-agent test harness, and SynthID watermark evaluation. It is a proposal: nothing in it was actually built or run.

Constructive critique:

This is the sharpest problem framing I have seen so far in the batch. The threat is real and current, the regional angle is genuinely under-served (ASVspoof and similar do synthetic-voice detection in isolation, with none of the brand-impersonation, multi-channel handoff, or mobile-money context that defines the actual African attack), and the author clearly understands both the attack chain and the detection landscape. The task design is thoughtful, the label schema is concrete, the metrics are appropriate and threshold-aware, the dual-use section is responsible (no real-voice cloning, gated dataset, watermarked samples, held-out test labels), and the whole thing reads cleanly and professionally. The novelty is real: a first-of-kind African vishing benchmark with a live adaptive-adversary harness is a direction others could build on.

The problem is that it stops exactly where the value starts. This is a PRD for a benchmark, not a benchmark. Hamel Husain and Shreya Shankar make the point that an eval is what you learn from putting real outputs through it, and that you cannot know the failure modes until you look at real data. Brendan Foody frames evals as the new PRD, and that is precisely what this is, a very good PRD, with the actual eval still unwritten. In a weekend hackathon judged partly on execution, a zero-implementation proposal is a hard ceiling. Even a tiny slice run for real, say 50 synthetic clips through the SynthID detector plus one LLM-judge transcript pass, with the actual precision and recall reported, would have transformed this from "here is what I would measure" into "here is what I found."

One thing to fix directly, and it is important: Table 4 ("Expected Baseline Results") presents invented numbers (0.98 AUC, 0.92 overall) in the exact visual form of a measured results table. A reader skimming could easily mistake projections for findings. Relabel it unmistakably as targets, or better, replace it with whatever you can actually measure. The most concrete next step is also the cheapest: SynthID coverage is your single most interesting and most runnable research question (it only flags watermarks from participating generators, so open-source TTS slips through), and you could produce a real coverage-gap number this week without building the full pipeline. Minor but worth tightening: one reference is a placeholder (Joycent, arXiv:2405.XXXXX), which undercuts the otherwise strong citation work.

Track + flags:

On-topic and a clean fit for Africa, Evals & Benchmarks. Central threat claims verified as real. Dual-use is handled responsibly, though the live attack-harness recipe is the most sensitive element and should stay gated. Note for the panel: this is a proposal with no executed work, and the projected-results table is presented in a way that could read as measured data. No plagiarism signals; one placeholder citation.

The project is well-executed and addresses an important problem. However, there are two notable limitations:

- The submission is a proposal rather than an executed benchmark. No dataset was constructed, no baseline was trained, no SynthID coverage was measured, and no live voice-agent harness was built; the "expected baseline results" table is populated with projected numbers based on published ASVspoof performance. As a research proposal it is thorough, but there is no empirical contribution to evaluate — the core claims about detectability of ATHR-style attacks, the Gemini/Claude LLM-judge accuracy, and the SynthID gap are all forward-looking rather than tested.

- Several design choices are asserted rather than justified against alternatives. The 1,300-sample size, the specific accent-group split (300 American / 200 British / 200 African-English / 100 mixed), the four-task structure, and the weighted overall score formula (0.35·T1 + 0.25·T2 + 0.20·T3 + 0.20·T4) are presented without power analysis, ablation, or comparison to how existing anti-spoofing benchmarks partition their tasks. Without piloting even a small portion of the dataset, it is difficult to know whether the proposed splits would be sufficient to distinguish detector performance at meaningful confidence.