Mimir: AI Image Provenance & Detection

- I like the swiss cheese model of detection with several layers but I'd like to see more of the implementation of each layer, eg: layer 4 mentions a CNN - what specific weights and model architecture were used should be mentioned in the paper itself

- As a technical paper, I'd also like to see a followup formalizing the decision engine mathematically since it describes a weighted score outputting a confidence number

- However 23 images is a tiny dataset and is just as likely to just be statistical anomalies. You can get more synthetic images from real ones using open tools, and run them through the Whatsapp APIs for compression etc to increase the size of your dataset programatically

- Layer 3 the local SQL DB will have a lot of performance bottlnecks if deployed at scale, so you'd need an alternative

One-line summary: A working prototype of a multi-layer AI-image detection system (five forensic layers: metadata/C2PA, invisible-watermark, perceptual hashing, a CNN/ViT visual scan, and Error Level Analysis, fused by a weighted decision engine), pitched as an API-first bot for low-data WhatsApp ecosystems and tested on 23 images.

Constructive critique:

The deployment insight here is the strongest part and it is a real one. Most deepfake-detection work assumes a user who will leave the app and visit a forensic website. Mimir starts from the lived reality that in Zimbabwe and similar markets WhatsApp effectively is the internet, data is expensive, and the platform strips metadata and compresses the image before anyone can check it. Designing detection to live inside the chat, and accepting that any single forensic signal will usually be destroyed, is a sensible framing that the crowded image-forensics field mostly ignores. The writing is also fluent and well-organized, and the engineering instinct of graceful degradation (one layer failing without taking the system down) is sound.

The trouble is that the evidence does not come close to supporting the conclusions, and in one place it actually contradicts the thesis. The whole system is validated on 23 images, six of them fake. There is no confusion matrix, no precision or recall, and critically no false-positive rate, even though the paper itself names the most dangerous failure (falsely accusing a real image) as the thing to avoid. The headline claim, that the system "survives WhatsApp compression," is asserted with no numbers attached to that specific test. And the strongest reported detections, the four of six fakes caught at 95%+, succeeded only because those images still carried intact C2PA metadata announcing they were AI. That is the exact scenario the introduction says never survives in the real WhatsApp pipeline. So the easy wins came from reading a label, not from forensics, and the hard case that actually matters rests on n=2. On top of that, the layer doing the real AI detection (Layer 4) is described only as "a trained computer vision model (such as a Vision Transformer or CNN)," with no model, no training data, and no standalone accuracy, which makes it impossible to tell whether the system detects anything novel at all.

Three concrete fixes. First, drop the rhetoric: "proves definitively," "structurally superior," "democratizes expert-level forensics," and "scaling digital trust globally" are claims a 23-image test cannot carry, and the over-reach undercuts otherwise credible work. The eval discipline Hamel Husain and Shreya Shankar describe applies directly here. Looking at outputs is the right instinct, but you have to do real error analysis: build the confusion matrix, report the false-positive rate on real images, and run a per-layer ablation so we can see which layer actually carries detection once metadata is gone. Second, run the one test that is your actual hypothesis: take a set of known fakes, pass them through real WhatsApp, and report detection with metadata fully stripped, separated cleanly from the metadata-intact case. Third, specify and benchmark Layer 4 on its own against a standard set, because that is where any genuine novelty would have to live, and right now it is the least defined part. As written, this is a promising deployment concept wrapped around an under-built and over-claimed evaluation.

Track + flags:

On-topic (Global South AI safety, detection tooling, Africa region). No info hazard, no conflict of interest, references are real and appropriate. Note for the panel: conclusions substantially outrun the evidence (n=23, no confusion matrix or false-positive rate), and the reported high-confidence detections rely on intact metadata, which contradicts the paper's own core premise.

Strengths: Presents a practical solution with a clear use case and a thoughtful system design. The project demonstrates good engineering fundamentals and addresses a real user need. Areas for Improvement: Include more rigorous benchmarking against existing approaches and provide quantitative performance metrics. Additional discussion of scalability, fault tolerance, and production deployment considerations would better demonstrate readiness for broader adoption.