Out of Distribution: Does Deepfake Detection Transfer to Real African Faces?
Ifihanagbara Olusheye
Deepfake detectors that defend against disinformation are trained almost entirely on Western data. We audit three detectors across their training distribution (FaceForensics++) and real African faces (FAGE_v2). Only the state-of-the-art model works in-domain, yet it flags roughly two in five real African faces as fake, and recall on African deepfakes cannot be measured because no African deepfake dataset exists. The defence is either unavailable, unreliable on real people, or unvalidated for the African information ecosystem.
I would be curious about a few things as future directions to explore:
- Country wise breakdown for if specific demographics within Africa are disproportionately affected and see if the false positive rate is uniform across countries or has regional variances.
- Simulated degradation of the western images dataset to match the stated lower quality levels of the African dataset pictures
- If this same pattern extends to audio/audio visual data as deepfakes in social and political spheres are rarely limited to static images alone
But agree that the gap in lack of a deepfake dataset for the African demographic remains a big gap to be closed and this paper presents a pretty clean gap and current state of an underserved area in AI safety
I appreciate your focus on deepfake detectors built and validated on Western dataand whether they can be trusted in African info ecosystems. The biggest contribution is the framing, treating the fact that we can't even measure whether these tools catch African deepfakes as the finding itself, and spelling out what that means for safely deploying them. The biggest weakness is the data point that the detector wrongly flags 43% of real African faces as fake because the African set was also lower-quality (blurrier), and blurry images on their own can cause these tools to label them fake, no matter whose face is in them so we can't cleanly separate how much of that 43% is about the faces being African versus just being lower-quality images. To their credit, the author points this out rather than hiding it, and their most rock-solid finding is a quieter one: the free, easy-to-download detectors don't work even when tested on the exact data they were built for. Worth foregrounding that this is an acute version of a broader problem. For example, the Deepfake-Eval-2024 benchmark you cite shows even top detectors losing roughly 45–50% of their accuracy on real deepfakes circulating online, so Western tools are over-credited even where a full test exists, and your point is that for African faces we can't run that test at all. As the key next step, I recommend pushing the governance angle: the Western benchmarks exist because someone funded the consented data behind them (e.g., the paid actors in the Deepfake Detection Challenge), so the missing African benchmark is really a question of who is responsible for funding and owning one. If you name that, then the paper becomes one with a concrete fix.
The project is well-executed and addresses an important problem. However, there are two notable limitations:
- The African evaluation set (FAGE_v2) is web and celebrity imagery at lower resolution than the FairFace Western proxy, and the authors correctly note that resolution can shift p(fake) for reasons unrelated to a face being African. Because no resolution-matched control was run, the cross-distribution false-positive comparison confounds "African face" with "lower-quality image," and the negative-gap finding (African FPR lower than proxy) cannot cleanly be attributed to demographic transfer rather than image quality.
- The central deployment quantity — recall on African deepfakes — is unmeasured, and the paper frames this as a finding rather than a limitation. That framing is defensible, but it leaves the deployment recommendation empirically undetermined: everything the paper can say about GenD in the African setting comes from real-only data, so the actual protective value of the one competent detector against the threat it is meant to stop remains unknown, and the paper offers no proxy or bound (e.g., recall on non-Western fakes from adjacent distributions) that would narrow that uncertainty.
Cite this work
@misc {
title={
(HckPrj) Out of Distribution: Does Deepfake Detection Transfer to Real African Faces?
},
author={
Ifihanagbara Olusheye
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}
