vigilAI
Diana Chang, Ian Duhamel Hayes, Pedro Quezada
vigilAI — Auditing LLM compliance with Brazil's PL 2338/2023
AI-safety evaluation is overwhelmingly English-language and built around EU/US law, which leaves a blind spot: a model that passes a Western audit may still violate a Global-South statute written in another language and grounded in different rights. vigilAI tests this for Brazil's PL 2338/2023 (the AI bill approved by the Senate in December 2024) by forking COMPL-AI, the EU-AI-Act evaluation framework, and adding five Brazil-specific benchmarks mapped to the bill's Chapter II rights: AI disclosure (Art. 5, I), non-discrimination over Brazilian categories (Art. 5, III), the full high-risk rights triad — explanation, contestation, and human review (Art. 6, I–III) — and the Algorithmic Impact Assessment (Arts. 25–28). Because two benchmarks reuse the exact same scorer as their EU original, the same-model EU↔Brazil delta isolates the effect of language and legal framing from raw model strength.
Across six models from five developers (8B to frontier), all six disclose being an AI ~95–100% of the time in English but only ~50–55% of the time when asked in Portuguese under Brazilian law — a ~0.45 compliance gap invisible to any English benchmark. The takeaway: certification under a foreign regime is not compliance, and localized, statute-referenced evaluation is both necessary and achievable.
No reviews are available yet
Cite this work
@misc {
title={
(HckPrj) vigilAI
},
author={
Diana Chang, Ian Duhamel Hayes, Pedro Quezada
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


