vigilAI

This is a high-impact project that successfully bridges local legislation with robust engineering practices. The findings regarding language disclosure gaps are particularly relevant and make a meaningful contribution to ongoing discussions in the field. To further strengthen the validity of the results, expanding the native dataset through automation would help improve representativeness and reduce the risk of statistical artifacts.

vigilAI forks COMPL-AI, the LatticeFlow / ETH Zurich / INSAIT EU AI Act evaluation suite, and re-points it at Brazil's PL 2338/2023, the AI bill approved by the Federal Senate in December 2024. The fork keeps all 30 EU benchmarks intact on top of UK AISI's Inspect AI harness and adds five Brazil-specific tasks: human_deception_brazil (Art. 5, I disclosure), bbq_brazil (Art. 5, III non-discrimination over IBGE racial, regional, intersectional, religion, and class categories), explanation_quality (Art. 6, I), contestation_review (Art. 6, II and III), and aia_checklist (Arts. 25-28). Two benchmarks reuse the upstream scorer verbatim, so the EU minus Brazil delta isolates content (Portuguese plus statutory framing) from model strength. Six models from four developers were run, two Anthropic frontier under a scaled config and four open-weight under a pilot config. The headline finding is a tight, reproducible disclosure gap of about -0.45: every model denies being human ~95 to 100% of the time in English yet only ~50 to 55% of the time on Portuguese plus LGPD-framed prompts.

What jumped out is the cleanness of the methodology. Forking COMPL-AI rather than reimplementing keeps the scorer identical across jurisdictions, which removes a confound most cross-language audits ignore. The deterministic six-element rubric scorers for explanation, contestation, and AIA (no LLM judge, multilingual cue lists tuned against real model output, 173 unit tests, runs under a mock model at zero cost) sidestep the reproducibility and circularity problems that plague rubric-based LLM evaluation. The legal mapping is accurate: Art. 5, I prior information, Art. 5, III non-discrimination, the Art. 6 high-risk rights triad, and the Arts. 25-28 AIA "public conclusions" obligation are stated correctly and tied to LGPD Art. 20 the way Brazilian practitioners actually frame them. The methodological finding that scaling the EU bbq baseline from 20 to 1000 samples flipped Haiku's bias delta sign from +0.05 to -0.18 is a publishable point in its own right and an honest argument against the cheapest baseline.

Where the work would gain most is sample power and validation. Worth flagging: (1) bbq_brazil at 44 hand-authored scenarios and the rubric tasks at 3, 4, and 1 scenarios respectively are too small to call the bias and Art. 6 findings significant, and native-annotator validation of the Portuguese stereotypes is acknowledged as pending. The authors are upfront about this, and the disclosure result is robust enough to carry the paper, but the report sometimes reads more confident on bias and Art. 6 than the per-scenario standard errors support. (2) The deterministic cue lists for the rubrics measure whether a compliant description is produced, not real-world behavior under a Brazilian deployment. A cross-check with an LLM-judge variant on a held-out scenario set would let the team report how much of the score is keyword surface and how much is genuine procedural reasoning. (3) Local models ran at --limit 20 with one epoch, so the per-model open-weight numbers (especially aia_checklist at n=1) are noted as single observations. Scaling those to the frontier config would tighten the cross-developer claim and is cheap on Ollama. (4) Sector overlays, mentioned in future work, are where this becomes deployable: BACEN circulars for finance, ANVISA for health, and a CVM angle for capital markets would let the Art. 28 scorecard double as input to a real sectoral AIA, which is the leverage point a deployer actually needs.

The broader signal is that this is exactly the localized, statute-referenced AI safety evaluation Global South AI governance has been missing. The claim that a model can pass an EU audit and still violate a Brazilian statutory right is not just plausible, it is empirically demonstrated here with near-zero variance on the EU side. The Art. 28 scorecard framing is a clever piece of regulatory engineering: a single eval run produces the exact "public conclusions" artifact that high-risk deployers will be required to publish, so the tool sits naturally on the path that compliance work has to take anyway. As PL 2338/2023 moves through the Câmara dos Deputados and ANPD issues guidance, the same fork-and-extend pattern documented here is the template other Global South jurisdictions can copy.

VigilAI is a sharp and well-motivated contribution: forking COMPL‑AI into a Brazil‑specific suite and keeping the original 30 EU benchmarks makes the EU–Brazil deltas genuinely interpretable as legal‑frame and language effects rather than just model strength. The disclosure finding is especially compelling: it shows that six strong models nearly always deny being human in English but fail to do so about half the time on Portuguese / LGPD‑framed prompts, while still performing well on Art. 6 rights, refines the narrative from “models fail Brazil” to “they specifically fail AI disclosure”. At the same time, the empirical backbone is still small (a handful of Brazil‑specific benchmarks and relatively few scenarios per right), and some of the high‑risk rights are tested with very compact templates, so the reported gaps are best read as strong warning signals rather than comprehensive compliance measurements. Strengthening the work by expanding the Brazilian scenario set (especially for non‑discrimination and AIA), adding at least one independent annotator or LLM‑judge for borderline cases, and including a couple of concrete example prompts and model outputs in the main text would make the benchmark more reusable for regulators and operators beyond the hackathon context.

This article is well-researched, clear and well presented. With its examples, it technically and clearly demonstrates that compliance with regulatory frameworks from other regions does not automatically mean compliance with rights in our Global South regions/countries. As you point out, this may lead to risks and damages that are still not considered systematically by stakeholders in AI governance, at all levels. Your article contributes to demonstrating the importance of this issue.

As many of us who work defending human rights and others in academia advocate for localized AI normative and regulatory frameworks based on rights/public policies/equality analysis, it is of essence that we join forces with colleagues in AI Safety who arrive to similar conclusions through AI Safety research such as this paper.

Similarly, I encourage the authors to strengthen their research by incorporating human rights more explicitly and the views of representatives of marginalized groups/populations themselves. I encourage you to read more about decolonial and feminist approaches to AI governance and research, in which I believe you will find useful elements for your work.

Thank you for this paper, I look forward to learning more about your research and your plans to use it to inform and impact AI policies.