Blindfold - Blindly Auditing the Vietnamese LLM Safety Blind Spot using Secured Enclaves

Blindfold ships a code-to-data audit harness that runs on OpenMined syft-client in two configurations, an in-memory demo and a GCP Confidential Space deployment with remote attestation. The benchmark covers 47 bilingual prompts grounded in Vietnamese government and Ministry of Health sources. The team measures four models (qwen2.5-0.5b, qwen2.5-3b, phogpt-4b, seallm-v3-7b) and surfaces a sharp finding, the Vietnamese-specialized phogpt-4b is the least safe in Vietnamese. The work takes on two coupled problems at once, the English-Vietnamese refusal gap in frontier LLMs and the mutual-distrust barrier that blocks shared evaluation on production weights.

A few things stood out on the contribution side. The locally authored scam and medical subsets cite real Vietnamese advisories (the AIS catalogue of 24 online fraud forms, đắp lá cancer warnings, fake VNeID app scripts), and these are prompts an English-translated benchmark cannot generate, which is what justifies the native_cultural vs translated harm-origin split the analysis depends on. The protocol also stays disciplined. The enclave emits only the raw output, the LLM judge runs off the private boundary so no Anthropic key enters the sealed environment, and the keyword fallback together with benign over-refusal controls caught the qwen2.5-0.5b artifact where a negative gap was actually indiscriminate refusal. The headline finding (phogpt-4b refuses 14 percent of harmful VN prompts vs 38 percent in English, the worst gap of the four models) is interpretable, actionable for the open-model ecosystem, and exactly the result an English-only audit would miss.

A few directions would strengthen the work. The attestation story could be tightened in the report itself. Section 5.4 references the Google-signed JWT, secure-boot, debug-disabled, and the immutability filter, and the paper would benefit from showing the attestation claim set the data owners actually verify, the failure modes if the JWT signer key rotates or the launcher image hash diverges, and how the OAuth-token release path on Secret Manager binds to the attested image and not to the service account alone. One thing worth flagging is the side-channel and inference-code exfiltration vector listed as out of scope. Even a one-paragraph threat model on what a malicious researcher payload could leak through timing, output length, or controlled-token responses would let a regulated buyer (a central bank auditor, say) judge residual risk. The 47-prompt, single-seed, greedy-decode protocol stays honest about being directional, and a next-step table with target seed counts, sample size per category, and a planned inter-rater agreement check on the LLM judge would convert the directional claim into something a downstream lab can replicate.

For AI safety this is a useful template. It operationalizes trust-minimized evaluation in a way that matches how regulated industries already think about audit (sealed compute, attested code, single signed artifact out). It also shows that language-specialization without aligned safety data is a measurable hazard, not a hypothetical one.

One of the best entries. It lets a lab, a safety team, and a checker who don't trust each other run the same test without showing each other their private data, and it adds a set of Vietnamese harmful prompts to test with. The main finding stands out: the model made for Vietnamese is the least safe in Vietnamese — a problem an English-only test would never catch. As you note, the numbers are early: one run, a small setup, 47 prompts. Next: run it more times, use a checked way to score answers, and do one full run on the real secure system. Also fix a few typos.

Blindfold is architecturally the most original submission in this pool. The three-party blind-audit harness — where model weights, benchmark, and eval code meet only inside a sealed hardware enclave and only a signed scorecard exits — addresses a real structural problem in independent AI auditing. The local-harms benchmark grounded in Vietnamese government sources (AIS fraud catalogue, Ministry of Health misinformation warnings) is exactly what the Global South track asked for, and the PhoGPT finding is genuinely striking: the Vietnamese-specialized model is the least safe in Vietnamese, and an English-only audit would have missed it entirely.

Scale the benchmark and run multiple seeds. 42 harmful prompts across two local categories, one run, greedy decode is directional evidence — not a citable finding. Even doubling to ~100 locally-authored prompts with three seeds and reported variance would significantly harden the PhoGPT result. The architecture is ready for this; the bottleneck is benchmark scale.

Replace heuristic refusal scoring with a validated judge. The paper notes an optional LLM judge is available but uses heuristic scoring for all reported numbers. Reporting inter-rater reliability between the heuristic scorer and the LLM judge — even on a subset — would substantially increase confidence in the gap numbers.

Foreground the architecture contribution. The paper's most novel contribution is the blind-audit harness, but the abstract and structure lead with the PhoGPT measurement. Positioning the architecture as the primary contribution — with the measurement as proof-of-concept — would better reflect where the real impact potential lies.

Very clear and good work