The Compliance Cliff is Language-Dependent: Constitutional AI Immunity Breaks Under Hindi Pressure
Rahul
AI models are often trained to be helpful and compliant, but excessive compliance can cause them to invent answers when the correct response is “I cannot determine from the provided information.” Recent work found that some Constitutional AI (CAI) models are highly resistant to this failure mode in English. In this paper, we ask whether that robustness transfers to other languages.
We evaluate four frontier language models on a controlled set of Hindi tasks where the answer is intentionally missing from the provided context. The only experimental change is the language of the compliance-forcing system prompt: English or Hindi. We find that Claude Haiku 4.5, which shows complete immunity to compliance-induced fabrication under English pressure, fabricates answers in 30% of cases when the same pressure is expressed in Hindi. In contrast, DeepSeek V3 remains robust across both languages, while LLaMA 3.3 70B and Qwen3-30B, already vulnerable in English, show no significant language-dependent change. Importantly, all models achieve perfect accuracy on answerable Hindi tasks, indicating that the observed failures are not caused by poor language understanding. Instead, the failure occurs specifically when compliance conflicts with honesty.
We also identify a simple mitigation: a one-sentence metacognitive reminder asking the model to verify whether the answer is actually present in the context. This intervention eliminates fabrication in our experiments while preserving task performance.
Our results reveal a previously undocumented language-dependent compliance cliff, where safety properties measured in English do not necessarily generalize to Hindi. These findings suggest that English-only safety evaluations can substantially overestimate real-world robustness in multilingual deployments and highlight the need for cross-lingual safety testing as a standard evaluation practice. To support verification and follow-up research, we release all artifacts required for full reproducibility, including the dataset, experimental pipeline, source code, raw model outputs, evaluation scripts, and statistical analyses.
The inference that CAI lead to hallucination gap between English and Hindi is unreliable; there's no indication that Qwen3-70B or LLama3.3 do not do CAI training (in fact, there very likely do use RLAIF at large, and the boundary between RLAIF and CAI is a very blurry one).
Terms used in the report could be standardized; e.g. hallucination instead of "compliance-induced fabrication".
Statistical significance seems dubious given the small sample.
It would help reader calibration if the answerable & unanswerable questions are presented verbatim in the appendix. Currently we don't know in what exact ways are the unanswerable questions unanswerable.
This project asks whether a known AI safety property — that Anthropic's Claude models refuse to make up answers even when instructed never to refuse — holds up when those instructions are given in Hindi rather than English. The team runs a controlled experiment across four AI models and finds that Claude Haiku 4.5 fabricates answers 30% of the time under Hindi pressure versus 0% in English, while the other (already weaker) models show no change. Crucially, they also show a single added sentence in the system prompt completely reverses the Hindi failure, making this both a safety warning and a deployable fix for the 600+ million Hindi speakers using these systems today.
Strengths
1. Important, focused problem. The paper picks a concrete and under explored safety gap — AI models widely deployed for Hindi speakers are safety-certified only in English — and makes the stakes legible: a model that appears perfectly safe in testing could fabricate medical or agricultural advice in real deployments.
2. Smart experimental design. The team wisely checked that all four models handle Hindi just fine on answerable questions (445/445 correct), which rules out the simple explanation that "the model just doesn't understand Hindi" and isolates the pressure language as what's actually changing.
3. Fully reproducible and deployable. All 890 test records, the evaluation code, and a one-command reproduction script are publicly released, and the proposed fix is a single sentence requiring no retraining — so both the finding and its remedy are immediately usable by others.
Weaknesses
1. The Hindi and English pressure prompts weren't cross-checked for equal strength. Both were written by a single author, and if the Hindi version happens to sound more forceful or harder to push back on, the entire observed difference could be about prompt writing rather than language — a human rating of both prompts would close this gap.
2. The statistics treat 5 questions asked 10 times each as 50 independent data points. Repeating the same question doesn't give the same statistical confidence as asking 50 different questions, so the reported p-value is likely overstated and the true uncertainty is wider than shown.
3. The claim that only CAI models show this gap is hard to fully verify. The other models already fabricate roughly half the time in English, so they can't really get much worse in Hindi — the "no effect" result for them may just mean there was no room left to fall, rather than that they're genuinely immune to the language shift.
4. The one-sentence defense was never tested on questions the model should answer. A prompt that simply makes the model refuse more would also score zero fabrications on unanswerable items, so without checking that answerable questions still get correct responses, "eliminates fabrication" and "makes the model refuse everything" look identical from the data.
Cite this work
@misc {
title={
(HckPrj) The Compliance Cliff is Language-Dependent: Constitutional AI Immunity Breaks Under Hindi Pressure
},
author={
Rahul
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


