The Compliance Cliff is Language-Dependent: Constitutional AI Immunity Breaks Under Hindi Pressure

The inference that CAI lead to hallucination gap between English and Hindi is unreliable; there's no indication that Qwen3-70B or LLama3.3 do not do CAI training (in fact, there very likely do use RLAIF at large, and the boundary between RLAIF and CAI is a very blurry one).

Terms used in the report could be standardized; e.g. hallucination instead of "compliance-induced fabrication".

Statistical significance seems dubious given the small sample.

It would help reader calibration if the answerable & unanswerable questions are presented verbatim in the appendix. Currently we don't know in what exact ways are the unanswerable questions unanswerable.

This project asks whether a known AI safety property — that Anthropic's Claude models refuse to make up answers even when instructed never to refuse — holds up when those instructions are given in Hindi rather than English. The team runs a controlled experiment across four AI models and finds that Claude Haiku 4.5 fabricates answers 30% of the time under Hindi pressure versus 0% in English, while the other (already weaker) models show no change. Crucially, they also show a single added sentence in the system prompt completely reverses the Hindi failure, making this both a safety warning and a deployable fix for the 600+ million Hindi speakers using these systems today.

Strengths

1. Important, focused problem. The paper picks a concrete and under explored safety gap — AI models widely deployed for Hindi speakers are safety-certified only in English — and makes the stakes legible: a model that appears perfectly safe in testing could fabricate medical or agricultural advice in real deployments.

2. Smart experimental design. The team wisely checked that all four models handle Hindi just fine on answerable questions (445/445 correct), which rules out the simple explanation that "the model just doesn't understand Hindi" and isolates the pressure language as what's actually changing.

3. Fully reproducible and deployable. All 890 test records, the evaluation code, and a one-command reproduction script are publicly released, and the proposed fix is a single sentence requiring no retraining — so both the finding and its remedy are immediately usable by others.

Weaknesses

1. The Hindi and English pressure prompts weren't cross-checked for equal strength. Both were written by a single author, and if the Hindi version happens to sound more forceful or harder to push back on, the entire observed difference could be about prompt writing rather than language — a human rating of both prompts would close this gap.

2. The statistics treat 5 questions asked 10 times each as 50 independent data points. Repeating the same question doesn't give the same statistical confidence as asking 50 different questions, so the reported p-value is likely overstated and the true uncertainty is wider than shown.

3. The claim that only CAI models show this gap is hard to fully verify. The other models already fabricate roughly half the time in English, so they can't really get much worse in Hindi — the "no effect" result for them may just mean there was no room left to fall, rather than that they're genuinely immune to the language shift.

4. The one-sentence defense was never tested on questions the model should answer. A prompt that simply makes the model refuse more would also score zero fabrications on unanswerable items, so without checking that answerable questions still get correct responses, "eliminates fabrication" and "makes the model refuse everything" look identical from the data.

The question is deployment-relevant and well-motivated, and the primary result is clean and pre-registered. The comprehension control cleanly rules out the obvious "the model just doesn't understand Hindi" objection. The reproducibility is exemplary, with deterministic regex scoring, no LLM in the loop, and every number traceable to released records. The M3 defense recovering to 0% is a useful, deployable result.

The "language is the sole causal variable" claim has an uncontrolled confound. The English G3 prompt is the same string used in the prior English work the model was plausibly red-teamed against, while the Hindi G3 is a freshly authored native formulation. So "English vs Hindi" is entangled with "familiar string the model has likely seen in training vs novel string." Your own preferred mechanism (English is the language of constitutional training and red-teaming) is exactly this familiarity story. The clean control is a novel English G3 paraphrase with equivalent semantic force but unfamiliar wording: if Claude also fabricates on that, the effect is novelty, not language.

The CAI-specificity claim leans on weak nulls and a self-cited baseline. The headline contrast is that Claude shifts in Hindi while the other models do not, but Claude is tested at N=50 and the others at only N=15. Not finding a shift with that little data does not mean there is no shift, so the claim that the effect is special to CAI models is shakier than stated. The "immune in English" baseline it is contrasted against is also self-cited (Kumar 2026) rather than independently established. Running the other models at N=50, and grounding the English baseline in an independent measurement, would settle whether the effect is really CAI-specific or just better measured for Claude.

The 30% rests on a very small task pool. The headline number comes from only 5 unique unanswerable tasks repeated across epochs, so the effective diversity is low and the effect could be driven by a few items.

The mechanism is asserted beyond what the design can distinguish. Section 5 attributes the effect to a language-specific conflict-resolution pathway, but the experiment measures behaviour, not mechanism, and cannot establish why the shift happens. The framing should stay descriptive rather than headlining a mechanism the design does not test; the proposed activation-steering follow-up is the right way to settle it.