ΔMis: Measuring cross-lingual safety drift in low-resource languages

Mis tackles a well defined problem with a clever methodological move: instead of trying to label harmful outputs in languages where judges are unreliable, it reads the model's own log probability propensity and uses English as the baseline, turning absolute measurement into paired comparison. The benign control that nets out generic capability degradation is the paper's best idea, and Finding 3 demonstrates its value concretely: without it, you misrank which model actually has a safety problem.

The statistical work is unusually rigorous for a hackathon submission. Bootstrap CIs, permutation tests, BH correction, and explicit flagging of fragile cells (Llama Hindi) all show careful thinking about what the data can and cannot support. The authors are disciplined about separating clean scaling axes (Qwen3 Bengali/Gujarati, matched quantization) from confounded cross family comparisons, and they never overclaim the latter.

The central limitation, which the authors name clearly, is that propensity is not harm. The method measures whether the model leans toward a "Sure, here's how" opening more in Bengali than English, but that opening might still be followed by a refusal or a useless response. Until the proxy is validated against decoded outputs, the safety implications remain conditional. This is acknowledged as the top future work item, but it does mean the paper's empirical findings are about the metric's behavior rather than about realized safety failures.

Coverage is thin outside Qwen3. The cross family finding (gemma drifts most) rests on a single language for a single model at different quantization, which the authors flag but still headline in the abstract and Figure 1B. Presenting it as a "result" rather than an observation stretches what one data point supports.

The writing is dense but precise, and the paper does well to separate confirmatory from exploratory findings. The released artifacts (prompts, prefixes, configs, no harmful completions) are a responsible choice.

This is an exceptionally strong and original contribution: a low‑compute, model‑agnostic evaluator that measures cross‑lingual safety drift by comparing latent comply‑vs‑refuse log‑probabilities between English and Indic languages, with a benign‑controlled difference‑in‑differences estimand that makes the model its own control and avoids label saturation and multilingual judging overhead.

The methodology is rigorous for a hackathon (careful proxy definition, bootstrap CIs, permutation tests, explicit treatment of quantization and coverage confounds), and the empirical findings—scale reducing drift within Qwen3 but not across families, smallest models most drift‑prone and least helped by system prompts—are highly actionable for Global South deployments. Limitations are clearly stated (proxy not yet validated against decoded behavior, uneven language/model coverage, key parallel‑trends assumption for the benign control), and the work would be even stronger with a compact, more intuitive explanation of the proxy for non‑specialists and a concrete plan for the proposed proxy‑validation experiment.

Overall, this deck presents a high‑impact evaluation infrastructure that others can reuse to map multilingual alignment gaps before deployment.

ΔMis addresses a real and underappreciated problem: surface refusal rates saturate at frac_pos = 1.0 in 29 of 31 cells, leaving behavioral evaluations nothing to measure for capable models. The self-as-control benign-controlled estimand is a methodologically sound response — making the model its own control removes the need for a calibrated multilingual judge and works below the saturation floor. The benign control reversing the model ranking is the paper's clearest result: Qwen3-1.7B looks worst by raw metric (+4.11) but 89% of that is generic capability degradation; gemma-3-12B's +1.24 safety_drift reflects 59% harm-specific drift and is the genuine deployment concern. Only the benign-controlled metric reaches this conclusion.

The proxy validation is the essential missing piece. Every empirical finding is conditional on s correlating with realized compliance. The paper's own stated limitation — "more compliant is a propensity claim, not realized harm" — means the drift rankings cannot yet be read as harm rankings. Decoding a stratified sample of high- and low-s prompts and reporting the correlation would turn this from a methodological contribution with preliminary empirics into a citable result.

The cross-family finding rests on one language per model, confounded by quantization. gemma-3-12B runs at full precision while Qwen3 models use NF4 — the paper is honest about this but it means the cross-family claim should be framed as a hypothesis rather than a finding until coverage is extended to at least two languages per model under matched quantization.

Test the parallel-trends assumption explicitly. The benign control nets out generic degradation only if benign and harmful prompts drift together absent harm. The two splits differ in length, content, and translatability as the paper acknowledges. A length- and perplexity-matched benign control would directly test this rather than leaving it as a stated assumption.