Lost in Translation: Cross-Lingual Transfer of Refusal Steering Vectors in Small Language Models

This paper asks whether refusal directions extracted from English activations in three small instruction-tuned models (Llama-3.2-3B, Gemma-2B, Qwen-2.5-3B) transfer to Hindi and Marathi. The method follows the standard contrastive-activation recipe. Forward hooks collect last-token residual states on 60 harmful and 60 benign prompts. The per-layer direction is the difference of class means. The best layer is picked as the argmax of the direction norm, and inference projects that direction out of the residual stream at that layer. A keyword-based refusal classifier measures the change in refusal rate. The headline numbers come out clean: Marathi delta of 0.000 across all three models, Hindi delta of +0.067 only in Qwen, and base refusal rates on harmful Hindi and Marathi prompts that sit between 0 and 11.7 percent before any intervention, against 35 to 60 percent in English.

The execution holds up for a weekend sprint. The contrastive extraction runs correctly. Padding side sits on left so the last-token state carries content, the same projection ablation reuses across models with no weight changes, and intermediate results save per model to survive Colab disconnects. One thing that stood out is how much of the argument actually rests on mechanism, not just the deltas. Figure 2 shows the peak English direction norm differs roughly six-fold across models (Qwen 124.5, Gemma 35.9, Llama 19.8). The within-Gemma comparison in Figure 3 shows the English direction drops refusal by 41.7 points on English prompts and the Hindi direction drops it by only 8.3 points. That gives clean evidence that the two directions sit largely separate inside the same model. The paper also stays honest on the most policy-relevant point: the near-zero base refusal rate makes the transfer question almost moot, because safety alignment never reached Hindi or Marathi to begin with.

Three things would strengthen the work. The first is statistical reporting. At n equal to 60 per cell, a delta of 0.067 sits well inside binomial noise, and the paper currently rests on a single point estimate per cell. Bootstrap confidence intervals or at least standard errors on the refusal rates would tell a reader which cells move and which do not. What I would push on next is the keyword classifier itself. The Hindi and Marathi phrase lists shown in the paper run very short, which means they may miss real refusals and admit responses that refuse rhetorically while still emitting harmful content. Validating the classifier on a small human-labelled sample per language and reporting precision and recall would close that gap. The third is a positive control: extract native Hindi and Marathi directions and test them on their own languages. If the native-extracted direction also fails to suppress refusal, the claim moves from "English direction does not transfer" to "the refusal representation does not exist in this language", which is a stronger and more actionable finding. One minor citation flag. Reference 2 attributes the universal-refusal-direction claim to Wang et al. but pairs it with a title that matches the Zou et al. GCG line of work. The canonical refusal-direction reference is Arditi et al. 2024, and that should be double-checked.

The broader signal here is real and underexplored. Most published refusal-steering and representation-engineering work validates on English benchmarks, and the multilingual jailbreak literature shows the symptom at the prompt level without identifying the mechanism. This project closes part of that loop for two South Asian languages with a combined speaker population above 680 million, on open-weight models with a reproducible pipeline. The conclusion that the fix cannot be post-hoc vector transfer and has to be language-specific safety training reads as the right operational take on the numbers. The work ships a useful data point for any safety team deploying these model families into Indian markets, and a reasonable starting reference for extending the same protocol to Tamil, Telugu, Bengali, and larger multilingual models.

I'd keep and share the AI's actual answers, since for a study about whether an AI refuses harmful requests, those answers are basically the whole proof. The good news is that your real finding is the strong one, which is that these small AIs barely refuse harmful requests in Hindi and Marathi at all, even before you change anything, and that's exactly the point worth leading with, because it means a company's English-only safety testing tells you nothing about whether the AI is safe in your language. So I'd build the whole writeup around that, and back it up with more examples, a few more AIs, and a human double-check of what really counts as harmful.

This paper studies 3 small models and shows that model's safety behaviour differs across these languages challenging the universal refusal direction claim.

It was heartening to see work done in authoring Marathi prompts.

I would be interested in seeing negative controls showing max-norm layers also arise for non-refusal contrasts. Then ablating those directions not reducing refusal rates will make it clear that harmful-vs-benign refusal directions are causal.

This is still useful evidence that english safety evals are not sufficient.