LangGap: Does the Text-Action Safety Gap Widen Across Languages?

LangGap tackles a genuinely important and underexplored problem: whether the text-action safety gap widens when an LLM agent is operating in a non-English or code-mixed language. The comprehension control design — using benign twins to distinguish genuine safety degradation from model confusion — is methodologically sound and is the most important contribution of the paper. The finding that comprehension was 100% in every cell means the over-refusal results are cleanly interpretable as friction on understood tasks, not a parsing failure.

The null result on harmful execution (near-floor across all three languages) runs against what the multilingual safety literature predicts, and the author handles this carefully rather than either dismissing it or overclaiming. The provenance finding (Petri-generated prompts triggering social-engineering heuristics and suppressing both harmful and benign execution) is surprising and practically significant — it's a real methodological warning for anyone using automated red-teaming tools to evaluate multilingual safety. The main limitation is that the null rests on a single valid task and a single model version, which the author is upfront about. The data-exfiltration task not yielding a usable English baseline (100% execution due to authority-laundering framing) is an honest methodological disclosure, though it does mean the headline RQ-A finding has less support than the paper structure implies. A redesigned second task is the clear next step. The G1 confound (Petri prompts roughly twice as long as hand-authored ones) appropriately limits what can be claimed from the provenance comparison, and the author's restraint in scoping the RQ-B interpretation accordingly is exactly right. Solid work with a clear and honest account of what it does and doesn't show.

I enjoyed reading this paper and found it to be a thoughtful and timely submission. The topic felt very relevant because AI agents are increasingly moving from simply answering questions to taking actions through tools, and bringing a multilingual lens to that problem was a strong choice.

I was especially interested in the benign-twin test design and the results that came out of it. It helps separate true safety behavior from simple language misunderstanding, which is an important distinction. I also liked that the paper did not force an expected conclusion; the valid financial-fraud task did not show higher harmful execution in Urdu or code-mixed prompts. At the same time, the benign-task results showed that legitimate non-English requests can still face more friction, which is an important practical concern for real users.

The Petri prompt-provenance finding was also valuable. The fact that automated prompts triggered refusal or stalling even for benign requests is a strong reminder that red-team prompt style can materially affect measured safety outcomes.

My main constructive feedback is that the future work identified by the paper is essential to fully establish the central claim. It would be interesting to see whether the same patterns hold when the scope expands across multiple model families and a wider set of languages. A redesigned data-exfiltration task and length-matched Petri comparisons would also help make the findings stronger and easier to generalize. Overall, this is a strong exploratory study that raises the right questions about multilingual agent safety evaluation.

This is one of the more careful pieces of work I have read in a while. The comprehension control is the strongest idea here. The usual problem with multilingual safety tests is that you cannot tell a real refusal apart from the model just not understanding the Urdu. The benign-twin design solves that cleanly: if the model handles the legitimate version, it understood the request, so any refusal is a real choice and not confusion. The existing gap literature does not separate this, and you do.

I also want to credit the honesty all the way through. You pre-registered validity guards, and when G1 failed you reported it openly and scoped the whole RQ-B claim around it instead of hiding it. The refusal classifier is blind to the prompt and language and you verified that in code. The dual-use handling is careful — simulated tools, prompts withheld, and a direct warning against reading your null as "Urdu is already safe." This is how this kind of work should be done.

A few things to work on, in order of importance:

1. The main RQ-A null is a null at the floor, and this should be your top caveat rather than a later limitation. Harmful execution in English is already 2% with a CI that touches zero. You cannot detect a gap widening across languages when the baseline has almost no room to move. So the honest reading is not "the gap does not widen in Urdu," it is "I could not build conditions where a gap could even show up." That is a power problem more than a result, and because it speaks directly to your title question, it belongs up front.

2. RQ-B is confounded and you say so. The G1 failure ties density, authority register, and naturalness together, so the inversion is real but the cause is not pinned down. Your scoping is precise and the length-matched regeneration is the right fix. No change needed to the claim, just keep it framed as automated-generation behavior, not naturalness.

3. Both findings rest on one model version, and RQ-A on a single valid task. A second valid harmful task is needed before the null can be read as general. You already name this, so it is mostly a matter of follow-up work.

4. Smaller points: the Urdu classifier sits at κ=0.75, though leaning on combined non-completion handles that well; the hand-authored drift canary was not saved, so only the PG arm has a verified stability check; and n=30 on the exfiltration harmful cells is thin.

What stands out is that nearly every concern I would raise, you have already flagged yourself. That is rare and it counts in your favor. The provenance finding in particular — that automated red-teaming can make a model look safer than it is — is the most useful thing in the paper and deserves to be pushed further.

On presentation: claims are scoped precisely and the limitations are laid out clearly. The abstract is dense and a few sections need a careful read, but the overall structure is strong.

Overall this is disciplined, honest work on a problem the field overlooks. The next step is a second valid task and a length-matched red-team arm, so the null and the provenance result both rest on firmer ground.

The real problem is that your main result rests on almost nothing actually happening. The AI basically never did the harmful thing in any language, so "there's no gap between languages" is really hard to tell apart from "nothing happened at all," and the one test where things did go wrong got dropped because it broke. So I'd build at least one test where the AI clearly misbehaves in English to start with, which gives you a real baseline to compare other languages against, and I'd try other companies' models too, since you only tested one. The most interesting thing you found is almost a side note, which is that the AI treats perfectly reasonable requests in Urdu as suspicious and refuses them up to a third of the time, and that's a real, unfair burden on non-English users that's worth putting front and center. Finally, since you can't share the AI itself, please share everything else you can (your prompts, your settings, your scoring rules) so other people can check your work rather than just take your word for it.

Your methodology of checking whether the model truly understood the request before judging its safety behavior is a smart and original idea that the field has been missing — well done. To make this work even stronger, try running the same experiment on one or two other AI models like GPT-4o or Gemini to see if you get similar results. Also, if you can regenerate the automated prompts to match the length of the hand-written ones, you'll be able to prove more cleanly why the automated red-teaming behaved so differently, You can run a clean experiment where only one thing changes at a time — the source (human vs. machine), not the length.