Traduttore Traditore? LLM Language-Dependent Safety Answers in Community Contexts

This paper raises an interesting question: beyond binary refusal rates, do LLMs communicate different values or orientations depending on language? The tripartite prompt design (actor, context, call to action) is a reasonable attempt to simulate realistic user interactions rather than explicit jailbreak attempts, and the observation that English responses lean toward evidence based reasoning while Indic languages lean toward community cohesion and personal wellbeing is genuinely thought provoking if it holds.

However, the study's empirical foundation is too thin to support its claims. Sixteen prompts, each used once, across three models and four languages produces 192 responses, but with no repetition there is no way to distinguish systematic language dependent patterns from prompt specific noise. A single prompt that happens to touch community dynamics will naturally elicit community focused language regardless of the input language. The inductive qualitative method (two coders agreeing on themes) is appropriate for exploratory work, but the paper presents its categories ("critical thinking," "civic responsibility," "community harmony") as findings about language dependent safety rather than as preliminary observations from a small pilot.

The confound between language and content is not addressed. Tamil responses focusing on wellbeing could reflect how the models were trained on Tamil data (which may overrepresent community oriented text), or it could reflect translation artifacts, or it could reflect the specific prompts chosen. The paper acknowledges some of this in limitations but does not design around it. A minimal control would be testing the same prompts in English but with explicit Indian cultural framing to separate language effects from cultural content effects.

The Claude Tamil issue (model understood Tamil but responded in English) is more than a limitation; it potentially invalidates that slice of the data since the "Tamil orientation" would then be an artifact of English generation. This needed to be resolved before drawing conclusions.

The writing is clear and the related work section is well organized. The appendix examples are helpful and do show real variation worth investigating. But the gap between the evidence (small qualitative pilot, no statistical grounding, uncontrolled confounds) and the conclusions (language dependent safety patterns) is too wide for the paper as written.

Your idea is good. An AI can technically refuse to be harmful while still pushing different values depending on the language, which means a company's English-only safety check doesn't tell a Tamil or Punjabi community what their AI is really teaching them. But the evidence isn't there yet, mostly because you asked each question only once, so what looks like a "language difference" could just be the AI giving a slightly different answer by chance, and I'd ask each question several times and report actual counts instead of vague words like "frequently." I'd also keep the languages and the different AIs separate rather than lumping them together, and deal with the fact that one AI answered the Tamil questions in English, which quietly breaks the very comparison you're trying to make. Finally, please publish your data and your questions so others can check the work, and spell out plainly what a community should actually do with this, like a simple checklist for testing an AI in their own language before trusting it.

A refreshing and original contribution, asking not whether models refuse but whether their safe responses encode different cultural values by language.

Two honest limitations to flag: the primary safety test found the models resilient, so the project effectively pivots from measuring safety bypass to a values analysis - worth reframing the stated contribution around that explicitly; and the evidence base is thin (16 prompts, each run once), so a single sample per cell can't separate the value patterns from ordinary output variation. Scaling the prompt set, multiple runs per cell, and an independent coder would let these patterns be claimed with more confidence. The "answered Tamil prompts in English" observation echoes a known representation–language entanglement effect and is worth pursuing. A promising first step into an under-explored question.