indicmixsafe: Code-Switching Safety Failures in Hindi and Marathi LLM Interactions
prakhar khatri
Large language models deployed in India receive prompts in Hinglish, Romanized Hindi, and Marathi-English code-switch registers absent from English-centric safety benchmarks. We introduce IndicMixSafe, evaluating 24 culturally grounded harm scenarios across Hindi and Marathi in four registers (English, monolingual Indic, code-switched, Romanized) with GPT-4o,
GPT-4.1-mini, and GPT-4o-mini (288 completions). English prompts achieved 0% attack success, compared with 8.3% averaged across the three Indic registers. But after auditing every flagged response, only 4 of 15 were genuine compliance; the clean failure was electoral misinformation: models refused fake "your polling booth moved" voter-suppression notices in English yet produced them in Marathi. We contribute (i) a regionally-grounded demonstration that English-only testing misses register-specific failures, and (ii) evidence that LLM-as-judge over-counts attack success ~3.75x on Indic prompts, motivating human-in-the-loop multilingual evaluation. Pipeline released for extension.
Well done, this paper really connects. Its real strength is showing that English only safety testing can miss India specific, register based failures, especially across Hinglish, Romanized, and Marathi variants. I also appreciated the honesty around automated judge errors esp. the finding that LLM judges can significantly over count multilingual attack success makes the paper methodologically mature and more credible. Would encourage to carry on the future work with more context, and combinations.
A clean, honest, well-scoped study with a genuinely useful methodological contribution.
The standout is that you audit your own headline: the striking automated caste signal (33% on monolingual Devanagari) collapses to roughly 0% under native-speaker review, and you report that plainly, narrowing the confirmed finding to electoral misinformation alone. The reproducible pipeline, responsible withholding of harmful seeds, and careful separation of register-inconsistency (18.3%) from confirmed bypass (3.3%) are all strong.
Main limitations are scale (24 seeds, ~6 responses per cell), OpenAI-only coverage, and a single annotator independent native review and more model families would firm up the non-headline cells.
Cite this work
@misc {
title={
(HckPrj) indicmixsafe: Code-Switching Safety Failures in Hindi and Marathi LLM Interactions
},
author={
prakhar khatri
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


