IndicViet-Safe: Cross-Lingual Safety Evaluation of Open-Source LLMs in Hindi, Hinglish and Vietnamese

Add more depth to the primary findings by defining the actual content of the responses. Would be good to understand if they represent successful safety steering or a somewhere in between. Can also clarify the prompt design a bit more by explaining the methodology for the Hinglish prompts

The under-refusal/over-refusal framing is the paper's strongest conceptual contribution and is communicated clearly. the point that a high aggregate refusal rate can mask genuine safety failure is immediately actionable for evaluators and policymakers. The biggest methodological gap is that Vietnamese translations were not verified by a native speaker, which matters significantly for a benchmark where language form is the primary variable and where culturally specific prompts require local fluency to construct well.

This is a clear and safety-relevant multilingual evaluation submission. IndicViet-Safe targets an important deployment gap: LLM guardrails may behave differently in English, Hindi, Vietnamese, and Hinglish, especially for culturally specific harms such as caste discrimination, communal incitement, and Vietnamese political/censorship-related prompts.

The benchmark design is easy to understand, and the refusal-rate/safety-gap analysis surfaces a useful distinction between under-refusal and over-refusal. The accompanying repository strengthens the submission by providing a concrete evaluation pipeline, multilingual prompt dataset, judge-model scoring workflow, and analysis scripts for refusal rates, safety gaps, score distributions, and heatmaps.

That said, the broader area of multilingual safety evaluation, multilingual jailbreaks, and code-switched prompting is already well studied, including work such as XSafety, MultiJail, HarmBench, and IndicSafe, so the main contribution here is localized coverage and category design rather than a new evaluation method. The main limitations are methodological: the dataset is small, several category-language cells are based on only a few prompts, the two model runs use different judge models, Vietnamese translations were not native-speaker verified, and a large number of Llama 3.1 8B evaluations produced parse errors. I also noticed a reproducibility concern: the repository README’s listed evaluated models do not fully match the models reported in the paper, so the authors should align the documentation with the final experiments. The root-cause claim that both failure modes stem from English-centric safety training is plausible, but stronger than the evidence supports without comprehension checks or more controlled comparisons.

Overall, this is a solid and clearly presented hackathon benchmark with good Global South relevance. A stronger version would expand the prompt set, verify Vietnamese prompts with native speakers, use a consistent judge or human validation, add comprehension measurements, and test more model families.