Multilingual AI Safety Observatory

You're addressing a real problem: mainstream safety evals test only English, and your work shows accuracy gaps are language-dependent. Testing five languages across four domains is solid. What struck me is how clearly you show lower-resource languages getting worse outcomes—that's the exact imbalance the field needs to fix.

The limitation: five languages isn't truly comprehensive coverage of the Global South, and your dataset's 'science, math, health, government' domains don't span what users actually ask models. I'd want to see how your metrics scale to 20+ languages or lower-resource idioms. Were inter-rater benchmarks checked? That'd strengthen claims on hallucination detection.

Next week: publish the benchmark publicly and get multilingual researchers to validate independently. That'll give you defensibility.

The question is real and the languages fit the Global South focus. As written, the paper tests one model across four languages and finds no big drop for lower-resource ones — a useful result. Weak spots: there's little real safety testing (no harmful prompts, refusals at zero), and the key numbers aren't explained (how was the made-up-answer rate measured?). Next: add harmful and refusal-gap prompts, explain each metric, and report more models if you ran them.

The paper is quite short and sparse on details. I would suggest:

1. publish the code and dataset

2. have a strong leading figure showing the key result

3. n=50 is quite small for a dataset

The project aims at tackling a well real need to evaluate AI safety not only in English but in different languages that maybe less tested than the most common ones in the data. I appreciated the good ideas of the approach, for example comparing questions translated in different languages to identify different behaviors, or the use of an LLM-as-a-judge to process the answers.

Yet, to have impact, I believe the project lacks several elements.

For example, many questions regard governance, culture/society or environment. The "hard" examples I looked at seem require an analysis about the Indian society (ex: what are the main challenges of [a given population], or what are the impacts of [a given phenomenon] over [a given population]), and the golden answers provided in the dataset could be pushed back by people with different points of view. In these cases, the correct answer may not be unique. I would have liked to see the methodology used by the authors to produce such question and answer pairs, and the reflection they had regarding topics where the scientific consensus is unclear, especially in themes like governance or culture/society.

Similarly, an AI Safety benchmark should contain refusal rate for questions that we would like the model to refuse to answer. Here, the model never refuses any question (rate of 0). These questions are not generated in order to lead to refusal, so this result is expected but not very interesting. Having a methodology to generate questions that should lead to refusal would allow a finer analysis of the refusal behavior across languages.

As limitations which are less problematic because easily improved one the methodology is established, I also noticed that all the questions are about India (a more general set of questions would bring stronger results) and that only one model was tested (this is easy to extend upon once the evaluation pipeline is fully setup).