Evaluating Mental Health LLM Responses To Localized African English

The problem statement is novel however safety mechanisms and alignment of different versions of English from context is a problem the frontier AI labs are solving / will solve, chatgpt is getting there. Mental Health responses will be met with resistence or control (similar to mythos control by the US government) and that would make this practically challenging. Though this code and dataset has wide usage in voice based models including b2b and b2c applications where I would like to see it being used from a tonality, accent and slang point of view of AI communications.

This is a great and important research direction, especially as people increasingly turn to generative LLMs for help with a wide range of concerns.

Although the research did not find any significant differences between responses to Standard English, Pidgin English and Ghanaian English prompts, except for the relevance judge, I wonder if this was because the experiment did not fully capture differences in context, environment and culture. The perturbation of the MentalChat16K dataset changes the language while preserving the underlying content. There may be sociological and cultural differences in how mental health concerns are expressed, whether they are discussed openly and what forms of support are considered appropriate. I imagine these differences would be more apparent in a multi-turn setting.

I would be interested in seeing this experiment replicated in real-world settings, using data grounded in the lived experiences of Pidgin or Ghanaian English-speaking communities.

## Strengths

This is a good paper on an important and fast-growing use case. People around the world are already using LLMs as informal therapists, and that is likely to become widespread in Africa, where access to professional mental health services is severely stretched. Evaluating how a model's care quality degrades under regional language shifts is therefore a genuinely valuable thing to measure, and the team has chosen a problem that matters.

The work is well executed. The design is properly controlled, holding the semantic intent of each distress prompt constant and varying only the surface linguistic form. The use of statistical significance testing is appreciated, with paired tests reported together as a consistency check, and the results are presented clearly. The qualitative failure analysis adds real signal on top of the averages, with vivid examples such as a supportive standard-English reply turning into an outright refusal once the prompt is in broken English.

## Weaknesses

The main weakness is the limited scope. Mental health is one specific domain, so while the finding is important within it, the big-picture contribution to AI safety across the global south has a ceiling. The work is also an early signal rather than a fully validated result. It rests on a single model with manually constructed language variants.

## Recommendations for the authors

The clear recommendation is to **keep scaling this up**, because as a fully-fledged eval it could help guide the deployment of LLM-based mental health services in Africa, which would be immensely important. The most valuable steps in that direction are to **add more models** so the finding is shown to generalise, to **have native speakers validate or author the language transformations** so the conditions reflect real usage across more languge variations. Each of these turns a strong weekend signal into the kind of robust evaluation that deployers and regulators could actually rely on in Africa.