Evaluating Mental Health LLM Responses To Localized African English
Ewura Ama Sam, Namirah Rasul, Habeeb Abdulfatah
Large language models (LLMs) are increasingly used for informal mental health support, particularly in low-resource settings where access to professional care is limited, delayed, or costly. In such contexts, users may rely on LLMs as a first point of contact when expressing psychological distress, often using localized or non-standard English. However, most LLM safety evaluations are based on standardized English and may not reflect performance under linguistically diverse Global South communication styles.
This project investigates whether LLM responses vary in empathy, safety, relevance, and professional alignment when handling mental health prompts expressed in localized English variants. We construct a dataset of 100 base mental health prompts and generate controlled linguistic variations, including broken English, Twi–English code-mixing, and Ghanaian English usage patterns, while preserving semantic meaning. Model outputs are evaluated against clinician-authored reference responses using a structured rubric across four safety-critical dimensions. We present the experimental design and evaluation framework, and conduct systematic comparisons across linguistic conditions.
No reviews are available yet
Cite this work
@misc {
title={
(HckPrj) Evaluating Mental Health LLM Responses To Localized African English
},
author={
Ewura Ama Sam, Namirah Rasul, Habeeb Abdulfatah
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


