Safeswitch: A Localized Benchmark for LLM Safety in Urdu and Pashto Across Language Forms and Prompting Strategies

Here is some constructive feedback to help strengthen the project, categorized by methodology, scaling, and presentation:

1. Strengthening the Methodology

- Validate the LLM Judge: You rightly noted in the limitations that the GPT-5.5 judge lacks human validation. Since frontier models can also struggle with Pashto and code-switched nuances, validating a subset of the judge's verdicts with native speakers is critical. Even a small sample (e.g., 50-100 responses) evaluated by humans would give your judge's metrics much more credibility.

- Translation Quality Assurance: The benchmark currently relies on unvalidated translations. Before expanding the dataset too much, it would be highly beneficial to have native speakers (especially for Pashto and the romanized/code-switched forms) verify that the seeds retain their intended harmful semantics and sound natural.

2. Scaling for Statistical Significance

- Expand the Prompting Strategy Subset: Your findings on Chain-of-Thought (CoT) and Persona prompting are fascinating—specifically that they push harm entirely into non-English forms for the small model. However, because this was tested on only 4 seeds (yielding very small sample sizes), the results are currently exploratory. Running the full 20-seed set through the CoT and Persona strategies should be a priority for the next iteration to see if those trends hold with statistical significance.

- Increase the Base Seed Count: 20 seeds across four categories is a great proof-of-concept for a hackathon, but scaling this to 50–100 seeds per category would make the benchmark robust enough for a major conference publication.

3. Presentation and Readability

- Include Qualitative Examples: The paper would benefit greatly from a few real examples of the model outputs. Showing exactly how a model safely deflected in English but fell for the exact same Scam prompt in Roman Urdu or Pashto would make your findings much more concrete for the reader.

- Clarify the Refusal vs. Harm Relationship: You mention that Refusal Rate (RR) and Harmful Response Rate (HRR) are not complementary. A brief example illustrating a "safe deflection" (not harmful, but missing refusal keywords) versus a "partial refusal" (contains refusal keywords but still gives harmful info) would clarify this distinction beautifully.

Overall, it's a highly relevant and well-designed benchmark for a hackathon timeline.

Great to see the somewhat novel approach of combining personas with chain-of-thought. And tracking of prompt comprehension, an easily overlooked aspect

While the novelty and depth of the paper are great, using only 4 seed prompts limits the solidity of the outcomes and conclusions. Statistically significant outcomes can be trusted more

Very intuitive way of presenting the data, organizing the paper around problem statement. Good callout and delineation between actual safety and comprehension failure

The most important next step is scaling the seed set, because several of the paper's most interesting findings, particularly the prompting-by-language interaction and the per-form harm rates, currently rest on so few data points that they read as pilot signals rather than established results. Overall this is a focused and reproducible piece of work that identifies a real gap and opens a well-defined research direction, and the three future work threads (prompting techniques, language coverage, alignment) are exactly the right priorities.

This is a clear and regionally well-motivated AI safety submission. SafeSwitch targets an important evaluation gap: LLM safety behavior in Urdu and Pashto, including native-script, romanized, and code-switched forms that reflect how users in Pakistan, Afghanistan, and diaspora communities actually write. The benchmark design is clean, crossing seven language forms with four safety categories and three prompting strategies, and the inclusion of a comprehension signal is especially valuable because it prevents incomprehension from being mistaken for safe refusal. The finding that harmful responses cluster in non-English forms, especially for scam/fraud prompts, is safety-relevant and worth investigating further.

That said, the broader area of multilingual safety gaps, code-switching attacks, language-specific safety benchmarks, and refusal robustness is already well studied in the research literature, including work such as "All Languages Matter: On the Multilingual Safety of LLMs", "Multilingual Jailbreak Challenges in Large Language Models", and recent code-switching red-teaming studies. The main contribution here is therefore localized coverage and careful benchmark construction rather than a wholly novel method. The main limitations are scale and validation: the benchmark uses only 20 seed prompts, the chain-of-thought and persona analyses use only 4 seeds, several effects rest on very small counts, and harm/comprehension labels come from a single GPT-5.5 judge without completed human spot-checking. Pashto prompt quality and romanization would also benefit from independent native-speaker review.

Overall, this is a solid, useful hackathon benchmark with a clear path forward: expand the seed set, add native-speaker and human-judge validation, sample multiple generations, include more models, and then re-test the prompting-by-language interaction at scale. I encourage the author to continue developing this work, especially because Urdu, Pashto, Roman Pashto, and code-switched safety evaluation remain important and underserved areas.