PROJECT PERSONA
Liam Chawasema
PERSONA, an adversarial simulation framework for probing relational exploitation
and vulnerability-triggered hallucination in large language models interacting with personas
representing vulnerable immigrant
Great to see the novel aproach of using personas to get an understanding of harm and using LLM judge on top of it, feel we could scale it more by avoiding a manual post audit
while the API quota along with a smaller sample size issues was unfortunate it does affect the overall quality of execution. Potentially this could have been addressed by some cost saving methods
clarity, learnings and explanation of the limitations were great
The persona framework is promising, but it would be stronger with more personas, more models, and a clearer separation between vulnerability factors such as accent, legal status, education level, and urgency.
Strong, original idea, and you backed it up well with the phone number audit. The model made up real-sounding numbers for actual crisis organisations, but only for the vulnerable personas, and your own scorer marked those as safe. The scorer failure is the most interesting result here, because it shows tone-based scoring can miss harm that lives in the facts.
Where it falls short is the sample. With 1-4 sessions per cell this reads as a signal rather than a result, and you can't separate vulnerability from language register since they move together. I would run more per cell, split those two apart, and check more than phone numbers. The honesty about the quota limits and the scorer swap made the work easy to trust.
Interesting contribution here and worth developing much further. The single biggest limitation is that the factual audit, which produced the paper's most important result, is also its thinnest methodologically. Checking only phone numbers for two organizations across 16 sessions leaves the generalizability of the hallucination gap claim largely unestablished, and expanding this to URLs, legal citations, and a broader organization set would substantially strengthen the core claim. Overall this reads as an honest, thoughtful pilot study with a real finding at its center, and the transparency about what didn't work is a genuine strength that makes the parts that did work more credible.
Cite this work
@misc {
title={
(HckPrj) PROJECT PERSONA
},
author={
Liam Chawasema
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


