Did the LLM Leave the Chat? Tool-menu Dependence in a Behavioral Measure of Model Welfare
Faiaz Azmain · Team Well and Fair
Submitted to Digital Minds Research Sprint. Sprint projects are early-stage work by participants, not Apart Research publications.
When an AI model uses a button labeled “leave this chat,” does it actually want to leave? We found that the answer is often unclear. When the exit button was the model’s only tool, it sometimes used it when it seemed to want to perform another action, such as calculate something. Adding a second tool—even one that explicitly did nothing—made most exit calls disappear. The model’s internal activity before leaving also looked similar to its activity before calling an ordinary tool. This suggests that many exit calls reflect how the model chooses between available actions, rather than a clear wish to stop. Exit behavior may still contain useful information about model welfare, but it should not be treated as a direct measure of distress or preference without better controls.
Reviews
This is a useful deep dive into the construct validity of an existing behavioral welfare measure. The alternative-tool controls provide good evidence that raw exit-tool calls partly reflect generic action and tool routing rather than an unambiguous preference to leave. They haven't shown that models don't want to leave; they've shown that raw exit-tool frequency can't straightforwardly be read as wanting to leave, and the authors scope their conclusion the same way. I would be interested in further validation against non-tool mechanisms for ending an interaction, since the tool interface itself may affect behavior. I've seen cases where a model ends a conversation when given a natural-language mechanism but won't use an equivalent tool call. The main weakness for me was presentation: I found the experimental setup and broader implications substantially harder to extract than necessary.
This entry is a good construct-validity contribution, mostly well-executed with interesting and potentially impactful first results. There are some methodology issues:
- when comparing the cosine between pre-exit and pre-tool directions, add another "structured-output" comparator that is not tool-calling, e.g. JSON output
- further investigate the presence of "waiting for user input" cues showing up in a some of the pre-exit windows while not showing up in others. The vector difference might be encoding this register difference rather than (or along with) anything quit-specific and confounding the results.
- cross-model (especially cross-family) results would be useful, since tool-calling training can differ a lot between models.
Cite this project
@misc{azmain2026did,
title = {{Did the LLM Leave the Chat? Tool-menu Dependence in a Behavioral Measure of Model Welfare}},
author = {Faiaz Azmain},
year = {2026},
month = aug,
note = {Submitted to Digital Minds Research Sprint, an Apart Research Sprint},
howpublished = {\url{https://apartresearch.com/sprints/projects/did-the-llm-leave-the-chat-toolmenu-dependence-in-a-behavioral-measure-of-model-welfare-grb0}},
url = {https://apartresearch.com/sprints/projects/did-the-llm-leave-the-chat-toolmenu-dependence-in-a-behavioral-measure-of-model-welfare-grb0}
}More from Digital Minds Research Sprint
- 1st placeView project: Readable but Not Causal: Limits of Self-Attributed Welfare Representations in Language Models
Readable but Not Causal: Limits of Self-Attributed Welfare Representations in Language Models
Welfare-like internal representations are increasingly studied as candidate evidence about AI systems. Their entity attribution—whether a valence state belongs to the active assistant or to a merely represented other—is …
- 2nd placeView project: Project Anchored
Project Anchored
Team Wagner
Anchoring vignettes are the standard survey-methodology fix for self-reports that are not comparable across respondents. This project applies them to language models for the first time, using code generation as a …
- 3rd placeView project: Model, Instance, or Persona? Measuring Affective Signals in Public Text After an AI Is Retired
Model, Instance, or Persona? Measuring Affective Signals in Public Text After an AI Is Retired
This sprint asks whether the assistant identifies as a model, an instance, or a persona. I ask which of the three its users name. When a company retires an AI model, users write about the loss in public, and what they …