Jan 11, 2026
AI Chat Analyzer
Trevor Lohrbeer, Darijan Ducic, Sascha Held
A web-based research tool for importing or running live AI conversations and evaluating different dimensions of manipulation. It provides real-time message- and trajectory-level scores alongside human ratings, enabling researchers to compare, calibrate, and improve their scoring models. Scores are provided by a Python server running the researcher's scoring model.
A strategically high-leverage tool! Addressing the bottleneck of translating researcher "taste" into prompts via a calibration interface is a great insight. I was particularly impressed by the choice to wrap UK AISI Inspect and Anthropic’s Petri in the backend; leveraging industry standards makes this significantly more robust than ad-hoc scripts. The "Step Mode" demonstrated is excellent for debugging how manipulation accumulates over a trajectory. However, the frontend-heavy data management is a clear scalability bottleneck. Moving dataset handling to the Python backend to support large-scale logs (like WildChat) would be the critical next step to make this a daily driver for safety researchers.
Overall, this is an excellent execution of a "tooling" project that could help researchers explore different conversation, explore their algorithms and their scoring methods.
Cite this work
@misc {
title={
(HckPrj) AI Chat Analyzer
},
author={
Trevor Lohrbeer, Darijan Ducic, Sascha Held
},
date={
1/11/26
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


