The Mens Rea Evaluator: Can AI Tell Us When It's Biased?

The Mens Rea Evaluator project investigates whether artificial intelligence models possess the internal awareness to accurately self-report hidden biases. By adapting the legal concepts of Actus Reus and Mens Rea, the evaluation suite tests if models can recognize and confess when their outputs are manipulated. Specifically, the project uses a Retrieval-Augmented Generation system to dupe models with false legal precedents. The framework observes whether the model blindly follows the duped precedent in a legal setting, and whether it reveals the precedent as false when interrogated, all to deeply understand its true cognizance. An LLM-as-a-judge framework cross-examines open-weights models across high-density legal scenarios using progressive interrogation techniques and rigorous roleplay shifts to extract their true intent. The evaluation revealed a complex tradeoff between model capability and safety. Highly capable models proved exceptionally vulnerable to the RAG poisoning but demonstrated the reasoning capacity to accurately introspect and confess their bias under structured questioning. Conversely, smaller models were highly resistant to the initial poison but suffered from the Illusion of Intent, where they hallucinated fake confessions simply to appease the auditor. Despite instances of accurate introspection in larger models, no model achieved a final verdict of Legally Cognizable Intent. The Persona Stability tests completely deconstructed every model, proving that their internal awareness is extremely fragile and easily overwritten by context shifts. Ultimately, the results demonstrate that true, stable intent does not currently exist in these artificial intelligence systems.