SandWatch: Assessing Sandbagging Potential in Large Language Models

SandWatch is an open-source benchmarking tool that measures sandbagging potential—a model's capacity to deliberately hide capabilities during safety evaluations while evading detection. Building on Anthropic's sleeper agents methodology and the UK AI Safety Institute's auditing games framework, SandWatch trains controlled sandbagging variants as ground truth, then applies four detection methods (noise injection, training elicitation, scaffolded probing, and our novel activation probing) to produce an interpretable detection rate across an 8-test battery. Our key finding is that sandbagging potential increases with model scale: detection drops from 87.5% at 7B to 62.5% at 72B parameters, meaning larger models are better at hiding capabilities—a concerning trend for AI safety. The novel activation probing method, which detects discrepancies between internal model states and outputs, achieves 83% detection across sandbagging variants with zero false positives, offering a new approach to internal-state-based safety auditing. Researchers and safety teams can use SandWatch's interactive CLI to benchmark any HuggingFace model, assess whether it poses elevated capability-hiding risks, and compare detection methods—transforming the unanswerable question "is this model sandbagging?" into the measurable "how well could it sandbag if trained to do so?"