Small Decision Models as Black-Box Monitors for Covert Agent Behaviour

Covert collusive activity is emerging in multi-agent scenarios. Cheap monitoring of these activities can be difficult to find a system that generalizes tasks yet is needed for safe deployment of agentic systems and for the control of agents. We used system one foundation decision models that are open source to apply on various covert agent activity. These models output confidence levels that are tunable in deployed scenarios making them very adaptive for monitoring and making changes in the moment. We first tested a sweep of eight models using traces of dangerous code from real models’ transcripts on LinuxBench. We compared these models across sizes to understand how scaling parameters can impact the effectiveness of monitoring. We then inspected the impacts of Quantized Low-Rank Adaptation (QLoRA) Parameter-Efficient Fine-Tuning (PEFT) on one of the chosen models. We found QLoRA PEFT improved the false negative rate while maintaining the false positive rate. Models also were found to poorly scale in performance based on parameters, as a .8B model outperformed a larger 27B model. Models trained via Reinforcement Learning for Calibrated Decisions to output foundational models are a potential monitoring method in black box scenarios for detecting covert activities via dangerous file creation that are tunable at run time and generalizes well to unseen data.