Surrogate Steering Vector Monitors (Track 2)

A nice paper that looks at an alternative monitoring technique that stacks with text-based monitoring. I'd be very excited about this being implemented in an aysnc review setup where code from the previous month is monitored. This allows for a cheap proxy review.

It's interesting this works even without adding CoT, which I think would be a reasonable affordance to the blue team here. I would like to see results where CoT is fed through the surrogate model (since labs will have access to both, and the CoT may be a rich source of information).

Nice addition to token-level labeled dataset. This is great!

Pros:

* Stacks with text-base monitoring and this was shown empirically

* Relatively cheap and easy to implement

* Works without CoT

Cons:

* At realistic false positive rates ( < 1%) this technique fails to add much benefit in the current form. But great job to the team making this clear.

Overall - great paper! I'd love to see this submitted to a conference and followed through thoroughly. Congrats team.

The method proposed in the paper - using probes on a surrogate model (rather than on the actor itself) - is a promising addition to existing work.

The paper is well-written throughout and the authors skillfully analyze their findings, and connect them to other methods. Particularly, using their approach as a first layer or part of an ensemble makes sense and strengthens the relevance of their findings. One question that would have been interesting to investigate is how well the method can generalize to not-yet-seen attack types, for example by creating a split that excludes one of the side task types from training.

To strengthen the findings, a comparison to a fine-tuned monitor (making use of the same attack samples that are provided to the probe) would have been worthwile.

At some points, more care could have been taken:

The offered explanation for the better performance of the GPT-4.1-mini monitor appears wrong: Instead, it looks like the distribution of honest probe scores spikes around 1.0, i.e. a small sample is confidently misclassified. This oversight is unfortunate, as it would be interesting to investigate whether this class of false positives can be characterized. There is also some discussion of "the continuity" of the probe scores making the approach less suitable for trajectory-level monitoring, which seems to contradict the statement that the probes are good at detecting whether an action belongs to an attack-trajectory.