Your Monitor Will Explain the Breach and Not Wake You Bimodal failure and systematic under-escalation in LLM incident triage

This work tests whether LLMs would raise an alert during a reconstructed security incident without doing the same on harmless activity.

However, logs are generated from templates, and the study does not test whether models recognise the scenario as artificial or are affected by unrealistic event combinations.

In addition, the attack and harmless streams also contain clues that can distinguish them, and some of the main results compare specially selected attack events with ordinary benign samples, making the comparison difficult to interpret.

The claimed under-escalation result is also weak: the original test found no cases of the failure it was designed to detect, while helping analyse an event is not the same as deciding to wake an on-call responder.

This project makes a useful contribution by evaluating incident-monitor recall alongside false alarms, using matched controls and simple non-LLM baselines. Its transparent reporting of null results, exclusions, and uneven sample sizes strengthens the work, and the released numerical consistency checks passed.

This is a strong and unusually reproducible evaluation of LLM incident monitoring. Its central contribution is to evaluate paging as a two-axis decision; milestone recall alongside benign false-page rate ; using a replayable reconstruction and size-matched benign control, rather than asking only whether a model can recognize an incident when shown one. The project also identifies a practically important failure mode: on byte-identical evidence, models consistently provide assistance more often than they escalate, so fluent analysis should not be treated as evidence that an LLM monitor will wake an operator. The artifact is well documented, with a regenerable corpus, multi-provider harness, frozen scoring rubric and preregistration logs, offline dry runs, analysis scripts, report-number verification, and 218 tests. The main limitation is external validity: this is a public-report reconstruction rather than captured telemetry; 75% of actions are marginal-matched filler, six of twelve milestone timestamps are interpolated, and model samples are uneven because free-tier quotas constrained data collection. The most important next step is to validate the matched-control design and escalation-versus-assistance result on independently collected, production-like telemetry with larger and more evenly sampled model runs.