DriftWatch

Driftwatch explores an important AI safety question: how model quality, latency, and automation bias interact to produce human oversight failures. The idea of modeling oversight as a dynamic process rather than a static probability is compelling and potentially valuable for deployment planning, particularly in resource-constrained settings.

The project would be substantially strengthened by:

- Calibrating the simulation parameters using empirical studies on automation bias, review behavior, or human-in-the-loop decision making rather than manually specifying degradation rates.

- Performing comprehensive sensitivity analyses to demonstrate that the observed effects are robust across a wide range of parameter values instead of depending on a particular configuration.

The project identifies a genuine problem of dealing with AI sloppiness and laziness during human-in-the-loop interventions with cheaper and more and more quantized models being released. However, the solution suggested in the simulation and the assumptions about human sloppiness might not be correct in the first place.

There were several aspects of this that I thought worked well. There was a great conceptual foundation; a creative way to frame a simulation based on a very interesting and somewhat unexplored concept, namely the "compute diffusion paradox." This is where as you get closer to zero compute, the human oversight burden remains constant, while the other piece, linking particular model characteristics i.e., error types (steady vs. burst), latency to how humans gradually lose their ability to be effective when using automated tools is a nice take on automation bias and gradual loss of control.

Additionally, the repeated statements throughout the document that these are simulation based findings, as opposed to measured claims regarding real models, and the fact that the author has taken steps toward thinking about the potential misuse of these findings ("dual-use") shows an understanding of safety concerns.

Ways to make it better:

(1) As stated above, all of the key results appear to have been predetermined through the input assumptions. For instance, the degradation rates shown in Table 1 (e.g., 22% base error + 65% burst error for INT4, 5% base error + 0% burst error for GPT-4o) are input parameters you selected as part of designing your simulation. Therefore, there may be some degree of circular reasoning in concluding that quantization speeds up oversight collapse. One of the best possible follow-on steps would be to conduct calibration studies. Even a little bit of measurement data (INT4 vs. FP16 error and latency on a benchmark) will allow you to transition from modeling your own assumptions to providing evidence.

(2) Be careful claiming that anything has statistical significance. While you refer to the findings being "highly robust" and indicate that they show "significant effects" across different backends, you do not provide either test results, sample sizes (seed counts), or confidence intervals. Since running simulations is essentially cost-free, consider providing distributional information over many random seeds instead of simply citing single-point estimates like 32.44 vs. 9.26. Also, perform actual hypothesis testing.

(3) Establish a baseline for the early warning predictor. It seems safe to say that achieving 100% recall at an accuracy rate of approximately 58%, on just 12 trials, means that your predictor says "there's going to be a problem" virtually every time. That is roughly equivalent to saying nothing informative. Include a simple majority class baseline, and significantly increase the size of the hold-out group prior to calling it a useful predictor.

(4) Distinguish between inputs and outputs for the benefit of the reader. Input values that look like they might actually be measurements (such as 72.7% for the prompt injection rate for INT4, or the per condition error rates) exist side-by-side with disclaimers indicating that those are simulative values. Clearly state whether the values presented in each table represent parameters you explicitly specified versus ones that emerged during the simulation process.