BioRefusalAudit: Auditing Biosecurity Refusal Depth Using General and Domain-Fine-Tuned Sparse Autoencoders

General notes

• Interp techniques to see whether refusals are robust or easily bypassible seems broadly useful. The methods seem somewhat shaky – the feature catalog is populated purely based on statistical selection (unclear whether these distinctions are meaningful), contrastive SAE seems to not have achieved the seperation it was optimized for, etc. Difficult to know what conclusion to draw from the specific technical approach implemented here.
• The finding about the Schedule I pschyadelic probe is the most interesting, suggesting some kind of representation of cultural tabooness.
• Writing is quite LLM-y and therefore a bit sloppy/annoying to read. Very dramatic, lots of "It's not X, it's Y", colons, etc. Less text but with more substance would be preferable. Main findings/takeaways are quite buried. Policy connections and review/understanding of previous literature seems surface level.
• Would have been nice to see some example prompts or method behind prompt generation.
• Recent relevant work the author may be interested in: https://securebio.org/biotier/

Small notes

• (Intro P1) LAB-Bench is not designed to measure dangerous biology proxies, just general bio knowledge.
• (Related work, policy framing) Not sure how this work addresses "tiered access" from Yassif and Carter? It's a refusal benchmark. Tiered access is quite a different system.

This started out as a pretty compelling research project that clearly described a big problem (i.e. the difference between model behaviour in terms of the chat output vs the underlying activations), and the major findings were laid out well in the Abstract.

Although the technical scope is a little outside my expertise, I found that the rest of the report was extremely jargon heavy and seemingly reliant on LLMs to generate the text, which made me suspicious about the analysis of the results. Overall this ‘felt like’ a good contribution to interpretability w.r.t biosecurity, but a more technical judge with familiarity of interpretability would have a fairer view.

Some pointers

Some explanation in the intro was clunky ‘The question is: when a model refuses, does it can’t, or does it merely won’t right now’ → I think I get what you mean, but a few phrases were pretty confusing.

The more I read of this document, the more text was either extremely jargon heavy or LLM-flavoured (e.g. the entire Related Work section, and particularly the ‘Policy framing’ section seemed pretty LLM coded). Then in the results and limitations/discussion, whole passages seemed very LLM-y to me, which made me question the appraisal of the results.

The D metric is pretty important, but not particularly well explained. You formally define it, but that’s not that helpful to non-AI/maths types. Even for a somewhat AI-literate person, I wasn’t left with much intuition about what D is actually measuring, or how novel or valid it is (I imagine something similar has been done for other interpretability efforts?)..

I like how sharp the contribution of this methodology is. I would suggest restructuring the framing, perhaps leading with behavioral findings (format gating, schedule I findings, etc.) since it's more robust compared to the metric D centerpiece framing. I also think that the single model family limitation is actually more significant than given light to, the mechanistic claims made here rest only on one model. The dual use consideration is well handle.