PRISM: On Sparse Autoencoder Feature Identifiability for Conditioned Introspection

The bookkeeping discipline here is the best I've seen in this batch. Keeping the two models' fits separate, refusing to headline the pooled estimate, making the eight judge refusals explicit rather than silently retrying them, and recomputing the Pythia-only analysis from the repaired ledger to confirm it reproduced the original coefficient exactly — that last check in particular is something most published work doesn't do. You also report a null cleanly and resist every available temptation to dress it up. I want to be clear that what follows is not a complaint about honesty.

The problem is that the primary experiment could not have answered the question. You're testing whether a feature-level property predicts detection, in a 70M-parameter model that produced one detection in 474 trials. An outcome with essentially zero variance has no predictor — not this one, not any one. So the Pythia null isn't evidence that identifiability fails to predict detection; it's evidence that Pythia-70m doesn't detect injections, which was fairly predictable in advance given that the literature you're building on demonstrates this behaviour in frontier models. The calibration pilot is where this shows: it tuned injection strength against response length, when the quantity that needed piloting was detection rate. Fifteen minutes of piloting for a nonzero base rate would have redirected the whole study to Gemma from the start.

That has a direct consequence for the reported statistics. A logistic regression with four predictors fit to one positive event doesn't produce an estimate — it produces separation, and the coefficient 0.50 with CI [-1.19, 2.20] should be read as "no fit was possible," not as a wide interval. Similarly, an AUC computed against a single positive is just the percentile rank of one observation in one ordering; the identifiability-0.63-versus-decoder-norm-0.76 comparison is where one trial happened to sit in two lists. Your prose is careful about this ("the ranking comparison does not support identifiability over decoder norm in this setting"), but the abstract and Table 1 still present these as results, and I'd rather see them reported as undefined. Gemma is fittable but thin: nine events across four predictors is roughly a quarter of the events-per-variable people usually want, so [-0.34, 0.92] is close to uninformative too.

Note also that the pooled coefficient (-0.59) has the opposite sign from both per-model estimates (0.50 and 0.29). You correctly decline to interpret the pooled fit, but the sign reversal is worth naming explicitly as a Simpson's-paradox artifact of the model-level difference, because a reader skimming Table 1 and the pooled figure may not see why they disagree. The pooled AUC of 0.82 is close to a measure of how well the model detects which model it is.

Two further things about the Gemma arm, which is the one carrying the events. The Gemma Scope SAE reports reconstruction quality of 0.6786 against Pythia's 0.9782. A dictionary that reconstructs the residual stream that poorly is a weak basis for computing geometric properties of individual atoms, so the arm with statistical power has the less trustworthy instrument. Worth stating as a limitation in its own right. Second, the strengths differ by more than an order of magnitude across arms (1–8 versus 20–400) with no shared normalisation — expressing injection strength relative to residual-stream norm at the hook site would make the two at least commensurable.

A definitional gap: "frame-theoretic identifiability" is the independent variable of the entire paper and it never gets a formula or a citation. "A score based on its separation from nearby decoder directions" doesn't let a reader reconstruct it — coherence with the nearest atom, mean coherence over some neighbourhood, a frame bound, something else? Figure 2 shows values from about 0.60 to 1.00, which is consistent with several candidates. Please define it explicitly; it's the one thing a replication would need most.

Smaller points. The methods say three seeds per feature-strength pair yield 480 systematic trials, but results report 474; the ledger reconciliation implies six of the eight judge refusals fell in systematic trials, and stating that where the number first changes would save the reader the arithmetic. The pooled CI is printed as [1.39, 0.21], which is malformed and presumably [-1.39, 0.21]. And since the single Pythia positive drives every Pythia statistic, say whether it was included in the 15-trial human review and whether the human agreed with the judge — with n=1, that one adjudication is the whole result.

Nothing is linked. The paper repeatedly invokes a shared trial ledger, scoring provenance records, and configs/experiment.yaml, which means the artifacts exist; without them the auditability the discussion rightly emphasises isn't available to anyone but you. Posting the repo would also let people reuse the harness, which I suspect is the most valuable thing you built.

The question itself is good and I'd like to see it answered — whether interpretable-feature geometry predicts which injections a model notices is a real bridge between SAE work and introspection work. The redesign is straightforward: pilot for a nonzero detection rate first, run in a model that clears it, use one SAE with decent reconstruction, vary identifiability widely while holding decoder norm fixed by construction rather than by covariate adjustment, and power for the event count rather than the trial count.

This is basically an extension of the Lindsey-style introspection work, asking whether feature distinctiveness predicts detection after activation injection, on two small models. Introspection does have implications for AI safety, although they do not discuss this in the main text. Prior work would suggest that small/base models are not in fact able to detect injection, and that is mostly what they find here, obviating their subsequent tests on feature distinctiveness - although they do report some small detection success in the Gemma model, there are insufficient controls or baselines reported to determine whether it is meaningful. Also, unlike that earlier work, here tests of detection and identification are conflated, so it's hard to make direct comparisons. In general the methodology is described in insufficient detail to allow a determination of whether the experiments were conducted correctly.