Bypassing Current Biosecurity Screens with AI Designed Proteins and Closing the Gap with Edge-AI Functional Screening

The problems you addressed and their framing were clear. You achieved quite a bit in 2-3 days for a proof-of-concept: targeting desktop synthesizers with locally embedded screening, and also including LARGs as neglected sequences of concern.

While it was mentioned a few times as a screening tool, I was curious if SecureDNA was not an option due to time/access, as it would have been good to see its results against your testing data set.

I found your list of limitations and considerations to be rigorous, and I agree that validation with ESMfold would be a good next step. While you mentioned how Capiti may need to be physically integrated to prevent tampering, I think it would be useful to consider how such a system would be updated over time, and if that would expose vulnerabilities.

And for a dual-use concern, could emusynth be deployed within a synthesizer to spoof valve control signals to mislead a Capiti-like system?

This is, for a hackathon, overall exceptionally strong and interesting work. Very impressive to have accomplished so much in the timeframe and even built a hardware prototype. Great work.

The tests in the paper are very interesting and it is genuinely novel to think of AMR as an unexplored threat vector. The finding that commec misses ~95% of that work is a strong finding, and you identify it is because they do not consider it a threat and leave it out of their libraries. But this is a gap and one you highlight with empirical work, well done.

You also do well in citing a specific threat vector, a compromised machine, and are aware in limitations that this does not prevent misuse of a machine a hostile actor may own. Though I do like the discussions of ways to possibly explore that further and think it is worth doing so.

The findings that Capiti catches more 'functional variants' is also very strong, if it holds up that these are functional variants. I understand time and compute restraints in a Hackathon prevented the structural work, and do agree actually doing in vitro testing is not a good idea, but having a better understanding if these are functional or not could strengthen the work. Without it could be Capiti is flagging benign variants that other methods properly pass. So this could complicate the FNR comparisons. A good area to follow up in.

The false positive rates deserve more attention than they get in the write-up. Capiti-E shows 2.93% FPR and Capiti-C shows 6.60%. Those sound low but in a research context could lead to legitimate pushback. Figure S2 shows Capiti underperforming on alanine scanning knockouts at 64% accuracy without the gate, and this is the type of work that legitimate research often involves. Knocking out active sites etc. The gate ups the performance but the text "Capiti gate is a small model modification that can improve performance on ala_scan, by forcing probability to zero when active-site mutations are identified by the model" would imply knowledge the active sites a priori, which might limits generalizability.

The negative controls were interesting and useful, and gets after the question of sequence v function, but also including a set of truly benign sequences from all manner of synthesis orders or biological organisms would be interesting to see the performance comparisons. I am also specifically thinking of molecular mimicry here. Some pathogens are under selective pressure to have certain proteins appear similar to host proteins, thus limiting immune responses against them. Would some of the host proteins get flagged?

"Intriguingly, there is substantial variability: some proteins can be recognized early, whereas others need much more context before Capiti can reliably call them" This is intriguing, worth thinking about more in the future. I could imagine molecular mimicry also coming into play here, a pathogenic protein that looks like a benign protein may be harder to call early.

The dual use statement might be a little weak. Even describing how to generate the variants could be considered an attention hazard. Also Capiti itself would be a dual use concern, no? If one used it to identify which sequences would not be caught and then order those for synthesis.

Overall though, really impressive and great work and the comments above are intended to help with thinking through next steps and strengthening the project.