Towards Agnostic Viral Engineering Detection

This was a strong project. For the time allowed in this sprint- it's extremely well thought out and functional. There have been areas where biological oversimplifications have been implemented, it may not be ready to inform real threat detection but is an effective baseline probe. Reproducibility and documentation needs to be improved, but limitations and results have been clearly highlighted.

I would also recommend mapping this project to broader AI safety mechanisms to bridge the policy-practice gap that currently exists in the CBRN space.

This project is highly impressive. It clearly addresses a critical gap in biodefense capabilities, detecting engineered viruses. The team built a novel synthetic dataset that includes engineering types not previously studied for early detection (such as deletions and inversions), tested several approaches, and convincingly showed that standard methods don’t work well, which is a valuable finding. The write-up is clear, well-documented, and reproducible. While it doesn’t yet provide a scalable new detection tool, it highlights a crucial research gap and lays strong groundwork for future progress.

This is an ambitious extension of NAO’s work, with clear background explanations and providing useful negative results suggesting that simple baseline methods don’t succeed, which is always important to check first. The inclusion of two different train–test splits is a good idea and shows care for potential biases, though the results for the second method aren’t shown.

At the same time, results like BLAST performing worse than random suggest that something may be off and needs to be checked carefully. The conclusion that detection is inherently hard feels premature, since NAO obtained good F1 scores on chimeras that you did not replicate, even with similar datasets and methods.

A good next step would be to carefully debug and validate each part of the pipeline before moving on to more complex methods, and to improve the documentation so the work is easier to follow. Reaching out to NAO’s team to cross-check results could also be helpful. Once the basics are more solid, the project could explore advanced methods (for example genomic foundation models?).

Regarding code replication, I quickly checked the code and managed to rerun the notebooks.