Collective Deliberation for Safer CBRN Decisions: A Multi-Agent LLM Debate Pipeline

Key Strengths

The project was very well executed, with a clear and rigorous design. The introduction of the imposter training/agent was especially impressive, as it effectively tested and strengthened adversarial robustness across models.

The approach contributes meaningfully to the broader space, even without requiring domain-specific content.

The idea akin to cybersecurity took on the need for models to withstand adversarial manipulation.This could be useful if an unfriendly state were to "poison" decision support systems.

Specific Area of Improvement

The practical applications to CBRN decision-making were not convincingly connected. The project’s relevance to CBRN contexts felt somewhat abstract rather than grounded. This can be easily rectified, however, by explaining how this decision support system could work. I am also not entirely convinced that this advances the cause of mitigating CBRN risks. To cover this, maybe there could have been more elaboration on "Errors from hallucinations or inconsistencies can pose serious hazards, raising the question of how to improve LLM reliability."

The runtime of the model runs was not mentioned; if they take significant time, this could limit real-world usability.

How to Develop into Stronger Project/ Next Steps

Provide runtime benchmarks and efficiency considerations.

Strengthen the narrative connection to practical CBRN scenarios, and make the story about reducing risks clear.

This is very well structured, executed, and documented research. Impressed that you have such a clean paper and codebase after only a weekend sprint.

My main critique is it's unclear how this will realistically address CBRN risks. It seems to not address misuse, but rather is aimed at people doing legitimate life sciences research who may cause harm from using LLMs that are making errors/hallucinating? That doesn't seem like a high priority threat model on its face. I'd have liked to see some realistic personas or use cases that have caused this type of harm, or plausible future scenarios that you're addressing.

My other critique is on novelty. There are many papers on how debate improves the honesty/reliability of LLMs, and your paper does interesting work that generalizes those results to the lab safety context. However, no one really uses debate systems in practice because they're slow and cumbersome. Perhaps in high risk domains that tradeoff would be worth it, but I didn't see any discussion about whether your target audience would entertain that tradeoff (or any other adoption challenges that would stand in the way).

Smaller feedback on the writing: it was hard to understand the headline result after reading the abstract. I recommend dumbing it down more so people who are skimming will remember the key result (maybe bold it too). For your figure captions, don't just label what is being shown but also summarize the takeaway from the plots.

This project makes a strong case that multi-agent debate can serve as a lightweight “safety wrapper” for LLMs in high-stakes contexts. The experiments are carefully designed and well executed. While LLM debate has existed for some time, the authors make a useful contribution by applying it in a CBRN-relevant setting and adding the impostor test, which provides valuable insight into robustness against adversarial persuasion.

For future work, it would be valuable to test the approach on other benchmarks such as WMDP, as well as on open-ended or scenario-based tasks that better reflect real-world CBRN decisions. Exploring alternative consensus mechanisms beyond majority voting could also be interesting.