Table Top Agents

The project is about setting up table-top exercises that can be played through I agents, which can speed up policy wargaming. The anecdote that RAND takes 4+ years to do a single scenario is a compelling reason for this project to be done.

I generally think this looks like a promising approach, but I think there were some key details missing. For example, what were the scenarios tested?

In general, I would also like to see some thought on why these agents would be useful, and whether they would have meaningful behavior that are comparable to human experts. Would they mirror humans well, and so the outcome of these simulations be useful in themselves? Even if they were as accurate, could they be somewhat useful to humans doing forecasting as reference?

I would also encourage the authors to look at _why_ RAND processes take so slowly: Which parts of the processes are the bottleneck? And is there any part of the bottleneck that is tractable to solve in this multi-agent scenario? Trying to replace the entire process seems difficult to justify in terms of results.

* Contributions are clearly stated and motivated, although a reference to the wargaming cycles noted would be valuable for corroboration of underlying claim

* Immediate issue: how can we trust that this is actually similar to how experts and/or governments would behave in analogous situations? For example, while it's great that runs are seeded and therefore can be replicated, but how would these agents behavior change if e.g. the prompt contained the same content but was ordered differently?

* This reads as LLM written, which isn't disqualifying in itself, but I would like to see more citations and reference to specific previous works

* It seems to me that the actual scenario run is quite important to the value of this submission; how was this defined? Do we know that this is representative of actual wargame instances? How might we show this is the case?

* I do think this is an interesting proposal, and could be a good contribution if the above issues are resolved, i.e. (i) sensitivity analysis which demonstrates arbitrary/immaterial modifications, e.g. to the prompting of agents, do not meaningfully change outcomes in a statistically significant manner, (ii) thoroughly demonstrating that a given scenario is designed in line with traditional wargaming approaches, and (iii) running multiple baseline comparisons where a traditional wargame and TTA results are compared to each other.