Modeling the political process to forecast the outcomes of hypothetical AI governance proposals

A novel and an exciting approach - I’d be excited to see you work on this more!

However, I’m quite confused by how the accuracy of the model is measured. The way I understand this works now is:

-When training, the model learns from the sponsorship patterns

-In testing, you ask whether a senator that is mentioned in the bill would support it. This feels odd to me - doesn’t the fact that a senator is mentioned in the bill mean that they support the bill?

It would be beneficial to hold out entire bills from training and then test accuracy on those, so that you can measure accuracy more accurately.

Currently, there’s no validation for the main use case. You generate predictions for the gradated hypothetical policies, but for them, there is no ground truth.

I also think the baseline comparison is flawed - GPT3-oss is not a state-of-the-art AI forecasting tool (you could have compared to this, for example: https://safe.ai/blog/forecasting). And the cutoff date makes the comparison quite unfair.

* While it seems to me that the actual contribution is quite interesting, the framing leaves a lot to be desired. The data used is from the 118th and 119th US Congress, but the framing is about international actors measuring feasibility of agreements being followed by specific nations. While the US Congress may approve a bill if it has been introduced from one of its own, buy in for international regulations is a meaningfully different scenario which doesn't map perfectly onto data used. Nonetheless, I think the contribution is quite interesting and could be leveraged to understand what self-imposed national level legislation is possible.

* The general idea described in the "Question Representation" section is interesting, but I am having trouble picking up on what exact information is provided to the model.

* Why filter to only the bills which mention "risks"? It would be my intuition that the other bills may have provided additional insight into how certain members of Congress feel about AI. Why was this decision made (e.g. cost saving)?

* Would like to see a more principled way of developing the tested bills, although this is definitely sufficient for a hackathon.

* Table 1 is confusing... what is actually being presented here? Why would you show the different versions of the policy and their variants compared to GPT4-oss-20B if it wasn't actually used to evaluate said policies?

* I would have been quite curious to see a more thorough exploration of how this differs from prior policymaker vote forecasting works. While it is certainly possible that the approaches introduced here build meaningfully on existing literature, it is hard to tell if this is the case.

* I would have liked to see more detail and focus on the novel contributions of this work, which I primarily see as the LLM-based question/answer framework used, and how one might improve this to make it more accurate.

* Lastly, I'd be curious what the naive approach of simply assuming that a policymaker would vote for the bill if it was introduced by a majority from their own party. My guess is that this baseline would definitely be above 0%, so it feels necessary to include.