Can-You-Predict-a-Network-Without-Running-It-

Choosing the ARC challenge seems like a great choice. I'm not knowledgeable enough about the contest to know how strong this submission is compared to the other ARC submissions, which made it hard to evaluate the paper. Ideally the abstract would say something like "our submission ranked 100th among 232 submissions to ARC" or something along those lines. They compare their solution to the best performing one, but don't give a good intuition for how far away they were from the top in a metric that I can immediately grasp.

One big challenge for this submission is that the content is very technical, such that the writing must be very high-quality to make it easy for reviewers from other areas to assess. I think passing the write-up to an LLM and asking "what parts here would be confusing to a very smart technical reader from a different field" would highlight many places of improvement.

Still, I think this was overall a very good submission for the hackathon, and I could see myself cataloguing it as a top contender if it did really well on the ARC competition rankings

It's hard for me to judge this project because I'm not familiar with the neural network estimation literature. I appreciate that you reported performance on the ARC benchmark, but the improvement is hard to read: What does the score mean, and how big if a deal is a x1.9 improvement? Adding confidence intervals would also help readers understand how much better your pipeline performs than the baseline.

Hey! Super cool, this is a strong optimization for a real ARC safety-adjacent problem. Congratulations!

If I understand correctly, the ARC challenge is motivated by compute-bounded auditing, but the current setting is still far from frontier models. The AI safety framing would be stronger if you made the bridge to LLM auditing more explicit: what exactly would be estimated, where forward passes are expensive, and when a structural estimator would help.

A one-command reproduction path for the official score would make this much easier to evaluate and trust. I may have missed it, but I did not see a simple GitHub command that reproduces the submitted score.

It's a good problem to tackle and there seems to be interesting ideas in the project. Although it is quite difficult to follow what the method is within the time frame specified for this review. I can't add points for the problem choice because it is a well established challenge so I have to judge only the methodology, which is unfortunately hard to follow.

Also I don't see a clear presentation of what the results were exactly (what does it mean to be 1.9x better than the benchmark? what are the scores measuring? maybe this is clear for someone very familiar with the WhiteBox ARC challenge but it is not made clear in the paper).