Empirical Measurements of Technique Effectiveness Across Model Sizes

* I like the use of the safetywashing paper to decrease the amount that the assessments you use are actually measuring capabilities.

* While TQA is a somewhat easy dataset to work with, it has severe limitations and really doesn't measure "truthfullness" at all (just a side note from someone who has worked extensively with the dataset). The fact that you found differing results from Ren et al. on this also backs up that we shouldn't be using the dataset anymore; it's incredibly confusing to work with due to it's non-standard design, meaning that almost no one actually uses the same assessment methodology. I'll stop bashing on TQA now...

* I really appreciate the breadth of data shared in the submission, but this makes it quite difficult to parse 'takeaways' from the different figures (not to mention the figure sizes are far too small). It would be beneficial to disentangle different aspects of the data in a more thorough study, but I understand that this would require replicated the studies used, which isn't feasible for various reasons.

* Another aspect which would be interesting to see in parallel with what is presented is what the cost of the safety intervention looks like, and how that scales with model size.

* I appreciate that you point out a key failure mode of AI safety benchmarking works! This is quite important, and the more people we have who know why comprehensive assessment and reporting is valuable, the better this will be in the future.

The project focuses on understanding how safety intervention techniques scale with model size. First, they look at the performance of the models on the safety benchmarks. Then, they apply safety intervention techniques and see how much they improve on the benchmarks, finding that the performance gains of different techniques differ depending on the benchmark.

I generally think that this is pretty interesting work. My main concern here would be how it is probably less likely for models to continuously get bigger in model size, so an x-axis of capabilities score (similar to the safety washing paper) may be more appropriate for better anchoring the abilities of the paper.

However, I am also concerned about the lack of details in terms of the implementation of the safety interventions: most of them are likely to be sensitive to theirhyperparameters, and I would not be sure how thorough the interventions were implemented. I think this is fine given as a weekend hackathon project, and would encourage the authors to further look into how the important determining the right hyperparameters of these techniques are for this project.

Lastly, it may also be valuable for the authors to focus on one or two specific interventions they think have promise in scaling.