Interactive Assessments for AI Safety: A Gamified Approach to Evaluation and Personal Journey Mapping

A great idea to cement learning through interactive, and nuanced decision-making. References early on in the paper create a clear scaffold for the project. While it was difficult to understand the analysis of current courses, the problem was clearly defined ("independent learning can be improved by moving beyond self-report").

I would love to see more detail written about the analysis of AI safety courses in sections 1 and 2, and how you chose the important concepts for inclusion in your interactive materials. This would be useful for others to build on your work, and to aid understanding of the reader. For example, what was the method of your analysis, and what did your analysis show, for the sentence: "we analyzed learning patterns from AI safety education initiatives..." ? Some citations did not show strong relevance to the context in which they were cited (e.g. Li et al 2019 re: iterative learning).

The AI mentor was a good idea, and I wonder how it could be extended to provide support and feedback that truly leverages the opportunity of an LLM chatbot; the demonstration seemed to follow a scripted conversation, with limited insight beyond the materials already available to the learner.

It was good to see a preliminary evaluation of the project, though analysis on the breadth and depth of content covered, and access to the course materials for deeper evaluation, would be useful to aid understanding of the technical quality.

Nice work on the effort put into this project! FYI I don't have access to the course, so could only rely on the demo video.

Here are some things I thought were particularly good:

- I admire the breadth approach in incorporating lots of different mediums for users to learn through, it shows a lot of creativity.

- Nice touch with making sure it is accessible!

Here are some places where it could have been improved:

- Questions from your user feedback might be slightly leading, given they are framed as positives. In the future, I would recommend doing user interviews given your small sample size. I'd recommend the Mom Test for a guide on how to do these user interviews well. Doing user interviews would more likely yield higher quality responses and provide more info on areas to improve.

- I would have loved to see more details on how you designed the chatbot mentor and how it would adapt to learners with specific needs. One of our major challenges is curating content for each user's unique background and interests.

- I would have loved to learn more about how the debate and assignments are graded. Is a human doing the grading or would an LLM do so? How would you ensure that user's get informative learnings from the feedback?