Inspiration

We have many ideas available for inspiration on the aisi.ai Interpretability Hackathon ideas list. A lot of interpretability research is available on distill.pub, transformer circuits, and Anthropic's research page.

Introductions to mechanistic interpretability

• Catherine Olsson's on getting starter with mechanistic interpretability research

  • 02:00 Start of the talk

  • 03:30 What is mechanistic interpretability?

  • Slideshow

• A video walkthrough of A Mathematical Framework for Transformer Circuits.

• The Transformer Circuits YouTube series

• Jacob Hilton's deep learning curriculum week on interpretability

• An annotated list of good interpretability papers, along with summaries and takes on what to focus on.

• Christoph Molnar's book about traditional interpretability

• Neel's barebones prerequisites for mechanistic interpretability research

See also the tools available on interpretability:

• Redwood Research's interpretability tools: http://interp-tools.redwoodresearch.org/

• The activation atlas: https://distill.pub/2019/activation-atlas/

• The Tensorflow playground: https://playground.tensorflow.org/

• The Neural Network Playground (train simple neural networks in the browser): https://nnplayground.com/

• Visualize different neural network architectures: http://alexlenail.me/NN-SVG/index.html

Digestible research

• Opinions on Interpretable Machine Learning and 70 Summaries of Recent Papers summarizes a long list of papers that is definitely useful for your interpretability projects.

• Distill publication on visualizing neural network weights

• Andrej Karpathy's "Understanding what convnets learn"

• Looking inside a neural net

• 12 toy language models designed to be easier to interpret, in the style of a Mathematical Framework for Transformer Circuits: 1, 2, 3 and 4 layer models, for each size one is attention-only, one has GeLU activations and one has SoLU activations (an activation designed to make the model's neurons more interpretable - https://transformer-circuits.pub/2022/solu/index.html) (these aren't well documented yet, but are available in EasyTransformer)

• Anthropic Twitter thread going through some language model results

Below is a talk by Esben on a principled introduction to interpretability for safety: