Computational Mechanics Hackathon

Unsupervised Recovery of Hidden Markov Models from Transformers with Evolutionary Algorithms

Team Stargazy Pie

Prior work finds that transformer neural networks trained to mimic the output of a Hidden Markov Model (HMM) embed the optimal Bayesian beliefs for the HMM's current state in their residual stream, which can be recovered via linear regression. In this work, we aim to address the problem of extracting information about …

Looking forward to posterity: what past information is transferred to the future?

Team Nonkansa

I used mechanistic interpretability techniques to try to see what information the provided Random Randox XOR transformer looks at when making predictions by examining its attention heads manually. I find that earlier layers pay more attention to the previous two tokens, which would be necessary for computing XOR, than …

Investigating the Effect of Model Capacity Constraints on Belief State Representations

Team Studying Generalization and Abstraction

Computational mechanics provides a formal framework for understanding the concepts needed to perform optimal prediction. Abstraction and generalization seem core to the function of intelligent systems, but are not yet well understood. Computational mechanics may present a promising approach to studying these …

Belief State Representations in Transformer Models on Nonergodic Data

Team Nerds

We extend research that finds representations of belief spaces in the activations of small transformer models, by discovering that the phenomenon also occurs when the training data stems from Hidden Markov Models whose hidden states do not communicate at all. Our results suggest that Bayesian updating and internal …

RNNs represent belief state geometry in hidden state

Team Keenan

The Shai et al. experiments on transformers (finding belief state geometry in the residual stream) have been replicated in RNNs with the hidden state instead of the residual stream. In general the belief state is stored linearly, but not in any particular layer, rather spread out across layers.

Steering Model’s Belief States

Team SimplexBreath

Recent results in Computational Mechanics show that Transformer models trained on Hidden Markov Models (HMM) develop a belief state geometry in their residuals streams, which they use to keep track of the expected state of the HMM, to be able to predict the next token in a sequence. In this project, we explored how …

Handcrafting a Network to Predict Next Token Probabilities for the Random-Random-XOR Process

For this hackathon, we handcrafted a network to perfectly predict next token probabilities for the Random-Random-XOR process. The network takes existing tokens from the process's output, computes several features on these tokens, and then uses these features to calculate next token probabilities. These probabilities …

Exploring Hierarchical Structure Representation in Transformer Models through Computational Mechanics

Team Aalto EA

This work attempts to explore how an approach based on computational mechanics can cope when a more complex hierarchical generative process is involved, i.e, a process that comprises Hidden Markov Models (HMMs) whose transition probabilities change over time. We find that small transformer models are capable of …