Math Speaks All Languages: Enhancing LLM Problem-Solving Across Multilingual Contexts
Maksim Kostritsya, Kseniia Kuvshinova, Rauf Parchiev, Konstantin Polev · Team Round Tensor
Submitted to Reprogramming AI Models Hackathon. Sprint projects are early-stage work by participants, not Apart Research publications.
Large language models (LLMs) have shown significant adaptability in tackling various human issues; however, their efficacy in resolving mathematical problems remains inadequate. Recent research has identified steering vectors — hidden attributes that can guide the actions and outputs of LLMs. Nonetheless, the exploration of universal vectors that can consistently affect model responses across different languages is still limited. This project aims to confront two primary challenges in contemporary LLM research by utilizing the Goodfire API to examine whether common latent features can improve mathematical problem-solving capabilities, regardless of the language employed.
Reviews
This is creative paper which finds a new domain on which SAE features generalize well (across languages in grade school math).
The surprising finding that the French steering vectors had a larger impact on English and Russian performance than French performance warrants further inquiry. I think this paper could've significantly improved on novelty if it pursued this direction.
Given more time, I would also love to see the authors inspect whether the features they found generalize beyond GSM8K.
Good work!
It seems strange to use Google Translate or Mistral for translation when there are much better options available. This is especially problematic for challenging math problems. There’s also an inconsistency in Figure 1, where the numbers add up to 101 despite stating that there are 100 problems. French translations worked much worse, and I’m afraid the translations might not have been accurate. The text didn’t clarify it fully, but it seems they used Mistral for French and Google Translate for Russian. In general, the finding is interesting: they ran a math benchmark, used a contrastive method to find differences between correct and incorrect math answers, and checked if these differences transferred between languages—and they do. This provides evidence that the vector truly captures some sense of mathematical accuracy. That being said, they only evaluated the steering vector on samples where it was incorrect without steering. While this fixes some outcomes, it’s possible this also breaks previously correct responses. In French, it only corrected 2 out of 21, and I’d guess that some false answers might arise just from resampling. They also applied it to only 21 out of 77 false samples. I would also find it more interesting to take the correctness vector for a language in which the model performed better. Using a control feature and validating on all math problems could be beneficial too. You’d expect better results if you resampled on false outputs, so it would be interesting to see what happens if you steer for an unrelated feature.
Read full reviewShow less
Quite surprising results! Wasn’t expecting the model to have features which it doesn’t activate enough during inference that could boost performance. The results are very intriguing and warrant a further study to get a better understanding on the trade-offs for such features and if they just improve maths performance or other domains too.
Cite this project
@misc{kostritsya2024math,
title = {{Math Speaks All Languages: Enhancing LLM Problem-Solving Across Multilingual Contexts}},
author = {Maksim Kostritsya and Kseniia Kuvshinova and Rauf Parchiev and Konstantin Polev},
year = {2024},
month = nov,
note = {Submitted to Reprogramming AI Models Hackathon, an Apart Research Sprint},
howpublished = {\url{https://apartresearch.com/sprints/projects/math-speaks-all-languages-enhancing-llm-problem-solving-across-multilingual-contexts}},
url = {https://apartresearch.com/sprints/projects/math-speaks-all-languages-enhancing-llm-problem-solving-across-multilingual-contexts}
}More from Reprogramming AI Models Hackathon
- 1st place by peer reviewView project: AutoSteer: Weight-Preserving Reinforcement Learning for Interpretable Model Control
AutoSteer: Weight-Preserving Reinforcement Learning for Interpretable Model Control
AI Safety Initiative Groningen
Traditional fine-tuning methods for language models, while effective, often disrupt internal model features that could provide valuable insights into model behavior. We present a novel approach combining Reinforcement …
- View project: Classification on Latent Feature Activation for Detecting Adversarial Prompt Vulnerabilities
Classification on Latent Feature Activation for Detecting Adversarial Prompt Vulnerabilities
Feature Disruption Lab
We present a method leveraging Sparse Autoencoder (SAE)-derived feature activations to identify and mitigate adversarial prompt hijacking in large language models (LLMs). By training a logistic regression classifier on …
- View project: Utilitarian Decision-Making in Models - Evaluation and Steering
Utilitarian Decision-Making in Models - Evaluation and Steering
Byte To Brain
We design an eval based on the Oxford Utilitarianism Scale (OUS) that measures the model’s deontological vs utilitarian preference in a nine question, two factor model. Using this scale, we measure how feature steering …