Nov 25, 2024

Math Speaks All Languages: Enhancing LLM Problem-Solving Across Multilingual Contexts

Maksim Kostritsya, Kseniia Kuvshinova, Rauf Parchiev, Konstantin Polev

Details

Arrow

Summary

Large language models (LLMs) have shown significant adaptability in tackling various human issues; however, their efficacy in resolving mathematical problems remains inadequate. Recent research has identified steering vectors — hidden attributes that can guide the actions and outputs of LLMs. Nonetheless, the exploration of universal vectors that can consistently affect model responses across different languages is still limited. This project aims to confront two primary challenges in contemporary LLM research by utilizing the Goodfire API to examine whether common latent features can improve mathematical problem-solving capabilities, regardless of the language employed.

Cite this work:

@misc {

title={

Math Speaks All Languages: Enhancing LLM Problem-Solving Across Multilingual Contexts

},

author={

Maksim Kostritsya, Kseniia Kuvshinova, Rauf Parchiev, Konstantin Polev

},

date={

11/25/24

},

organization={Apart Research},

note={Research submission to the research sprint hosted by Apart.},

howpublished={https://apartresearch.com}

}

Review

Arrow

Reviewer's Comments

Arrow

Simon Lermen

It seems strange to use Google Translate or Mistral for translation when there are much better options available. This is especially problematic for challenging math problems. There’s also an inconsistency in Figure 1, where the numbers add up to 101 despite stating that there are 100 problems. French translations worked much worse, and I’m afraid the translations might not have been accurate. The text didn’t clarify it fully, but it seems they used Mistral for French and Google Translate for Russian. In general, the finding is interesting: they ran a math benchmark, used a contrastive method to find differences between correct and incorrect math answers, and checked if these differences transferred between languages—and they do. This provides evidence that the vector truly captures some sense of mathematical accuracy. That being said, they only evaluated the steering vector on samples where it was incorrect without steering. While this fixes some outcomes, it’s possible this also breaks previously correct responses. In French, it only corrected 2 out of 21, and I’d guess that some false answers might arise just from resampling. They also applied it to only 21 out of 77 false samples. I would also find it more interesting to take the correctness vector for a language in which the model performed better. Using a control feature and validating on all math problems could be beneficial too. You’d expect better results if you resampled on false outputs, so it would be interesting to see what happens if you steer for an unrelated feature.

Recent Projects

Mar 24, 2025

Attention Pattern Based Information Flow Visualization Tool

Understanding information flow in transformer-based language models is crucial for mechanistic interpretability. We introduce a visualization tool that extracts and represents attention patterns across model components, revealing how tokens influence each other during processing. Our tool automatically identifies and color-codes functional attention head types based on established taxonomies from recent research on indirect object identification (Wang et al., 2022), factual recall (Chughtai et al., 2024), and factual association retrieval (Geva et al., 2023). This interactive approach enables researchers to trace information propagation through transformer architectures, providing deeper insights into how these models implement reasoning and knowledge retrieval capabilities.

Read More

Mar 25, 2025

Safe ai

The rapid adoption of AI in critical industries like healthcare and legal services has highlighted the urgent need for robust risk mitigation mechanisms. While domain-specific AI agents offer efficiency, they often lack transparency and accountability, raising concerns about safety, reliability, and compliance. The stakes are high, as AI failures in these sectors can lead to catastrophic outcomes, including loss of life, legal repercussions, and significant financial and reputational damage. Current solutions, such as regulatory frameworks and quality assurance protocols, provide only partial protection against the multifaceted risks associated with AI deployment. This situation underscores the necessity for an innovative approach that combines comprehensive risk assessment with financial safeguards to ensure the responsible and secure implementation of AI technologies across high-stakes industries.

Read More

This work was done during one weekend by research workshop participants and does not represent the work of Apart Research.