✨ Overview

Shape the future of AI systems through smarter model routing and evaluation! Join us for a groundbreaking hackathon focused on creating judges that evaluate model outputs and routers that direct queries to the optimal models. Whether you're a researcher, developer, or ML enthusiast, this hackathon offers a unique opportunity to tackle crucial challenges in AI optimization and safety.

Judges and routers play a crucial role in AI systems by optimizing decision-making and task delegation. Judges evaluate the quality, accuracy, or relevance of AI-generated outputs, ensuring models produce reliable results. Routers, direct queries or tasks to the most suitable model, improving efficiency, performance and robustness by leveraging specialized AI models. Together, these components enhance AI safety and deliver better model accuracy than any one existing model can. These components better support model specialization - democratizing the model creation process. This approach and Mechanistic Interpretability techniques enable new ways to understand model capabilities.

We're thrilled to partner with Martian, pioneers in model routing technology, to bring you this specialized hackathon. Their groundbreaking approach aggregates the best capabilities across multiple AI models, achieving higher performance than any single model while reducing costs.

Each participating team will receive $50 in model API credits to power their projects, with additional credits available for promising implementations. You'll have access to Martian's judge and router APIs, along with sample code libraries to kickstart your projects.

Sign up here to stay updated for this event

Why This Hackathon Matters:

The Model Mapping Hackathon addresses fundamental challenges in AI deployment as models continue to proliferate:

• Efficiency & Cost Reduction: By routing queries to the most appropriate models, we can dramatically reduce computational costs while improving results

• Democratizing AI Creation: Specialized models excel in narrow domains but struggle with universal tasks - router technology enables these specialized models to thrive

• Safety & Reliability: Better evaluation mechanisms (judges) help ensure outputs meet safety, ethical, and quality standards before deployment

• Understanding Model Capabilities: Developing judges requires deeper mechanistic understanding of how models work and their inherent limitations

Challenge Tracks

Creating Judges Track

1. Model Characteristic Analysis: Create a dataset that tests an interesting model characteristic pertinent to safety (e.g., ethics, hallucinations, gender bias). Build a judge using this data and evaluate multiple models.

2. Judge Evaluation Metrics: Develop methods to measure judge accuracy, completeness, and reliability for specific characteristics.

3. Mechanistic Interpretability for Judges: Apply MI techniques to model internals to create better or more interpretable judges that can evaluate outputs based on how they were generated.

Creating Routers Track

1. Risk-Sensitive Routing: Build efficient routing algorithms considering judge scores, dollar costs, and system reliability.

2. Multi-Objective Routing: Create routers that balance multiple evaluation criteria (e.g., ethics and legality) according to user preferences.

3. Task Decomposition: Develop systems that break complex tasks into subtasks that can be routed to specialized models before recombining results.

Inferring Judges & Routers Track

1. Reverse Engineering: Given a black-box LLM or router, infer its embedded judge (reward signal) for specific characteristics.

2. Efficiency Analysis: Quantify potential electricity/resource consumption reduction from widespread adoption of optimal routing technologies.