Reprogramming AI Models Hackathon

Use this template for your submission [Required]

‍

Template in Overleaf/LaTeX

‍

Submission Requirements

Each team should submit a research paper that includes:

1. Project title and team members
2. Executive summary (max 250 words)
3. Introduction and problem statement
4. Methodology and approach
5. Results and analysis
6. Discussion of implications for AI interpretability
7. Conclusion and future work
8. References

Additionally, teams should provide:

• A link to their code repository (e.g., GitHub)
• Any demo materials or visualizations (if applicable)

Evaluation Criteria

Submissions will be judged based on the following criteria:

1. Interpretability Advancement
   • Does the project contribute to the field of AI interpretability?
   • Does it provide new insights into understanding or steering AI model behavior?
   • How well does it align with the hackathon's focus on reprogramming AI models?
   • How original and innovative is the approach?
2. Relevance for AI Safety
   • How important is the contribution to advancing the field of AI safety?
   • Do we expect the results to generalize beyond the specific case(s) presented in the submission?
   • Does the approach introduce new safety mechanisms or enhance existing ones in innovative ways?‍
3. Methodology & presentation‍
   • How well is the project executed from a technical standpoint?
   • Is the code well-structured, documented, and reproducible?
   • How effectively does it utilize Goodfire's SDK/API and other provided resources?
   • How clearly and effectively is the research presented in the paper?
   • Quality of visualizations and demos (if applicable)
   • Clarity of methodology explanation and results interpretation

Key Points to Remember:

• Focus on creating tools that provide meaningful insights into model behavior or enable precise modifications.
• Consider how your approach might scale to larger, more complex AI systems.
• Ethical considerations should be at the forefront of your design process.

💡 Frequently Asked Questions

General Questions

Q: What is the timeline for the hackathon?

A: The hackathon runs from Friday, 22nd November, through Monday, 25th November(3 AM UTC). There will be two brainstorming sessions (Tuesday and Thursday), with API access beig granted starting Thursday morning.

Q: What team size is recommended?

A: Teams should be 2-4 people, with a maximum of 5. While individual submissions are allowed, team participation is encouraged for better project outcomes.

Q: How do we get access to the API?

A:

1. Form your team
2. Have the team lead fill out the registration form
3. Receive API access credentials (starting Thursday morning 21st November)

Technical Questions

SDK Specifics

Q: What exactly do we have access to in the SDK?

A: The SDK provides:

• Feature inspection and intervention on Llama 3 8B and 70B
• Single residual stream layer access
• Feature activation analysis
• Contrastive and semantic search
• No raw neuron activations or decoder weights

Q: How do the different search methods work?

A:

1. Semantic search: Uses embedding similarity with cosine similarity
2. Contrastive search: Compares feature activations between datasets
3. Reranking: Filters features based on relevance to specific queries

Q: What are the recommended weight ranges for feature intervention?

A:

• Range: -1 to 1
• Emphasis: Start with 0.5
• Ablation: Start with -0.3
• Multiple features can be modified simultaneously

Feature Understanding

Q: How can we find relevant features for our task?

A: The SDK provides several methods:

1. Search: Query features using string searches
2. Inspect: Analyze feature activations for specific inputs
3. Contrast: Compare features between different datasets

Q: Can we modify feature activations?

A: Yes, you can perform feature interventions to modify model behavior, including conditional interventions.

Q: Are the SAE features proprietary?

A: Yes, Goodfire's SAE features are proprietary but may be open-sourced in the future.

‍

Q: How are features discovered and organized?

A:

• Features correspond to frequency in LMSIS training data
• More common concepts have better feature representation
• Some moderation for safety concerns
• Features have associated correctness scores

Q: What types of features are most reliable?

A:

• Common properties and concepts
• Frequently discussed topics
• Specific nouns and properties
• General behavioral patterns

Q: Can we modify the feature discovery process?

A: No, features are pre-discovered, but you can:

• Use contrast analysis to find relevant features
• Combine multiple features
• Create feature groups
• Experiment with feature weights

Technical Implementation

Q: How do we handle data preparation?

A:

• Can use external libraries for PDF processing
• OCR tools available for document processing
• Text extraction supported
• Dataset creation tools available

Q: Can we integrate with other frameworks?

A:

• Yes, SDK works alongside other tools
• OpenAI-compatible API interface
• Can combine with external libraries
• Support for custom evaluation frameworks

Q: What are the computational requirements?

A:

• $50 compute credits per team
• Rate limits with exponential backoff
• Can request increased limits with justification
• Efficient API usage recommended

Project Development

Q: How should we approach feature exploration?

A:

1. Start with semantic search
2. Use contrastive analysis for refinement
3. Experiment with feature weights
4. Build on example notebooks

Q: What are good starting points for projects?

A:

• Chain of thought analysis
• Feature visualization
• Model behavior steering
• Ethical reasoning investigation

Q: How can we evaluate our results?

A:

• Use provided evaluation frameworks
• Compare with baseline performance
• Test across different contexts
• Document feature interactions

Support & Resources

Q: Where can we get help during the hackathon?

A: You can:

1. Use the help desk channel on Discord
2. Contact the technical support team directly
3. Attend office hours (Saturday/Sunday)
4. Participate in hack talks

Q: What if we need increased rate limits?

A: Submit a request through the provided form with justification, and the team will review it case by case.

Q: Are there example notebooks available?

A: Yes, Goodfire provides several example notebooks, including:

• Feature exploration examples
• Decision tree with features
• Latent space exploration

Q: How can we submit feature requests or feedback?

A: While immediate changes during the hackathon may not be possible, Goodfire welcomes all feedback and feature requests for future improvements.

Q: What if we encounter technical issues?

A:

• Use Discord help desk
• Contact technical support
• Attend office hours
• Check documentation

Q: How can we get help with project direction?

A:

• Participate in hack talks
• Join brainstorming sessions
• Use project/teams channel
• Request feedback from organizers

‍