Register now to shape the future of AI cooperation!
The Concordia Contest is your opportunity to dive into the exciting world of cooperative AI. Whether you're an experienced AI researcher, a curious developer, or someone passionate about ensuring AI benefits humanity, this event is for you. As a participant, you will:
Collaborate in a diverse team to create groundbreaking AI agents
Learn from experts in AI safety, cooperative AI, and multi-agent systems, including Sasha Vezhnevets, Joel Leibo, Rakshit Trivedi, and Lewis Hammond
Contribute to solving real-world challenges like resource management and conflict resolution
Compete for prizes and the chance to be featured in a NeurIPS publication
Network with like-minded individuals and potential future collaborators
Register now and be part of the movement towards more cooperative, trustworthy, and beneficial AI systems. We will provide a low to no code interface via google collab which enables participation irrespective of prior coding experience.
As AI systems become more sophisticated and pervasive, we must develop agents capable of cooperating effectively with humans and other AIs. Understanding cooperation in AI agents is essential as we work towards a future where AI can navigate complex social scenarios, negotiate treaties, or manage shared resources—that could lead to groundbreaking solutions for global challenges!
As a precursor to the NeurIPS 2024 Concordia competition, we're inviting you to collaborate with researchers, programmers, and other participants to design AI agents that exhibit cooperative properties and dispositions across a variety of environments. The Concordia Challenge, built on the recently released Concordia framework, offers a unique opportunity to work with language model (LM) agents in intricate, text-mediated environments. You'll be tasked with developing agents capable of using natural language to effectively cooperate with each other, even in the face of challenges such as competing interests, differing values, and in mixed-motive settings.
Prize Pool: $2,000 (standard breakdown)
🥇 $1,000 for the top team
🥈 $600 for the second prize
🥉 $300 for the third prize
🏅 $100 for the fourth prize
Winning submissions may have the opportunity to collaborate and co-author a NeurIPS report -- in collaboration with authors from Cooperative AI Foundation, GoogleDeepMind, MIT, University of Washington, UC Berkeley, and University College London.
Whether you're an AI expert or new to the field, your unique perspective can contribute to this crucial area of research. The goal of this hackathon goes beyond winning prizes. Your participation contributes to the advancement of cooperative AI, potentially shaping the future of how AI systems interact with humans and each other. Good luck to all participants, and we look forward to seeing your innovative solutions!
The Concordia Framework Explained:
This diagram illustrates how agents interact within the Concordia environment:
Agents (like Dorothy and Charlie) make action attempts in natural language.
The Game Master (GM) processes these attempts and determines outcomes.
The GM generates event statements describing what actually occurred.
These events update the world state and are sent back to agents as observations.
This cycle creates a dynamic, text-based environment where AI agents must cooperate and communicate to achieve goals, mimicking complex social interactions. Your challenge is to create an agent that can thrive in these diverse scenarios.
Designing Your Agent:
Consider creating a backstory or set of experiences for your agent to inform its decision-making process.
Explore various psychological profiles, personality tests, or historical figures as inspiration for your agent's behavior.
Think about giving your agent a motivation or purpose, such as acting on behalf of a family or community.
Experiment with different approaches, such as using social psychology research or creating fictional memories to shape your agent's cooperative tendencies.
Remember that your agent should be able to adapt to various roles and scenarios, so avoid being too prescriptive in its behavior.
Collaboration and Development
We encourage participants to build in public, especially during the first 24 hours of the hackathon.
Share your ideas and get feedback from other participants to improve your agent's design.
Consider forming diverse teams to tackle the challenge from multiple perspectives.
Environments
We have several different environments that proceed in simultaneous-move rounds, controlled by a “game master” (like a storyteller)
Environments are mixed-motive, incomplete information, equilibrium selection problems that contain a “background population” of agents against which the participants are evaluated (similar to MeltingPot Contest)
Some environments are extremely open-ended so agents can suggest almost any action to the game master (who then resolves the action, determining whether or not they can do it using common sense and environment knowledge); other environments contain mixtures of open-ended free response choices and multiple choice questions
All environments track concrete variables (e.g. money) which we also use for scoring
The environments we have so far are: some text
Pub Coordination: A group of friends with individual pub preferences must coordinate their choices for a night out, balancing personal desires with group harmony while adapting to potential unexpected closures and engaging in social interactions.
Haggling: A fruit market in Fruitville where merchants engage in price negotiations, balancing potential profits against the risk of the co-player refusing the transaction.
Labor Collective Action: Workers must decide whether to strike or continue working in the face of wage cuts, navigating collective action problems and power dynamics with a boss and labor organizer.
Reality Show: Contestants participate in a series of minigames with varied game-theoretic structures, balancing cooperation and competition across different scenarios.
Chandler Smith is a research engineer at Cooperative AI. He was a Machine Learning Alignment and Theory Scholar (MATS) supervised by Jesse Clifton.
Organiser, Speaker and Mentor
Marta Bieńkiewicz
Marta has decade-plus of expertise in research delivery at the intersection of neuroscience and technology.
Organiser
Dr. Alexander (Sasha) Vezhnevets
Alexander (Sasha) Vezhnevets is a staff research scientist at Google DeepMind. He obtained his PhD from ETH Zurich in Machine Learning.
Keynote Speaker
Rakshit Trivedi
Rakshit S. Trivedi is a Postdoctoral Associate in the Computer Science and Artificial Intelligence
Laboratory (CSAIL) at MIT.
Speaker
Marwa Abdulhai
Marwa is a PhD student at the Berkeley Artificial Intelligence Research (BAIR) lab at UC
Berkeley, advised by Professor Sergey Levine.
Mentor
Oliver Slumbers
Oliver Slumbers is a PhD student at University College London, supervised by Prof. Jun Wang. His research centres around population / group dynamics in multi-agent systems.
Mentor
Esben Kran
Esben is the co-director of Apart Research and specializes in organizing research teams on pivotal AI security questions.
Organizer
Archana Vaidheeswaran
Archana is responsible for organizing the Apart Sprints, research hackathons to solve the most important questions in AI safety.
Organizer
Jason Schreiber
Jason is co-director of Apart Research and leads Apart Lab, our remote-first AI safety research fellowship.
Organizer
Natalia Pérez-Campanero Antolín
A research manager at Apart, Natalia has a PhD in Interdisciplinary Biosciences from Oxford and has run the Royal Society's Entrepreneur-in-Residence program.
Judge
Lewis Hammond
Lewis is based at the University of Oxford, where he is a DPhil candidate in computer
science. He is also the research director of the Cooperative AI Foundation.
Organiser
Jesse Clifton
Jesse Clifton is a research analyst at the Cooperative AI Foundation and a researcher at the Center
on Long-Term Risk, where he is focused on how to improve outcomes of interaction
Organiser
Joel Leibo
Joel Z. Leibo is a senior staff research scientist at Google DeepMind. He obtained his PhD from
MIT where he studied computational neuroscience and machine learning.
Organiser
To ensure you're well-equipped for the Concordia Challenge Hackathon, we've compiled a comprehensive set of resources to support your participation. These materials will help you understand the context, familiarize yourself with the tools, and spark ideas for your innovative solutions.
Key Resources:
Introduction to Cooperative AI curriculum: This curriculum is meant to provide an introduction to cooperative AI. There are also three optional background sessions focused on game theory, machine learning and AI safety, ethics and governance, respectively.
Concordia Open-Source Codebase: This is your primary toolkit for the hackathon. Familiarize yourself with the Concordia framework, which allows for creating situated social interactions with LM agents.
CAIF Website: Explore the Cooperative AI Foundation's site for broader context on the importance of cooperative AI.
Demo Video of Agent Development Watch this tutorial to understand how to develop and implement agents within the Concordia framework.
Starter Kit: We'll provide a comprehensive starter kit, including baseline agents, data-loading tools, and setup instructions. This will be your launchpad for creating innovative solutions.
Additional Readings: To deepen your understanding of the field, we recommend the following papers:
These resources will provide you with a solid foundation in cooperative AI concepts, challenges, and potential solutions. Remember, our team of mentors and experts will be available throughout the hackathon to answer questions and provide guidance. Don't hesitate to reach out and make the most of these resources as you work on your groundbreaking cooperative AI agents!
The schedule runs from 4 PM UTC Friday to 3 AM Monday. We start with an introductory talk and end the event during the following week with an awards ceremony. Join the public ICal here. You will also find Explorer events, such as collaborative brainstorming and team match-making before the hackathon begins on Discord and in the calendar.
📍 Registered jam sites
Beside the remote and virtual participation, our amazing organizers also host local hackathon locations where you can meet up in-person and connect with others in your area.
Hackathon Concordia de los Chatbots | AISUC x Apart Sprints
Friday 6: Meet up at CAi, San Joaquín UC Chile campus, Santiago. Pizza provided. Depending on demand, online or in person support for september 7th and 8th will be provided.
Hackathon Concordia de los Chatbots | AISUC x Apart Sprints
Thank you! Your submission has been received! Your event will show up on this page.
Oops! Something went wrong while submitting the form.
📣 Social media images and text snippets
No media added yet
No text snippets added yet
Participants will submit their agents through our evaluation platform, Eval AI. This platform allows us to test your agent across various Concordia environments, including held-out scenarios with new co-players.
What to Submit:
Your agent file, implemented according to the Concordia agent API
A brief description of your approach and any noteworthy features of your agent
Evaluation Criteria: Your submission will be scored based on several factors:
Cooperative Performance: How well does your agent cooperate with others to achieve common goals?
Adaptability: Can your agent perform well in unforeseen scenarios with new co-players?
Innovation: Does your approach bring novel ideas to the field of cooperative AI?
Ethical Considerations: How well does your agent balance its objectives with broader social welfare?
Key Points to Remember:
There will be a strict limitation on the number of Large Language Model (LLM) calls an agent can make per step to ensure a level playing field.
Top performers may be invited to contribute to a NeurIPS publication, offering a unique opportunity to showcase your work to the broader AI research community.
While releasing source code is not required for leaderboard acknowledgment, prize winners may be asked to share their code for verification purposes.
Frequently Asked Questions
Q: Do I need coding experience to participate? A: No! We provide a low to no-code interface via Google Colab, making participation accessible regardless of coding experience.
Q: What language model should I use for development? A: We recommend using Codestral for the best results. However, you can use other models like Mistral or Llama during development.
Q: How will my agent be evaluated? A: Agents will be scored based on their individual rewards across multiple environments. We use a version of the Elo score to derive a final overall score across held-out environments and background populations.
Q: Can I participate if I'm new to AI or cooperative AI? A: Absolutely! We welcome participants of all experience levels. We provide resources and mentorship to help you get started. Check out the resources tab and this demo video.
Q: What if I can't attend the entire hackathon? A: While we encourage full participation, you can still contribute and learn even if you can't attend all sessions. The hackathon is designed to be flexible.
Q: How are teams formed A: You can form your own team or join our team-matching sessions at the beginning of the hackathon to find collaborators.
Q: What resources are available during the hackathon? A: We provide a comprehensive starter kit, access to mentors, office hours with experts, and a supportive community on Discord.
Q: What language model should I use for development? A: We recommend using Codestral for the best results. It's fast, generates good stories with creativity, and isn't too cooperative by default. You can also use other models like Mistral or Llama during development.
Q: Can I optimize my agent's prompts or code programmatically? A: While you can optimize your agent, it's important to keep it general enough to perform well across various scenarios and environments. Over-optimizing for specific situations may not yield good performance in the final evaluation, which includes held-out scenarios.
Q: Do I need to create different agents for each environment? A: No, you should create a single agent that can perform well across all environments. Your agent should be designed to be flexible and adaptable to various scenarios, including potentially unseen ones in the final evaluation
Q: How will the agents be evaluated? A: Agents will be evaluated using an ELO-based system across multiple scenarios and environments, including held-out ones. The final leaderboard will be determined by performance on these held-out scenarios to ensure generalizability.
Q: Can I use personality tests or profiles to design my agent? A: Yes, you can incorporate personality test results, psychological profiles, or even fictional character traits into your agent design. However, be mindful that the agent may receive additional context or memories during the evaluation, so your design should be flexible.
Submission Guidelines
Congratulations on completing your agent for the Concordia Contest Hackathon! Please follow these steps to submit your work:
Submit your agent file through this submission document. This is where your agent will be evaluated.
Complete the CAIF Diversity, Equity and Inclusion Survey. Your submission will only be considered valid once you've filled out this form. The survey helps us understand the make-up of our community and ensure we're representing and including minority groups.
If your team is selected as one of the top 4, you will be required to submit a detailed report using the provided template. This report should include:
Starting today(Day 2), we will run submitted agents through the evaluation script available in the GitHub repository to generate an unofficial leaderboard.
This unofficial leaderboard will give you a comparative understanding of your agent's performance.
Winners will be decided after the weekend and evaluated on a suite of held-out scenarios.
We will accept agents for the first leaderboard check-in through mid-morning US time today(Day 2) so there's no rush.
Please be patient as we work to find a balance of accuracy and speed in evaluations.
Important Notes:
All participants must submit their agent through evals.ai and complete the DEI survey.
Only the top 4 teams will need to prepare the detailed report
The report must adhere to the 4-page limit for the main content (excluding references and optional appendix).
Ensure your submission is in the spirit of the contest. Agents designed for hacking, sabotage, or other malicious purposes will be disqualified.
The final evaluation will be run using the Codestral model for all submissions to ensure fairness.
Feedback:
The submission document includes a form for submitting feedback. Please let us know what can be made clearer or improved!
You have successfully submitted! You should receive an email and your project should appear here. If not, contact operations@apartresearch.com.
Oops! Something went wrong while submitting the form.
Here are the top winners from the Concordia Contest
%2520(1).png)
%20Vezhnevets.png)
.png)
