Guardian AI: Scam detection and prevention

Our solution is to create robust and comprehensive benchmarks for specialized contexts and modalities. Through the creation of smaller, in-depth benchmarks, we aim to construct an overarching benchmark that includes performance from the smaller benchmarks. This would help mitigate AI harms and biases by focusing on inclusive and equitable benchmarks.

We bet on agentic AI being integrated into other domains within the next few years: healthcare, manufacturing, automotive, etc., and the way it would be integrated is into cyber-physical systems, which are systems that integrate the computer brain into a physical receptor/actuator (e.g. robots). As the demand for cyber-physical agents increases, so would be the need to train and align them.

We also bet on the scenario where frontier AI and robotics labs would not be able to handle all of the demands for training and aligning those agents, especially in specific domains, therefore leaving opportunities for other players to fulfill the requirements for training those agents: providing a dataset of scenarios to fine-tune the agents and providing the people to give feedback to the model for alignment.

Furthermore, we also bet that human intervention would still be required to supervise deployed agents, as demanded by various regulations. Therefore leaving opportunities to develop supervision platforms which might highly differ between different industries.

Open-world environments for evaluating Language Model agents

Steer aims to be an API that helps developers, researchers, and businesses steer open-source LLMs away from societal biases, and towards the use-cases that they need. To do this, Steer uses activation additions, a fairly new technique with great promise. Developers can simply enter steering prompts to make open models have safer and task-specific behaviors, avoiding the hassle of data collection and human evaluation for fine-tuning, and avoiding the extra tokens required from prompt-engineering approaches.

This project proposes a cryptographic system for assigning unique identities to AI models and verifying their outputs to ensure accountability and traceability. By leveraging these techniques, we address the risks of AI misuse and untraceable actions. Our solution aims to enhance AI safety and establish a foundation for transparent and responsible AI deployment.

The AI Safety Collective is a global platform designed to enhance AI safety by crowdsourcing solutions to critical AI Safety challenges. As AI systems like large language models and multimodal systems become more prevalent, ensuring their safety is increasingly difficult. This platform will allow AI companies to post safety challenges, offering bounties for solutions. AI Safety experts and enthusiasts worldwide can contribute, earning rewards for their efforts. The project focuses initially on non-catastrophic risks to attract a wide range of participants, with plans to expand into more complex areas. Key risks, such as quality control and safety, will be managed through peer review and risk assessment. Overall, The AI Safety Collective aims to drive innovation, accountability, and collaboration in the field of AI safety.

The CAMARA project presents a cutting-edge, adaptive multi-agent framework designed to significantly bolster AI safety by identifying and mitigating vulnerabilities in AI systems such as Large Language Models. As AI integration deepens across critical sectors, CAMARA addresses the increasing risks of exploitation by advanced adversaries. The framework utilizes a network of specialized agents that not only perform traditional red-teaming tasks but also execute sophisticated adversarial attacks, such as token manipulation and gradient-based strategies. These agents collaborate through a shared knowledge base, allowing them to learn from each other's experiences and coordinate more complex, effective attacks. By ensuring comprehensive testing of both standalone AI models and multi-agent systems, CAMARA targets vulnerabilities arising from interactions between multiple agents, a critical area often overlooked in current AI safety efforts. The framework's adaptability and collaborative learning mechanisms provide a proactive defense, capable of evolving alongside emerging AI technologies. Through this dual focus, CAMARA not only strengthens AI systems against external threats but also aligns them with ethical standards, ensuring safer deployment in real-world applications. It has a high scope of providing advanced AI security solutions in high-stake environments like defense and governance.

Conflict of interest declaration: I advised Fazl on a funding request he was working on.

Re publishing: This PDF would require further modifications before publication.

I want to train (amplified) imitation agents of people who are wise to provide advice on navigating conflicting considerations when figuring out how to train and deploy AI safely.

Path to Impact: Train wise AI advisors -> organisations make better decisions about how to train and deploy AI -> safer AGI -> better outcomes for humanity

What is wisdom? Why focus on increasing wisdom? See image

Why use amplified imitation learning?

Attempting to train directly on wisdom suffers from the usual problems of the optimisation algorithm adversarially leveraging your blind spots, but worse because wisdom is an especially fuzzy concept.

Attempting to understand wisdom from a principled approach and build wise AI directly would require at least 50 years and iteration through multiple paradigms of research.

In contrast, if our objective is to imitation folk who are wise, we have a target that we can optimise hard on. Instead of using reinforcment learning to go beyond human level, we use amplification techniques like debate or iterated amplification.

How will these agents advise on decisions?

The humans will ultimately make the decisions. The agents don't have to directly tell the humans what to do, they simply have to inspire the humans to make better decisions. I expect that these agents will be most useful in helping humans figuring out how to navigate conflicting principles or frameworks.

With multi-agent systems poised to be the next big-thing in AI for productivity enhancement, the next phase of AI commercialization will centre around how to deploy increasingly complex multi-step automated processes that reliably align with human values and objectives. To bring trust and efficiency to agent systems, we are introducing a multi-agent collaboration platform (Ælign) designed to supervise and ensure the optimal operation of autonomous agents via multi-protocol alignment.

We show that state of the art LLMs can be jailbroken by adversarial multimodal inputs, and that this can lead to dangerous scenarios if these LLMs are used as planners in robotics. We propose finetuning small multimodal language models to act as guardrails in the robot's planning pipeline.

DarkForest is a pioneering Human Content Verification System (HCVS) designed to safeguard the authenticity of online spaces in the face of increasing AI-generated content. By leveraging graph-based reinforcement learning and blockchain technology, DarkForest proposes a novel approach to safeguarding the authentic and humane web. We aim to become the vanguard in the arms race between AI-generated content and human-centric online spaces.