Devising Effective Bechmarks
Nancy Vigil, Ashish Rai · Team Harshest Critics
Submitted to Hackathon for Technical AI Safety Startups. Sprint projects are early-stage work by participants, not Apart Research publications.
Our solution is to create robust and comprehensive benchmarks for specialized contexts and modalities. Through the creation of smaller, in-depth benchmarks, we aim to construct an overarching benchmark that includes performance from the smaller benchmarks. This would help mitigate AI harms and biases by focusing on inclusive and equitable benchmarks.
Reviews
No public critique yet.
Cite this project
@misc{vigil2024devising,
title = {{Devising Effective Bechmarks}},
author = {Nancy Vigil and Ashish Rai},
year = {2024},
month = sep,
note = {Submitted to Hackathon for Technical AI Safety Startups, an Apart Research Sprint},
howpublished = {\url{https://apartresearch.com/sprints/projects/devising-effective-bechmarks}},
url = {https://apartresearch.com/sprints/projects/devising-effective-bechmarks}
}More from Hackathon for Technical AI Safety Startups
- 1st place by peer reviewView project: DarkForest - Defending the Authentic and Humane Web
DarkForest - Defending the Authentic and Humane Web
DarkForest
DarkForest is a pioneering Human Content Verification System (HCVS) designed to safeguard the authenticity of online spaces in the face of increasing AI-generated content. By leveraging graph-based reinforcement …
- View project: Jailbreaking general purpose robots
Jailbreaking general purpose robots
Luax Labs
We show that state of the art LLMs can be jailbroken by adversarial multimodal inputs, and that this can lead to dangerous scenarios if these LLMs are used as planners in robotics. We propose finetuning small multimodal …
- View project: AI Safety Collective - Crowdsourcing Solutions for Critical AI Safety Challenges
AI Safety Collective - Crowdsourcing Solutions for Critical AI Safety Challenges
AI Safety Collective
The AI Safety Collective is a global platform designed to enhance AI safety by crowdsourcing solutions to critical AI Safety challenges. As AI systems like large language models and multimodal systems become more …