LLM-prompt-optimiser based SAAS platform for evaluations
Anton ZHeltoukhov, Iulia Levin · Team Mesa
Submitted to AI Safety Entrepreneurship Hackathon. Sprint projects are early-stage work by participants, not Apart Research publications.
LLM evaluation SAAS platform built around model based prompt optimiser
Reviews
focus on improving evaluation quality through automated prompt optimization is interesting but may be too narrow in scope.
(1) Technical approach is sound but limited in scope.
(2) Addresses important but narrow aspect of safety evaluation.
(3) Implementation details need significant expansion.
Definitely a possible solution if you can get access to the prompts and evals from all theh top tier labs. However, this seems rather odd if it's just purely using what they've already created. Additionally, it might be hard to create standards for constantly changing questions.
This is a solid start, but it feels like a rough draft. The authors have clearly identified a problem and proposed a solution, but they could benefit from more detail and a more polished presentation.
Cite this project
@misc{zheltoukhov2025llmpromptoptimiser,
title = {{LLM-prompt-optimiser based SAAS platform for evaluations}},
author = {Anton ZHeltoukhov and Iulia Levin},
year = {2025},
month = jan,
note = {Submitted to AI Safety Entrepreneurship Hackathon, an Apart Research Sprint},
howpublished = {\url{https://apartresearch.com/sprints/projects/llm-prompt-optimiser-based-saas-platform-for-evaluations}},
url = {https://apartresearch.com/sprints/projects/llm-prompt-optimiser-based-saas-platform-for-evaluations}
}More from AI Safety Entrepreneurship Hackathon
- 1st place by peer reviewView project: AntiMidas: Building Commercially-Viable Agents for Alignment Dataset Generation
AntiMidas: Building Commercially-Viable Agents for Alignment Dataset Generation
the commonwealth
AI alignment lacks high-quality, real-world preference data needed to align agentic superintel- ligent systems. Our technical innovation builds on Pacchiardi et al. (2023)’s breakthrough in detecting AI deception …
- View project: Scoped LLM: Enhancing Adversarial Robustness and Security Through Targeted Model Scoping
Scoped LLM: Enhancing Adversarial Robustness and Security Through Targeted Model Scoping
FocusAI
Even with Reinforcement Learning from Human or AI Feedback (RLHF/RLAIF) to avoid harmful outputs, fine-tuned Large Language Models (LLMs) often present insufficient refusals due to adversarial attacks causing them to …
- View project: Prompt+question Shield
Prompt+question Shield
Seon's team
A protective layer using prompt injections and difficult questions to guard comment sections from AI-driven spam.