TL;DR AI safety is crucial, yet the vast majority of investment in AI is funneled into advancing capabilities rather than ensuring safety. Big Labs like OpenAI, once devoted to safety, now prioritize speed and commercialization. While Big Tech pushes forward with a “move fast and break things” mentality, relatively unsustainable and underfunded non-profits struggle to fill the gap. For-profit AI safety ventures also face significant challenges: A reluctance within the field to pursue them in the first place, the current market dominance of Big Tech, and the risk of failing or becoming misaligned. Yet, examples like the Traitorous Eight and SpaceX show small groups can have a big, positive impact. Anthropic, the leading for-profit AI safety company, proves you can scale while staying aligned - but new AI safety startups might want to focus on the frontier of technical challenges that lie outside the compute-heavy AI race.

Bill Gates, congressman Ted Lieu, and Turing Award winner Yoshua Bengio, among many others, signed a statement to put the risks from AI on the same level as those from pandemics and nuclear war. The consensus among experts is quite clear: Mitigating the risk of AI extinction is a top priority [1]. In other words, AI safety is really important. Despite this, the dynamics within Big Tech and between superpowers like the US and China lead to neglect of safety. While billions are funneled into advancing AI capabilities, a mere fraction of that investment is directed toward safety research. This disconnect forces us to question the ambitions we set in our commitment to AI safety.

The current landscape

OpenAI

OpenAI, once seen as highly committed to AI safety, has shifted its priorities. Their ‘Superalignment Team’ (the one focused on existential risks from advanced AI) effectively disassembled itself amid boardroom drama rumored to be about (you guessed it) the priority of AI safety [2]. Key figures like Jan Leike, Ilya Sutskever, and John Schulman exited, signalling a departure from their original safety-first stance [3]. Sutskever, perhaps the most pivotal figure in the deep learning revolution, since went on to found "Safe Superintelligence Inc." [4]. And of course there’s the OpenAI CEO, Sam Altman,  with remarks like, "I don't care whether we burn $50B a year... we are building AGI [artificial general intelligence]," [5].

Big Tech

Meanwhile, Big Tech - Google, Microsoft, and Meta - might already be doing exactly that: Unapologetically driving forward with the “move fast and break things” mentality (originally coined by Facebook and Zuck himself). With a collective investment of over $189 billion in AI projected for 2024 [6] and possibly hundreds of billions more to follow, it’s hard to imagine any executive team, let alone a single manager, having the power to slow down this juggernaut. Meta, arguably the most risk-tolerant of them all, recently released an open-source & open-weights model on par with the most advanced systems available. Needless to say, the model was jailbroken within days, raising concerns that it could be misused for developing bio-weapons or malware [7].

Non-Profit

‍This is where non-profit AI safety efforts attempt to step in, trying to fill the void left by Big Labs and Big Tech. Foundations like OpenPhil, Survival and Flourishing Foundation, and Schmidt Futures have funded hundreds of researchers to work on making advanced AI safe. As promising as this might sound, there’s perhaps another bitter lesson to be learned in AI: The resources available to non-profits are dwarfed by the massive budgets of the for-profit space. With AI capability research, tens of billions are invested in data centers alone [8], and implementation teams of single organisations can reach the size of the whole AI safety ecosystem [9][10]. In contrast, there is less than $1 billion total funding available for technical AI safety research [11].  As we’ve argued in our previous post, if we’re serious about scaling AI safety research, we need to leverage the resources of the for-profit world.

AI safety is financially constrained and academic research often doesn’t translate into real-world impact.

Funding constraint

AI safety research has long been rooted in the non-profit sector. From OpenAI’s early days as a non-profit to organizations like the Future of Life Institute, METR, and the Center for AI Safety - the field has depended heavily on charitable donations. As a non-profit ourselves at Apart, we’ve managed to raise several hundred thousand dollars this year, and while I like the non-profit model as much as the next guy, I’m realistic about its limitations. Non-profits lack an engine for growth and don’t benefit from the compound interest that drives for-profit ventures. They remain perpetually at the mercy of their funders - indefinitely.

While I’m immensely grateful for the few funders who have supported much of the critical AI safety work to date, relying on a handful of individuals backing those foundations to secure the future of AI safety isn’t a sustainable strategy. Unlike a for-profit, non-profits don’t enjoy an average of 8% market growth, nor can they count on fresh capital year after year to keep them afloat. Organizations which received funding through SBF were harshly reminded of how fragile this setup can be [12], yet little has changed in the way we fund our research.

Academic AI safety research often doesn’t translate into real-world impact

Another reason to expand AI safety into the for-profit space is the gap between academic research and real-world application. AI safety is no exception. While academic efforts generate valuable insights, some safety measures often struggle when tested in chaotic, profit-driven environments where AI is actually deployed. Anthropic’s research highlights this disconnect, showing how academic evaluations frequently fall short when it comes to real human-AI interactions [13]. Similarly, research works on AI-based vulnerability management, “are significantly impacted by the lack of extensive real-world security data and expertise” [14]. It seems clear that academic approaches alone won’t cut it. If we want AI safety to be effective, we need to apply our research to the real world while utilizing the resources and urgency that only the for-profit sector can provide.

Why for-profit AI safety is hard

The researcher mindset

One reason we’ve seen so few AI safety startups is the lack of people willing to start a company and the dominance of a “researcher mindset” in the field. Most researchers reliant on non-profit funding here are driven by a genuine desire to do good. However, few are willing to jump into the high-risk, high-reward world of for-profit ventures.

For a lot of people, especially those deeply rooted in the Effective Altruism community, the startup route might be too unconventional, and the abundance of low-impact startups makes it easy to dismiss this option without giving it any second thoughts. The startup path often involves unpredictable outcomes and the necessity to make gut decisions with little to no available data. This might be too far out of the comfort zone for your average safety researcher to try and prove that the vast resources available in the for-profit world can be utilized to drive real AI safety innovation and push research at unprecedented speeds. Naturally, the absence of success stories further reinforces the belief that starting your own startup is not a viable option. Without these examples, AI safety risks remaining an academic exercise - one that lags behind the very technologies it seeks to safeguard.

The Big Tech bottleneck

Another reason why for-profit AI safety is a bit tricky, to say the least, is the uneven playing field and dominance of a few Big Tech players. The kind of advanced AI that safety research focuses on is currently being developed by only a handful of companies, oftentimes behind closed doors. This creates a limited market for AI safety solutions, as these few companies would be the primary consumers of advanced AI safety tools. If you’re building a startup in advanced AI safety, your potential client list is small, and having an impact on the tech stack of these giants can be nearly impossible.

Here is where the opportunistic “for-profit mindset” might help: If you manage to solve real safety problems for , the opportunities are massive. And, as AI continues to advance and adoption becomes more widespread, the market for AI safety solutions is likely to expand drastically. As more industries integrate advanced AI into their operations, the demand for robust, practical AI safety measures will grow. This opens up a significant opportunity for new solutions to emerge from the next wave of research.

Staying aligned

In the for-profit world, it’s all too easy to get caught up in Silicon Valley’s existential dilemmas - like choosing the right Patagonia vest, figuring out how to fit your preordered Cybertruck into your garage, or posting about B2B SaaS. These distractions can pull focus from the real mission: conducting AI safety research that genuinely makes a difference.

The venture could fail, or worse, you might be distracted by something more glamorous but far less meaningful, diverting potential AI safety innovators from their original mission. Besides your ability to align the product with impact, considering that the R&D budget of a single successful startup could eclipse the entire non-profit AI safety space, it might be a risk worth taking.

Why for-profit AI safety can be done

Traitorous Eight: Breaking away to reshape an industry

In 1957, eight engineers, frustrated with the direction of Shockley Semiconductor, left to form Fairchild Semiconductor. Known as the Traitorous Eight, this group, which included Gordon Moore (namesake of Moore’s Law) and Robert Noyce, pioneered innovations such as the integrated circuit. Their breakaway sparked a technological revolution that gave Silicon Valley its name and produced numerous spin-offs, including Intel, driving advancements that shaped the modern tech landscape. Even though their impact was not tied to AI safety, we may be witnessing a similar moment in our current age of AI, where top talent in Big Tech, disillusioned with the AI race, could choose to focus on building a future aligned with their vision, knowing that it might take just eight determined engineers to change the world [15][16].

SpaceX: A model for aligning research and profit

SpaceX, founded by Elon Musk in 2002, has redefined what’s possible in space exploration by making it both affordable and sustainable. From pioneering reusable rockets with the Falcon 9 to launching crewed missions to the ISS, SpaceX has turned ambitious research into tangible, scalable solutions. In 2023, the company reportedly generated $8 billion in revenues while operating profitably [17], demonstrating that you can tap into the vast resources in the for-profit world to pay for groundbreaking and high-impact research. For AI safety startups, SpaceX’s approach illustrates how mission-driven innovation can lead to both significant impact and financial success [18].

Anthropic: A for-profit model that prioritizes safety

Anthropic, potentially one of the frontrunners of for-profit AI safety, shows that a company can scale while staying committed to AI safety. They’ve raised billions and attracted top talent like OpenAI refugees Jan Leike and John Schulman precisely because of their safety focus [19]. So far, Anthropic has shown that it is possible to build and scale a for-profit venture without losing sight of your mission.

However, there is a caveat with Anthropic’s model, as it doesn’t fully align with my vision of an ideal for-profit AI safety startup. Anthropic, for better or worse, has been pulled into the AI race, and like the other big boys, they’re spending millions, if not billions, on compute to stay competitive. We do not need another contender in that race, and I am convinced that there are many alternative routes an aligned AI safety startup can take, without having to compete on this dimension.

So, what does a successful AI safety startup look like?

Beyond the positive impact of founders reinvesting their wealth into non-profit research - a model that’s currently sustaining much of AI safety work - I want to advocate for a different kind of success story. One where we tackle critical problems, like implementing safety mechanisms for advanced multi-agent AI systems on the web, defending against AI-driven cyber-attacks, or ensuring your AI model consistently tells the truth. These are tough challenges that need solutions within the next years. History shows us that small teams can catalyze industry-wide change. Given the large gap in funding and real-world application, the time is ripe:

Now is the time to build an AI safety startup.