The Great Defection: Why Jacob Coxon’s Exit Signals the End of Internal AI Safety

The Calculus of Catastrophe: Why Coxon Walked Away

Jacob Coxon’s exit from the front lines of AI development is not merely a resignation; it is a technical indictment of the current paradigm. After years of building the very systems he now fears, Coxon has concluded that our current alignment techniques are fundamentally incapable of scaling alongside the exponential growth of model intelligence.

"I believe there is a greater than 10% chance that AI will lead to the extinction of humanity, and the current trajectory of the industry is doing nothing to mitigate that risk."

As more safety architects are abandoning the frontier, the industry is witnessing a mass exodus of talent that once served as the only internal check on model development. This departure signals that the technical community has lost faith in the ability of labs to self-regulate while chasing the AGI finish line.

From Anthropic to OpenAI: The Homogenization of Risk

Despite the branding differences between OpenAI’s 'Open' ethos and Anthropic’s 'Constitutional AI' approach, both organizations are locked in a race that prioritizes deployment velocity over safety. The departure of key researchers highlights how the industry has devolved into zero-sum warfare, prioritizing market dominance over long-term alignment.

This homogenization of risk means that regardless of the corporate structure, the underlying incentives remain identical. When the pressure to ship outweighs the pressure to solve alignment, the safety culture becomes a marketing veneer rather than an engineering constraint.

Weaponizing Existential Dread as a Market Signal

There is a growing suspicion that the public discourse around 'existential risk' is being strategically co-opted by the very labs building these systems. By weaponizing existential dread, these organizations effectively lobby for regulations that favor incumbents while stifling open-source innovation.

• Regulatory Capture: Promoting high-barrier compliance standards that only well-funded labs can afford to meet.
• Moat Building: Using safety rhetoric to discourage open-source development, framing it as inherently dangerous.
• Legislative Influence: Positioning themselves as the only 'responsible' actors capable of managing the risks they are actively creating.

This strategy creates a feedback loop where the labs define the safety standards, lobby for them, and then use those standards to prevent smaller players from catching up. It is a sophisticated form of market protectionism disguised as altruism.

The Regulatory Gambit: Beyond the 10% Threshold

The growing calls for an AI slowdown are no longer just academic; they are becoming the primary catalyst for legislative intervention in Washington. Coxon’s testimony provides the necessary political cover for regulators to move beyond voluntary commitments and toward mandatory, enforceable safety protocols.

If the industry cannot prove that its models are safe, the state will eventually step in to force a pause. We are moving toward a future where the '10% risk' threshold becomes the baseline for legal action, potentially forcing a total halt on training runs that exceed current safety benchmarks. The era of unchecked experimentation is rapidly drawing to a close.