Anthropic Researcher Quits as Colleague Puts AI Extinction Risk Above 10%

Jacob Coxon says leading AI labs are racing toward self-improving systems without a credible way to keep superintelligence aligned.

The race to make AI more capable is increasingly colliding with the challenge of keeping it under human control.

Core Takeaway

A viral resignation has exposed a difficult contradiction inside the artificial intelligence industry: some of the people building frontier models believe the technology could eventually become catastrophically dangerous, yet commercial pressure and competition between laboratories keep the race moving.

For investors, the immediate question is not whether one researcher's extinction estimate is correct. It is whether AI capabilities are improving faster than companies and regulators can develop credible systems for controlling them.

On September 8, Jacob Coxon resigned from Anthropic after spending roughly three years conducting pretraining research at Anthropic and OpenAI. In a post on X, he accused both companies of racing toward self-improving superintelligence without acting responsibly.

"They are racing straight to self-improving superintelligence and gambling with our lives," Coxon wrote.

The warning spread unusually quickly. It reached more than 100 million people overnight, according to the Associated Press, while a later social-media count placed the post above 123 million views.

Coxon's resignation warning surpassed 100 million views within a day, according to the Associated Press.

The identity of the person issuing the warning is important. Coxon was not simply an outside activist commenting on hypothetical risks. He worked on pretraining—the process responsible for creating a model's underlying capabilities.

His departure therefore suggests that concern about the speed of AI development is spreading beyond teams specifically assigned to study safety.

Coxon told TIME that his decision did not follow one secret breakthrough or isolated incident. Instead, it reflected two broader conclusions: AI development was accelerating, and the industry did not have the situation under control.

The Warning Is Coming From Inside Anthropic

Coxon's argument is not that today's consumer chatbots could suddenly destroy humanity. The scenario he fears begins with what researchers call recursive self-improvement.

In its early stages, AI helps human researchers write code, run experiments and analyze results. More capable models then help develop their successors, which can further accelerate the next development cycle. If enough of this process becomes autonomous, years of AI research could theoretically be compressed into months.

Anthropic has publicly documented signs that AI is already accelerating portions of its own development. As of May 2026, more than 80% of the code merged into Anthropic's codebase was written by Claude, while the typical engineer was merging approximately eight times as much code per day as in 2024, according to the company's report on recursive self-improvement.

The same report, however, includes an important qualification. Current models remain materially weaker than experienced researchers at selecting goals, identifying which problems deserve attention and exercising senior-level judgment. Anthropic explicitly says fully autonomous recursive improvement has not arrived and is not inevitable.

That distinction did not prevent Evan Hubinger, Anthropic's head of alignment stress testing, from publicly supporting Coxon's warning.

Hubinger said he personally assigns a probability greater than 10% to AI causing human extinction within the next decade. He added that Anthropic is attempting to address the danger but does not yet have a plan for aligning a future superintelligence.

In a follow-up post, Hubinger clarified that his estimate concerned future superintelligence, not the relatively low risk posed by present models.

That qualification is essential. The figure is not an official Anthropic forecast, a scientifically measured probability or evidence that Claude currently presents an extinction threat. It represents one researcher's judgment under extreme uncertainty.

Still, concern about catastrophic outcomes is not limited to a tiny group of AI critics. A survey of 2,778 AI researchers found widespread concern about extremely negative outcomes, although participants disagreed sharply about the likely timeline, magnitude of the risk and whether slower AI development would produce a safer outcome.

The disagreement is partly about whether AI can overcome the constraints that continue to limit technological progress.

Models can already automate growing portions of software development, but AI research still depends on scarce computing power, physical infrastructure, reliable data and human judgment. Skeptics argue that these bottlenecks will prevent the sudden "intelligence explosion" feared by Coxon.

The industry therefore faces two very different possibilities. AI-assisted research could advance gradually while remaining constrained by chips, energy and data, or it could begin removing those constraints and create a much faster feedback loop. No existing benchmark can reliably determine which trajectory will prevail.

The Real Fight Is Over Who Controls the Race

The controversy is ultimately less about one percentage than about governance.

Coxon argues that decisions carrying consequences for the entire world should not be made through internal conversations at private companies. His concern is that every leading laboratory believes it must continue accelerating because it cannot trust competitors to behave responsibly.

Under that logic, even a company that recognizes the danger may decide that slowing down is more dangerous than continuing. The result is a race in which no participant believes it can safely leave.

Elon Musk has also repeatedly acknowledged that highly advanced AI could produce catastrophic outcomes. However, he disagrees with the idea that safety should be delegated primarily to a separate corporate department.

Pointing to Tesla and SpaceX, Musk argues that safety must be embedded in the work of every engineer and manager. In his view, an isolated safety team can become a symbolic department without enough authority to change how products are actually developed.

Musk argued on X that safety should be an organization-wide engineering responsibility rather than the job of a separate department.

The two approaches are not necessarily incompatible, but they emphasize different failure modes.

Coxon fears a competitive race in which every company accepts potentially unacceptable risks because it distrusts its rivals. Musk fears safety departments that lack engineering authority and allow the rest of an organization to treat safety as someone else's responsibility.

An effective governance system would need to address both problems. Safety would have to be integrated throughout model development, while independent decision-makers would still require the authority to delay or block the release of a model that crosses specified risk thresholds.

For markets, Coxon's resignation does not immediately change AI demand, model revenue or the need for data-center infrastructure. Its importance lies in the longer-term risk premium surrounding the industry.

More aggressive regulation could slow model launches, require expensive external evaluations and increase legal or insurance costs. A serious safety failure could weaken enterprise adoption, while visible disagreement inside frontier laboratories could complicate the enormous valuations and potential public listings being considered by AI companies.

The bullish AI investment thesis assumes that increasingly capable models will generate enough productivity and economic value to justify unprecedented spending on chips, data centers and energy.

Coxon's warning identifies the other side of that bet: the faster AI improves, the greater the burden on developers to demonstrate that capability growth is not outrunning their ability to control it.

Investors do not need to accept a precise probability of human extinction to recognize the financial implication. AI safety is moving beyond an abstract ethical debate and becoming a material question of regulation, corporate governance and valuation.