The Doom Business: How Anthropic Turned Extinction Into a Sales Pitch
Anthropic built the most valuable brand in artificial intelligence on a single proposition: that this technology could kill us, and that it alone could be trusted to say so. The proposition was true as marketing long before it was tested as policy.
In July 2023 Dario Amodei sat in front of a United States Senate subcommittee and told it that artificial intelligence posed “extraordinarily grave threats” to national security within two to three years. His written testimony described a near future in which AI systems could fill in the missing steps of a biological weapons programme for people who had no expertise to do it themselves. He was not a critic observing an industry. He was the chief executive of one of the industry’s five most important companies, and he was asking Congress to make his product harder to ship.
Three years later the same company removed the commitment that had made those warnings credible, negotiated with the Pentagon over the use of its models, and published research that declined to support the employment forecast its own chief executive had been making on stage. This is not a story about a company that lied. It is a story about a company that built an asset and then managed the asset, which is a different and more interesting thing.
What the fear bought
Anthropic was founded in 2021 by people who left OpenAI because they disagreed with how it weighed safety against competitive pressure. That origin story is real, and it is worth remembering, because it is the only part of this that cuts in the company’s favour. What followed was also real: a genuinely novel governance framework, published before anyone asked for one, and a research programme in interpretability that produces few headlines and is the only route to independently verifying any behavioural claim a lab makes about its own model.
What the fear also bought was differentiation. In a market where OpenAI owned the consumer imagination, a rival could not out-shout it on capability in 2023. It could, however, own the register of seriousness. And seriousness turns out to be worth a great deal of money in two specific places.
- Procurement. A buyer in a regulated industry who must justify a vendor choice to a board will almost always prefer the vendor who can describe its safety process in detail. That preference is not about safety. It is about being able to survive the meeting.
- Talent. Researchers who want to work on the alignment problem rather than a release cycle have few employers, and a lab that funds the slow work attracts a particular kind of scientist — often at a lower wage than the market would otherwise set.
- Policy. The Responsible Scaling Policy became the template. Eleven other companies adopted comparable frameworks, and the framework fed into California’s SB-53, New York’s RAISE Act and the European Union’s AI Act.
That third item is where the critique bites, and it deserves to be stated at its strongest rather than at its loudest.
The capture charge, in its best form
In October 2025 David Sacks, then serving as the White House AI and crypto czar, described frontier-lab safety messaging as “a sophisticated regulatory capture strategy based on fear-mongering.” The phrase is loaded, and the mechanism it points at is ordinary.
Here is the mechanism. If you are the company that already publishes its testing methodologies, then legislation requiring every frontier developer to publish its testing methodologies costs you approximately nothing. It costs a competitor that does not do this a great deal. A rule written in the language of your existing practice is a rule you have already passed.
Amodei’s own preferred instrument fits that description precisely. In June 2025 he published an op-ed opposing a proposed ten-year moratorium on state-level AI regulation as “far too blunt an instrument.” What he proposed instead was narrower: require frontier developers to disclose their testing methodologies and risk mitigation strategies. The Senate rejected the moratorium 99–1. The disclosure requirement he favoured would have codified what Anthropic, OpenAI and Google DeepMind were already doing voluntarily.
I think there’s a 25% chance that things go really, really badly.— Dario Amodei, at the Axios AI+ DC Summit. He added that there is a
75% chance that things go really, really well, and of the p(doom) framing itself:
I really hate that term.
This is not proof of bad faith. A disclosure regime may well be the right rule, and it is entirely possible to want it because it is right. But the coincidence between “what Anthropic already does” and “what Anthropic proposes to make mandatory” is the kind of coincidence that a critic is entitled to notice and a company is obliged to answer.
The numbers travel further than the hedges
Amodei has been careful in ways the record does not usually preserve. His 2025 figure is widely rendered as a “25% chance of civilisation collapse.” He did not say civilisation collapse. He said things go “really, really badly,” and he has publicly rejected the p(doom) framing that compresses the question into a single number. His jobs warning is widely rendered as AI “eliminating” half of entry-level white-collar jobs. The verb reported was “displace,” scoped to entry-level white-collar roles on a one-to-five-year horizon, paired with a 10–20% unemployment figure that most retellings drop.
None of that rescues the substance, and it is worth being precise about why. A hedge that is structurally guaranteed to be stripped in transmission is not a hedge. The number reaches millions of people; the qualification reaches the people who read to the end of the article. A chief executive who dislikes the p(doom) frame and continues to supply numbers for it has made a choice about which version of his view will circulate.
When the commitment became expensive
The Responsible Scaling Policy published in September 2023 contained a categorical promise: Anthropic would not train a system unless it could show beforehand that its safeguards were adequate, and it would pause if capabilities outran them. That promise was the reason the 2023 testimony carried weight. It was also the first thing to go.
On 24 February 2026 Anthropic published version 3.0. The categorical pause was removed and replaced with a dual condition: development is delayed only if Anthropic is leading the race and catastrophic risk is material. Both must be true at once. Two weeks earlier, Mrinank Sharma, who ran the company’s safeguards research, resigned, writing that the organisation “constantly face[s] pressures to set aside what matters most.” Co-founder Jared Kaplan later told TIME that the idea of a clean line between dangerous and safe had been “naive,” and that unilateral strict commitments were not realistic while competitors went “full speed ahead.”
In March 2026 the company’s own economic research team published a labour-market paper that explicitly declined to forecast, noted that observed AI usage sits far below the theoretical ceiling, and reported that unemployment in AI-exposed occupations has not systematically risen. The chief executive had been forecasting mass displacement on a one-to-five-year horizon since May 2025. The company’s data department did not follow him.
Then came the Pentagon. Anthropic refused to remove contractual bans on mass domestic surveillance and fully autonomous weapons, was designated a “supply chain risk” by Defense Secretary Pete Hegseth on 3 March 2026, and was barred from Pentagon use. On 26 March a federal judge issued a temporary injunction, agreeing the designation appeared to be “classic First Amendment retaliation.”
The case for the defence, stated properly
Three of these deserve to be taken seriously rather than waved away.
First, the collective action problem is not a fig leaf. If Anthropic slows to implement mitigations that rivals skip, the rivals take the lead, and a less responsible actor sets the pace. The company’s stated conclusion — that it would lose its ability to do safety research at all — is a real argument, not a dodge.
Second, version 3.0 added things, not only removed them. Risk Reports go beyond the existing system cards by giving an overall risk assessment. The Frontier Safety Roadmap publishes non-binding safety goals across security, alignment, safeguards and policy, and commits Anthropic to grading its own progress against them. The Governance Institute’s assessment was initially negative and became more positive on closer reading, concluding that it is “better to be honest about constraints than to keep commitments that won’t be followed in practice.”
Third, and most to the point: under genuine financial pressure over a $200 million contract, Anthropic held two red lines rather than quietly lifting them. In September 2026 Amodei called for the industry to slow the pace of development and said Anthropic would allow independent evaluators permanent access to its models. A company running a fear-marketing playbook does not volunteer permanent outside inspection.
Why the church line matters
The argument of this publication is not that the risks are invented. It is that existential-risk discourse has a structure, and the structure concentrates authority in exactly the wrong place.
It has a creation story and a fall. It has a priesthood — the safety researchers, who are the only people qualified to interpret the signs. It has indulgences, which are the voluntary commitments a company publishes and then grades itself against. And it has an eschatology, which is the number. The laity are asked to accept the assessment on the authority of the person who gave it, and the person who gave it is the person who sells the remedy.
That sentence is not a denial that the technology carries real risk. It is a refusal of a particular arrangement of authority: one in which the same institution defines the danger, sells the safeguard, grades its own homework, and asks the public to treat the result as doctrine. The alternative is not cynicism. It is verification — independent access, published evaluation, procurement that reads the paperwork rather than admiring it, and a press that checks whether the numbers say what the headlines say they say.
Amodei has said, correctly, that the danger and the solution to the danger are often coupled, and that being at the frontier is what allows a lab to see ahead and warn. Fine. Then the warning is a public good and the frontier position is a private one, and the public is entitled to insist on a mechanism that separates them. Anthropic has begun to offer one. Whether it survives the next funding round is the only question that still matters.
Sources for every claim in this article are dated and listed on the receipts page. Image credit and licence details are on the credits page. This article is editorial content; it carries no sponsored material.