I should probably do a post on all this. Someday.
From Yahoo Finance, November 10:
Anthropic says a key AI safety alarm has been triggered, days before its IPO
A week ago, Anthropic told would-be investors that AI speeding up its own research might "imminently" trigger one of the company's key safety alarms. On Tuesday, in the same filing that set a price range for its record-setting IPO, it said that has now happened.
"We made a determination that we have met this threshold," the company wrote in its amended prospectus. It based the call on "what we expect from future models and our overall rate of progress, rather than on what we have observed from current models."
The safety threshold sits in Anthropic's Responsible Scaling Policy — the company rulebook for adding tougher safety measures as its AI gets more powerful. This one is triggered when AI is doing enough of Anthropic's research to roughly double how fast its models improve. The policy says that pace would pack two years' worth of AI progress, at its 2018-2024 rate, into a single year. The filing says AI "has already been accelerating our research and development." Crossing the line means more safeguards, not a shutdown.
The fear is AI will improve itself faster than people can check it. In 2023, Anthropic CEO Dario Amodei joined other AI leaders in warning that the "risk of extinction from AI" should be a global priority alongside pandemics and nuclear war.
Anthropic says humans stay in charge. It doesn't expect to safely use Claude "to fully autonomously design and build successive generations of frontier models without meaningful human supervision or understanding," the filing says, and it is weighing policy changes "to keep humans in control."
An outside group sees the same speedup. METR, an independent nonprofit that tests AI systems, estimated last week that Anthropic's models have been improving nearly twice as fast since October 2025.
In a separate review, METR called Anthropic's August argument that it was still below the threshold "overstated" and said it was "plausible" the line was crossed by mid-July.
A $1.4 trillion debut
Anthropic set its price range at $70 to $78 per share on Tuesday as it began its roadshow, the round of investor meetings before pricing. That values it at as much as about $1.4 trillion, counting every share that could exist. Trading could start as soon as Nov. 19, according to reports....
....MUCH MORE
For what it's worth METR is hardly "an independent nonprofit" and is in fact hand-in-glove with Anthropic et al.
As for regulation, the current system, from police to penitentiaries,can serve until we come up with something harsher.
