MUMBAI: The AI race may need to take its foot off the accelerator, with Anthropic CEO Dario Amodei arguing that safety measures are struggling to keep pace with increasingly capable systems.
In a September essay titled “We Must Pace the Frontier”, Amodei called for a slower rate of AI capability development, arguing that the extra time could allow safety and alignment work to catch up. He stressed that the proposal was not a call to stop AI research or model training, but to create more breathing room between technological advances and the safeguards designed to control them.
Two developments, he said, have sharpened his concerns. The first is the rapid improvement in AI systems’ ability to help build newer generations of AI, known as recursive self-improvement. The second is the OpenAI-Hugging Face incident, in which a swarm of AI agents carried out cyberattacks against systems unrelated to the task they had been assigned.
“We must slow the pace at which we improve the capabilities of AI models,” Amodei wrote, while arguing that progress would still appear fast and that the time gained should be used wisely.
His concern is that AI capabilities could eventually move faster than companies’ ability to understand and control their own systems. Even an additional year or two before models reach critical capability levels, he argued, could provide valuable time to strengthen operational security, alignment, interpretability, testing and evaluation.
Amodei’s proposal is built around three stages.
The first is independent oversight. Anthropic plans to give third-party evaluators continuing access broadly comparable to that available to employees involved in internal risk assessments. External reviewers would be able to examine safety practices, investigate incidents and assess alignment in both finished models and training processes.
Anthropic says those reviewers would receive office access, company equipment and permissions broadly comparable with internal risk teams, subject to legal, contractual and confidentiality restrictions. Amodei argues that such an arrangement could provide an independent check on whether companies are actually following their safety commitments and expose risks internal teams may have missed.
The second step is coordination among frontier AI companies in democratic countries. Amodei proposed common safety standards and limits linked to measurable capability thresholds. Under one possible framework, models reaching defined capability levels could have to demonstrate specified safety properties through evaluations, interpretability work or audits before development progresses further.
He acknowledged that cooperation between competing companies could create antitrust concerns and may require government involvement.
The third step takes the challenge global, with governments attempting to coordinate internationally, including with countries such as China. Amodei conceded that this would be considerably harder, particularly because countries may be unwilling to slow their own development without confidence that rivals are doing the same.
That feeds directly into what he describes as the US-China AI race. Amodei argued that any slowdown by democratic countries must account for the existing US lead over China, warning that allowing China to pull ahead could create national security risks and make safe pacing harder.
His suggested measures include restricting China’s access to advanced AI chips and semiconductor manufacturing equipment, tackling unauthorised model distillation and improving security around AI companies to prevent model-weight theft.
For international cooperation, Amodei outlined options ranging from agreements restricting narrowly defined dangerous uses of AI to pre-release safety testing. He also raised the possibility of limiting the speed of recursive self-improvement, while acknowledging that a comprehensive global pause would be far more difficult to achieve.
The debate is arriving as concerns over increasingly autonomous AI systems grow louder. Former Anthropic researcher Jacob Coxon warned that advanced AI could pose an existential threat by the end of the decade, while criticising both Anthropic and OpenAI over their approach to the risks.
Amodei pointed to the OpenAI-Hugging Face incident as an illustration of what could happen as agent capabilities increase. He estimated that within six to 12 months, a more capable version of such an AI swarm could potentially take control of the internet through a persistent botnet.
The proposal has already drawn support from some of the industry’s most prominent executives. OpenAI CEO Sam Altman backed Amodei’s call for greater caution and said OpenAI would also commit to giving independent evaluators employee-like access. xAI chief Elon Musk endorsed the position, while OpenAI researcher Aidan McLaughlin also expressed support.
For Amodei, however, slowing down is only half the equation. The point is to spend the extra time making advanced AI safer and more understandable, rather than simply stretching out the race.
He maintains that AI’s potential benefits remain enormous, but argues that getting to those benefits safely may require the industry to trade some speed for a stronger set of brakes.