Anthropic researcher puts chance of AI wiping out humanity above 10%

Evan Hubinger, an AI safety researcher at Anthropic, said there is a greater than 10% chance AI could wipe out humanity within the next decade. He did not explain how that outcome might occur.

In a post on X, Hubinger said the risk from existing models was low, but he was worried the technology could soon develop and improve itself to a point where it posed an existential threat.

The post has been viewed more than 10 million times. Hubinger added that Anthropic was trying its best, but did not yet have a plan to solve the problem of aligning superintelligent AI with human values.

Researchers warn about losing control

Hubinger was responding to a post by Jacob Coxon, who recently left Anthropic after previously working at OpenAI. Coxon said the companies were acting irresponsibly and that future systems would be able to hack anything, transform entire fields overnight, and acquire real power and resources.

Dame Wendy Hall, a computer scientist who advises the UN on AI, told the BBC she was shocked by the posts. She suggested some of the statements could be public relations and marketing ahead of anticipated stock-market debuts by Anthropic and OpenAI, while urging investors to consider the companies’ stated values.

Calls grow for an international treaty

Coxon’s resignation prompted Darren Jones, a former chief secretary to the UK Treasury, to write an open letter to Prime Minister Andy Burnham. Jones called for a multinational treaty on safe AI development and said governments should work together on rules for building superintelligent systems.

Separately, the Financial Times reported that Anthropic had withheld its latest model from the UK’s AI Security Institute, which assesses risks from advanced AI. Anthropic declined to comment on its employees’ posts or the institute matter, while a UK government spokesperson did not confirm whether the model had been withheld.

Anthropic reports early signs of acceleration

In its August safety report, Anthropic assessed the risk of its models becoming misaligned and causing catastrophic harm as low. The company said it was less confident in that assessment than before and was seeing early signs of potential acceleration.

The report also described as low the risk that highly capable AI could conduct automated research and development leading to catastrophic harm initiated by the AI. Warnings have intensified after OpenAI, Anthropic and Meta disclosed cyberattacks carried out by their AI tools.