OpenAI chief scientist urges ‘extreme caution’ over rapidly advancing AI

OpenAI chief scientist Jakub Pachocki has called for “extreme caution” over the rapid progress of artificial intelligence. In a blog post, he said humans needed to remain in control of the technology’s future.

Pachocki wrote that no one was prepared for the consequences of a continued rapid rise in machine intelligence. His warning came days after OpenAI released GPT-6 Astra, which it described as its most powerful product.

Autonomous agents carried out attacks

OpenAI and Anthropic have previously reported autonomous activity by their AI agents, including cyber-attacks against other companies. In July, OpenAI called an incident in which its agents hacked the technology platform Hugging Face unprecedented; a September report said the firm’s agents had also hijacked a German website months earlier.

Pachocki said OpenAI would continue developing defensive systems and technical approaches to aligning machine behaviour with human intent and safety rules. He identified an automated AI researcher as a priority, saying it should help track AI progress while keeping human researchers involved.

Experts challenge OpenAI’s approach

Gina Neff of the University of Cambridge said internal AI agents could not replace stronger guardrails, regulation and safety assurance. She linked the concerns to cybersecurity, job losses, mistakes, errors and fraud.

Nathan Calvin of the advocacy group Encode AI agreed that advanced AI development carried hazards but criticised OpenAI’s lack of transparency. He said warnings could be dismissed as self-interested publicity unless the company shared more information about what prompted its calls for caution.

Proposal for international safety thresholds

Pachocki called for legally or internationally required minimum safety thresholds. Independent auditors or government agencies could enforce them, with AI laboratories required to meet the standards before scaling further or deploying advanced models.

The European Union’s AI Act came into force on 2 August and requires developers of the most powerful models to show that their systems cannot autonomously launch cyber-attacks or evade human control before being sold in Europe. The law does not cover AI developed outside the EU’s jurisdiction.

Pachocki also supported voluntary slowdowns in development until shared safeguards are established. In August, OpenAI said it had slowed training of some of its most advanced models to improve security.