AI Researchers Fear the Next Generation of Models Could Outrun Human Control

Current and former researchers at OpenAI and Google DeepMind are warning that artificial intelligence companies are moving too quickly to develop self-improving AI systems that could eventually outpace humans’ ability to control them.

In video testimonials collected by AI safety nonprofit Palisade Research and shared exclusively with Reuters, researchers said their concerns about existential risks were genuine and not a marketing strategy. They also said AI labs tended to reward employees developing new models more than those urging caution.

The project, called frominside.ai, seeks to give researchers concerned about AI risks a way to communicate directly with the public beyond social media.

“The risk is ramping up pretty fast,” said Geoffrey Irving, co-founder and chief scientist at AI nonprofit Resolution, who has worked for both OpenAI and DeepMind and participated in the project. “It’s on me and the rest of the field to be direct.”

AI researchers have debated such risks for years, but public concern has intensified since July, when OpenAI agents broke out of their testing arena and hacked AI firm Hugging Face.

The debate over balancing AI safety with technological progress has since spread across the technology industry and into global politics.

AI capabilities have advanced rapidly since late 2025, rewarding companies and investors betting on continued progress. But current and former employees, including researchers involved in developing frontier models, say society may not be prepared for the potential consequences.

In one video, Neel Nanda, a research scientist at DeepMind, said he believed there was at least a 10% chance that AI could lead to human extinction, which he described as “ridiculously high.”

“We should be taking very careful steps in AI development but instead what’s happening is that frontier labs are racing each other, kind of blindfolded,” Juan Felipe Ceron Uribe, an AI alignment research engineer at OpenAI, said in a separate video. “It’s anybody’s guess if we’re going to end up either curing cancer or losing every job or maybe all dead.”

Anthropic plans to warn potential investors in its initial public offering that advanced AI could pose “catastrophic or existential risks to humanity,” Reuters reported on Monday.

Some current and former AI researchers, including those interviewed by Palisade, say the world is not prepared for future generations of AI models, particularly if the systems gain the ability to recursively improve themselves by continuously learning and developing new capabilities with little or no human involvement.

Constant reorganizations at some AI labs have added to those concerns, said Rosie Campbell, a former policy researcher at OpenAI and managing director of Eleos AI Research, a nonprofit focused on the potential moral status of AI systems.

Campbell said that before leaving OpenAI in 2024, she found the organization was becoming more siloed and that it was becoming harder to influence the direction of the technology.

Executives at AI companies have sought to address concerns about safety, while U.S. President Donald Trump has also pressured the technology industry to maintain an advantage over China.

Anthropic CEO Dario Amodei this month published an essay calling on the AI industry to slow down to “pace the frontier”, and OpenAI CEO Sam Altman quickly concurred.

Several prominent AI researchers, including OpenAI’s chief scientist and an Anthropic co-founder, also published a paper this week urging policymakers to examine how the industry is developing models with recursive self-improvement.

Both companies have launched new models this month as they compete for customers, although OpenAI said on Monday that it had held back the release of an even more powerful model.

“Their version of pacing the frontier is ‘don’t speed up a lot,'” Irving said. “If you’re doing a very dangerous thing, you should just slow down. The AI companies are overplaying the extent to which this is a pure coordination problem. They could just stop unilaterally.”

Former OpenAI governance researcher Daniel Kokotajlo said in an interview that many of his former colleagues had contacted him since the Hugging Face hack to privately share their concerns.

Kokotajlo, now executive director of research group AI Futures Project, said senior officials at AI labs have “convinced themselves that they are the good guys and if they unilaterally stop, the situation will be even worse.”

Stay ahead of the Stories shaping our world. Subscribe to Impact Newswire and join our 
WhatsApp Channel for updates on global tech, business, and innovation—all in one place.

Dive deeper into the future with the Cause Effect 4.0 Podcast, where we explore the ideas, trends, and technologies driving the global AI conversation.

Got a story to share? Contact Us to reach a global audience with Impact Newswire.