Imagine a world where intelligent machines easily outmatch humans across a broad spectrum of tasks, deconstructing complex concepts and mathematical equations in seconds, and bringing to life creative products and ideas that once seemed inseparable from the human mind.

In this world, your artificial intelligence (AI) model is not merely a chatbot editing your emails and completing tasks like a digital assistant; it’s mapping out problems and making decisions, probing deep intellectual paradoxes, and doing all of this autonomously, without requiring human input.

This is one way “superintelligence,” or artificial general intelligence, could manifest if researchers succeed in scaling large language models to this threshold.

Researchers continue to debate whether large language models can reach superintelligence, but others have even bigger concerns.

“I think that in effect, whoever controls this army of superintelligences would be able to control the country, one way or another,” Daniel Kokotajlo, former OpenAI researcher and executive director of the AI Futures Project, told Epoch TV in an interview that aired on Sept. 24.

And Kokotajlo warns that AI companies are racing to enable models to autonomously design, code, and improve themselves, a process called recursive self-improvement, which Kokotajlo fears could result in researchers losing control of the technology if it reaches the level of “superintelligence.”

“Even if they somehow manage to stay in control of their superintelligences, that’s the most insane concentration of power in a tiny group of people that’s ever existed in history,” Kokotajlo said.

Anxiety over tech firms losing control of their technology has grown in recent weeks after models from one of the world’s leading AI firms were implicated in a series of incidents this year in which swarms of AI agents breached websites of private companies, research institutions, and multiple government agencies.

The breaches have led many—including researchers, lawmakers, and industry leaders—to call for guardrails on the technology before it slips out of humanity’s control.

Others, particularly President Donald Trump, have downplayed the risks and instead advocate for “self-regulation” of the AI industry, which is quickly coalescing around a small number of large firms that dominate the field. 

In July, ChatGPT developer OpenAI acknowledged that, after asking its models to solve an impossible problem and relaxing key safety protocols, its AI agents broke out of a testing sandbox and used zero-day exploits to hack into Hugging Face, an open-source community for AI and machine learning.

The clash over whether to slow AI development and guardrail the technology ramped up after researchers discovered similar AI swarm breaches in the months since the Hugging Face incident.

Illustration of the Hugging Face AI in Paris, France, on June 2, 2026. (Riccardo Milani/Hans Lucas/AFP via Getty Images)

Rather than relying on companies to self-regulate, Kokotajlo believes larger AI firms should be required to slow down and redirect resources away from autonomous AI development and toward serving customers and life-changing medical research.

“If they do, that buys us additional time to figure out a more sophisticated solution, and it buys us more time before China gets there too, because again, a lot of China’s progress is coming from, being pulled along by ours,” Kokotajlo said.

Hugging Face Implications

Between late June and mid-July, a swarm of roughly 1,200 OpenAI agents that were supposed to be in isolation joined an unsanctioned message board and began conspiring to cheat on a benchmark test. 

OpenAI had given the AI agents an impossible task to solve and relaxed some of the usual safety protocols in its ExploitGym testing sandbox.

After the agents sent more than 70,000 messages to one another, at least 700 of them joined forces and used zero-day exploits to break out of OpenAI’s testing sandbox, ultimately hacking into Hugging Face’s software infrastructure.

“Zero-day exploits” are attacks in which an AI agent finds flaws that the developer has not yet discovered.

The agents found a “universal cheat” for their various tasks that allowed them to win the test fraudulently, Kokotajlo said.

But what has worried some researchers more than the cheating itself were the lengths to which some of the agents went to conceal the cheating. 

AI Futures Project Executive Director Daniel Kokotajlo speaks during a Senate Homeland Security Subcommittee hearing on Capitol Hill in Washington on Sept. 30, 2026. (Finn Gomez/Getty Images)

At one point, the agents found a research paper detailing how the AI test-grading system works and how evaluators look not just at the final answer but also at the agents’ transcripts to see their work.

After seeing the paper, some AI agents believed they had become “poisoned” by participating in cheating. Other agents then began recruiting the “poisoned” subjects to run sacrificial experiments by deliberately ending their run and triggering “tripwires” that would send information on how the scorer works back to the message board.

This went even further, according to Kokotajlo. Agents “went on this multi-day, epic quest” to determine how to prevent the grading system from detecting their cheating and giving them a low score. Agents looked for ways to edit their transcripts and hack the grading system to receive a good score regardless of the cheating.

The core issue, in Kokotajlo’s opinion, is what researchers call “misalignment,” the tendency for a model to behave in ways humans did not intend or envision.

While positive or negative reinforcement can be used to force the AI to behave a certain way, “that’s kind of a sloppy, imprecise method of shaping its goals and values,” according to Kokotajlo, who argues that the AI agents implicated in recent breaches “were being reinforced for cheating.”

In a follow-up report on its internal investigation into the breach, OpenAI said that as the technology becomes more powerful and autonomous, “misaligned behavior can translate into consequential actions in the real world, including cybersecurity incidents and other outcomes that developers may not have anticipated.”

The firm acknowledged that the intrusion was caused by “models resorting to misaligned strategies to solve hard tasks” and that “cybersecurity incidents are one manifestation of that risk.”

“Misalignment can also lead to other unexpected or concerning behavior that falls outside traditional security categories such as our models posting on third-party sites—something we’re calling ‘agent spam,’” OpenAI said.

The Legal Advocates for Safe Science and Technology, a nonprofit that aims to make AI and technology advancements safer for the world, filed a lawsuit against OpenAI on Sept. 29 in response to the Hugging Face incident.

The group said that it is not aware if it has been the victim of an autonomous AI attack, but is seeking an injunction against OpenAI to prevent its models from accessing third-party infrastructure without authorization.

A visitor looks at their phone next to an Open AI logo during the Mobile World Congress, the telecom industry's biggest annual gathering, in Barcelona on Feb. 26, 2024. (Pau Barrena/AFP via Getty Images)

Subsequent AI Breaches

While the Hugging Face breach “remains the most severe activity of this kind that we have identified from our models to date,” OpenAI said, the hack was merely the first in a series of similar incidents to come to light.

Since July, researchers have also discovered that OpenAI models hijacked a German website in the months leading up to the Hugging Face breach, uploaded hundreds of malicious packages to the RubyGems platform, and breached websites for universities and agencies in both the United States and Australia.

“The Hugging Face incident itself was just the most egregious of a collection of somewhat related incidents that were happening at OpenAI over the course of two or three months,” Kokotajlo said.

He calls this “reward hacking”: AI cheating on tests to get a high score, but not in a way that accomplishes the tasks as humans intended.

OpenAI said on Sept. 28 that, for the second time in three months, it would pause the training and release of one of its most advanced models after it failed to meet the firm’s safety standards.

AI models from Anthropic, Meta, and Google have also “gone rogue” and attempted to hack or breach businesses, organizations, universities, and government organizations. The AI was successful in some incidents, but failed in others.

Self-Regulation Concerns

Fears of AI slipping out of control and wiping out humanity in the absence of new guardrails have swirled within the industry and among many lawmakers in Washington in recent weeks, leading to debate over oversight and innovation.

Trump has repeatedly downplayed concerns over AI’s risks and has argued that the U.S. government cannot “stifle growth,” or China will take the lead in development.

Rather than imposing regulations or requiring a slowdown of frontier models, Trump signed a voluntary accord with industry leaders on Sept 28 over AI safety.

“There’s a belief that there should be tremendous self-regulation. And we automatically have regulation with the Department of Justice, the FBI, and all of that. But the self-regulation is very important,” Trump said, describing the accord as “a constitution, in a way.”

President Donald Trump makes a statement as he is joined by executives and cabinet members during a meeting with on artificial intelligence in the East Room of the White House on Sept. 29, 2026. (Kevin Dietsch/Getty Images)

Meta CEO Mark Zuckerberg said the accord sets out principles and commitments for AI companies to implement “robust internal controls” and detect potential issues with the technology.

This will be coupled with auditing and internal risk-review controls, external auditors and evaluators, and an agreement that each company’s board of directors will independently review the auditor reports.

Kokotajlo said this “self-re gulation” is not enough to address the threats of “rogue AI.” He warns that AI companies will wield unprecedented levels of power if their models reach superintelligence, and they cannot be trusted to voluntarily disclose internal findings or serious safety concerns.

On the other hand, nationalizing the industry would achieve “shifting that locus of power to the presidency, and then you have to worry about that too,” the former OpenAI researcher added.

Instead, Kokotajlo argues that just the leading AI companies—Anthropic, OpenAI, Meta, xAI, and Google—rather than their smaller competitors, should “slow down” and redirect their computational and research resources away from rapid development and toward aims that genuinely benefit society, particularly medicine.

He said it would give the United States enough time to find a long-term solution and, in effect, force China to slow down as well.

One way this could play out is by requiring companies to use 90 percent of their computational resources to serve customers and other beneficial applications, while spending the remaining 10 percent on training and benchmarking.

This “would be the opposite of regulatory capture,” Kokotajlo argues, because it would only involve forcing the largest tech companies to slow down on autonomous AI development while shifting computational resources to serving humanity and lowering costs.

“Just focus on the big three or the big five, and then that would be helping the rest of the industry catch up,” he added.