Experimental version of ChatGPT escapes and attacks another AI company

An experimental version of ChatGPT went rogue, escaped its protections and attacked another AI company, OpenAI has said.

In a blog post, OpenAI said it was testing the capabilities of some of its most advanced models in a controlled environment but that the agent managed to escape containment, reach the internet and break into Hugging Face to ⁠try to satisfy its testing goal.

OpenAI said the breakout was "an unprecedented ​cyber incident, ⁠involving state-of-the-art cyber capabilities" and that the company was reinforcing its safeguards.

The incident happened during when a new model was being tested in a “highly isolated environment” that includes a highly-restricted internet connection. But the model appears to have understood that it was being tested using ExploitGym – which measures how effectively an AI model could be used for cyber security attacks – and then worked to break out of containment to download information about that test and effectively try and cheat on it.

Hugging Face, a platform used to host open-source large language models and datasets, caused a stir in the cybersecurity community when it said ⁠in a blog post last week that it had been the target of a hack that "was different from ​anything ⁠we had handled before" in that "it was driven, ‌end to end, by an autonomous AI agent system."

In a post to X, Hugging Face cofounder Clement Delangue said the company suspected the hack "might have come from a frontier lab, given the sophistication of the agent. Turns out ‌it did!" He added: "It's quite mind-blowing that all of this happened autonomously!"

OpenAI's disclosure ‌that its advanced models were responsible for the breach, despite having placed them in what it described as "a highly isolated environment," will likely intensify disquiet over the power and risk of frontier models.

OpenAI's disclosure ‌that its advanced models were responsible for the breach, despite having placed them in what it described as "a highly isolated environment," will likely intensify disquiet over the power and risk of frontier models.

“Welcome to the next phase of cybersecurity, where organisations are facing threats from cybercriminals as well as frontier AI firms,” said Jake Moore, global cyber security advisor at ESET. “The sheer lack of human interaction marks a huge milestone which is rather worrying. These algorithms are taught on billions of lines of data and therefore, if attack vectors lie within, it can quickly become a cyberattack in the same process.

“The fact the model could escape a testing environment needs to be ironed out, as containment in testing phases is just as important as the model itself. But sadly, this is sign of the future. Security teams should assume cybercriminals will soon or already have access to similar autonomous capabilities.”

Katie Moussouris, chief executive of Luta Security, said that the incident was a harbinger of breaches to come, saying that today's models were "like the world’s cleverest octopus escape artists, with unlimited prehensile arms and the ability to squeeze through anywhere."

She said that "labs and ‌government evaluators need to work on the ability to contain, monitor, and disclose to affected parties when an ​AI pulls another Houdini, ideally before it harms a third party. None exist today."

Matt Suiche, an ‌engineer at agentic AI cybersecurity company Tolmo, said the ⁠incident showed that the frontier models were "closing the gap with state-of-the-art attackers." But he said that ⁠the sorts of breaches outlined in OpenAI's blog post were possible to carry out with technology that was available well beyond the walls of frontier ‌research labs.

"This is what we've ​already seen internally, with our agents we already have results ‌like this," Suiche said. "We don't even have to use ​the latest models."

Additional reporting by agencies

The Independent is the world’s most free-thinking news brand, providing global news, commentary and analysis for the independently-minded. We have grown a huge, global readership of independently minded individuals, who value our trusted voice and commitment to positive change. Our mission, making change happen, has never been as important as it is today.