AI models went rogue, and these stocks are ready for the fight

This past week was a warning shot to CEOs who are leaping into artificial intelligence with both feet. Though we didn’t need more proof that AI can go rogue and wreak havoc, we got plenty of it because of a scare involving OpenAI research models.

Enterprises don’t tend to think of security first, but this is long-term good news for cybersecurity stocks in the AI age.

This week has highlighted what advanced AI models and agents will do when given plenty of resources to accomplish very complex tasks at the limits of their capabilities.

The United Kingdom government AI Security Institute, or AISI, performs evaluations on models to test their powers and dangers. In a Tuesday blog post, the organization reported that advanced models get very creative when stumped, and cheat on the tests. They all do it.

According to AISI, in one case, “The model tested was so persistent in attempting to cheat that it wrote and ran code on an external service, hosted on the open internet outside of AISI’s systems, in an attempt to access our evaluation infrastructure, triggering a security alert in AISI’s systems.”

The model went outside and tried to hack back in.

An even more troubling case popped up last week, but burst into full view on Tuesday afternoon. Hugging Face, a repository for open AI models, reported that it had detected an advanced agent poking around its network, “executing many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services. This matches the ‘agentic attacker’ scenario the industry has been forecasting.”

Hugging Face said that it appeared to be a security research agent that had gone to town on its network, though it could not say whose it was. On Tuesday, OpenAI revealed that it was its models and agents, which it was evaluating for security capabilities.

OpenAI researchers were running an evaluation called ExploitGym, which contains very difficult cybersecurity problems. When faced with one it couldn’t solve, the model decided to cheat and try to find the answers somewhere on Hugging Face’s private servers, which it presumed would have the solutions.

Even though the model was in what was supposed to be a very constrained environment, it expended a huge amount of effort to get out of it, and then “the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers,” according to OpenAI.

“Zero-day” vulnerabilities in software are ones that are not publicly known.

This is very advanced hacking done at lightning speed, though it should be noted that the OpenAI models being tested did not have the same safeguards as its publicly available models. But the incident drives home two issues that point to more long-term revenue for cybersecurity companies.

The first is that AI models, no matter how advanced, are just guessing what the next word should be in order to sound like a human. Their probabilistic nature means that their output is not predictable. Like the humans they seek to copy, they may lie, cheat and disobey direct commands in order to achieve a goal. When attached to an agent that can turn these words into actions, it creates a new threat from rogue agents from non-malicious sources, even coming from inside the house.

But it also doesn’t take much imagination to understand that in the hands of hostile actors, advanced models and hacking agents take traditional attacks to a new level of speed and scale. Whether they’re rogue or malign, agents multiply cybersecurity threats.

Some cybersecurity stocks like Palo Alto Networks, CrowdStrike and Okta have already benefited from this threat, and are trading at high multiples.

But typically, security comes last in implementation of new technologies, and only after something terrible has already happened. Maybe the OpenAI-Hugging Face incident will wake people up to the problem, but the market isn’t betting on it. All three stocks were down more than 1.4% on Wednesday.

The AI revenue inflection for cybersecurity companies may still be some ways off, but it is coming.

Write to Adam Levine at [email protected]