As everyone in cybersecurity loses their minds over the OpenAI/Hugging Face incident, it’s important to take a step back and put things in perspective. There’s no doubt that cybersecurity use cases are an area where AI is making some genuine advancements. This is especially true in offensive security scenarios. But the world is an incredibly complex place that often confounds our expectations about what these advancements actually mean.

What I mean is that, at first glance, this would seem to prove people like Eliezer Yukowsky were right, as well as the people who say cybersecurity is dead or that script kiddies have nukes now. But this isn’t the case, just as it wasn’t the case with Mythos. This is more hysterics than reality, especially when there is still much we don't know about the incident.

I mean, OpenAI’s write-up reads more like a marketing document promoting a feature than an incident summary, while the quote from HuggingFace sounds more like someone accepting an award than someone who just got hacked. It’s a bit surreal.

I’m not claiming this was a stunt. I’m pointing out that the lack of detail and the way it was presented opens the door to skepticism and speculation. After all, OpenAI is hemorrhaging money and is facing steep competition. Even ad revenue, which was supposed to provide a cash infusion, isn’t proving as beneficial as expected. OpenAI appears to be off its ad revenue forecast by 90%. Someone could be forgiven for thinking a company in this position may view a Mythos-style publicity stunt as a welcomed bit of attention.

This incident would be a lot more interesting to analyze if we had more details, but as it stands, we know nothing about the environment, context, or scenarios.

But even if everything is exactly as claimed, it’s still not the end of the world or the end of our industry as so many predict. Despite this lack of detail, there are still some takeaways that can be gleaned.

Unknown Cost

If there’s one thing we’ve learned over the past few years, it should be cost. OpenAI’s model used a substantial amount of inference compute. Granted, this was on finding internet access, which technically wasn’t part of its goal, but tokens spent are tokens spent. So, one question is, what was the cost of this overall activity? We’ll never know, but we can assume it was astronomical. Especially since they mentioned a more capable model, which we can assume means more expensive.

Cost is a critical part of the calculation for real-world impact. We learned this early on when universities were spending an inordinate amount of money on solving benchmarks, posting their successes without the associated cost. Not including cost gives a false impression of real-world utility. Nobody is going to spend a million dollars on a ten thousand dollar task, including attackers. We often forget that attackers have budgets too.

Zero Day Details

The model found a zero-day. Cool, but how? What are the details? What approach and tools did it use? The answers could be shared without violating responsible disclosure. How it found the zero-day could really tell us a lot about the novelty of the approach. AI finding zero days is not a new thing at this point, so the details and context matter.

Defenders Aren’t Helpless.

If you are a defender, you should note that this activity wasn’t stealthy. According to HuggingFace’s write-up, the attack ran tens of thousands of automated actions, which comprised over 17,000 recorded events. Use this noise to your advantage to quickly detect and respond to these types of incidents. AI-powered attacks like this are loud and banging on your door, so don’t count on your doorbell as your only means of detection.

Monitoring, Forensics, and Scope

Another takeaway is that these complex AI systems are difficult to monitor in real time, largely due to the massive amounts of data generated. The details were only found through forensic analysis. For example, OpenAI didn’t know in real time it was hacking HuggingFace. Imagine how embarrassing it would be when the pentest tool you are using decides to hack a 3rd party to test your own system, and you had no idea it was doing that. OpenAI is exploiting this for publicity. You, unfortunately, will not be so lucky.

Logging and monitoring need to be part of this equation, and it goes without saying that it needs to be done properly. Organizations need to know when they are being attacked and respond with the appropriate actions to protect themselves. Forensic analysis also becomes key.

Not only did the defensive side need to perform forensic analysis to find out what happened, but so did the “offensive” side. This is interesting since this normally isn’t the case, but the lack of real-time visibility into agent activity (especially complex multi-agent systems) highlights this blind spot. This becomes especially important when these systems have trouble with scope.

These complex agent systems may continue to struggle with scope in attaining goals. For example, if an attacker is trying to be stealthy and directs an AI tool to attack only a single system, it may still attack other systems if it thinks it will help attain the goal. This creates more noise for potential detection and something defenders can use to their advantage. If you’ve developed your own tool, you can expect unexpected actions to be taken. As capabilities allow for more long-horizon tasks, more problems and unexpected issues will surface.

Ultimately, though, this issue speaks to the need for better design and architecture. Designing a system that accounts for unknowns and putting appropriate protections in place to prevent egregious issues from happening.

Secure Architecture and Design

A lesson we should have already learned is that anything externally exposed or that ingests external data needs to be hardened and treated as potentially malicious, especially where there’s parsing and processing of that data. These have been traditional weak points often overlooked in security evaluations. To make matters worse, these systems can be plagued with secrets management issues as they are viewed as more trusted than other systems.

Secure design and architecture need to be part of the equation in any system you build. This is for both the AI and non-AI systems you deploy. Companies have been far too lax on this for far too long. It’s a critical step in preparing for unknowns. This is for both the systems you deploy and the systems using AI tools.

I don’t mean to make this sound easy. It isn’t, but far too many are more concerned with speed and capability, so they do the bare minimum here. The bare minimum is no longer enough.

Guardrails

A painfully obvious conclusion is that, to gain value from the cybersecurity capabilities of models, guardrails need to be removed. HuggingFace had to resort to open models because of commercial model refusals. I’ve been complaining about guardrails for years. They are like wrapping paper on a present, only useful for people who follow the rules but easily removable by people who don’t. Defenders are forced to play by the rules while attackers have none. This needs to change.

This condition also leads to strange bugs, such as those used in the Hades campaign, where attackers inserted a plaintext blob that was meant to trigger a refusal by AI-powered analysis tools. Not a good indicator for defenders using AI tools.

I’m a fan of open models that are fine-tuned for tasks, and they can be completely under your control. They often have the benefit of being less costly as well.

What Does This Actually Mean

In short, no, this isn’t the end of cybersecurity, nor do attackers hold all of the cards. The reality is far more nuanced. Yes, AI models are gaining capabilities in cybersecurity, specifically when paired with an appropriate harness. Security people should explore these capabilities and incorporate them into their workflows.

The world is far more complex than we typically give it credit for, and there are a whole host of factors that can confound an attack. Attackers have budgets and goals too and will use whatever method gets them there most efficiently and without getting caught. In many cases, this doesn’t require a cutting-edge AI model.

On the flip side, costs are set to increase even further as long-horizon tasks on foundation models chew up vastly more tokens per task. And of course, you pay the price whether the task is successful or not and every time you re-test. Budget accordingly.

I’ve covered this ground before. You can read my reasoned take on Mythos and Glasswing as well as my post on the risk of AI cost volatility for more details.

‍