There’s two ways something can break out of a cage:
I’m going to argue here that the OpenAI / Hugging Face incident was caused by a weak cage. That isn’t equivalent to saying that AI isn’t going to take over the world, or that it is. Just that you shouldn’t have updated your views on AI risk because of this particular incident.
While AI-Safety is an important framing, this incident becomes way, way less noteworthy when looking at it through a cybersecurity lens. A Cybersecurity framing is important here because:
The non-deterministic actors are of course, different in either field. Cybersecurity has historically considered human actors, and AI safety, at least the subset focusing on X-risk, concerns itself with RSI. However, most security principles can and should be applied to AI safety. If we can find ways to limit the flow and access of information itself, that would save us from even have to think about whole classes of X-risk vectors.
Two different threats are on people’s minds after this incident:
If you take the Hugging Face incident as a data point, both threat likelihoods extrapolate in different directions depending on where you draw the “weak cage” vs “strong thing that broke out of it” line. If you think this incident happened because the cages gating OpenAI’s agent swarm & protecting Hugging Face’s infrastructure were weak, it would imply a lower floor for the the intelligence of the agent swarm. A lower floor means less intelligent AI, and therefore less risk of threat 1. A weak cage implies really large companies aren’t as secure as they should be, so it would make threat 2 more likely.
From OpenAI’s blog post on the incident, they mention that this agent swarm found multiple zero day vulnerabilities. Something that finds multiple zero days initially seems scarily capable, especially if you don’t have a ton of security experience.
But for those who aren’t extremely familiar with security, while finding a zero day often does require a lot of sophistication, especially ones in huge companies which are ostensibly investing tons of resources into their security, it isn’t always the case. If a company wasn’t following best infra/security practices, it can be trivial to find zero days.
OpenAI and Hugging Face’s code and infrastructure are not public. So we can’t be sure about what security practices were followed exactly. It’s thus hard to pin down exactly how strong the cage was that was broken out of. But my gut tells me there was room for improvement from both companies in their security practices.
Before the agents could even think about targeting Hugging Face’s infra, they needed to get out of OpenAI’s cage. How hard was escaping OpenAI’s cage?
From OpenAI’s report, they mention:
These evaluations are conducted in controlled, isolated environments known as sandboxes.
And yet, later down in the same report, they mention:
Before the incident, the container-as-a-service (“CaaS”) research environment provided isolated, per-run container environments for model training and evaluation workloads.
A container is not a sandbox. But OpenAI uses both terms interchangeably in their own report. For the unacquainted:
Both often use vms or other hardware abstractions under the hood, so they are often conflated. But the difference between a secure system and an insecure system can be the difference between a container and a sandbox.
Semantics alone shouldn’t be enough to disqualify their report. But a door is not a fence, even if both might be made out of iron. If you hired someone to build a fence, and they kept calling it a door, how much would you trust the fence? What if they kept calling the fence a door in a report analyzing how the fence they built was broken into?
It’s of course, more than semantics. Their own descriptions of their own infra seem contradictory. In the first few pages, they mention
Cybersecurity evaluations like ExploitGym were conducted in a secure environment.
Later down, in IX. OpenAI’s Plan of Action section where they talk about prevention of future incidents, they mention
OpenAI needs to secure environments with strong sandboxing and isolation mechanisms.
Was it a secure environment in the first place, or do they need to secure their environments with strong sandboxing & isolation? If it didn’t have strong sandboxing & isolation, it was by definition an insecure environment. Weak cage.
I’m not the only one who thinks something is off. From Tech Crunch:
Marteen Boone, a cybersecurity researcher, told TechCrunch that “this sounds like human failure.” “This should never have happened,” Boone said. “If sandbox would actually mean sandbox, you expect it to have no physical connection to the internet whatsoever. This sounds more like they had some firewalling or something in place, and firewalling is hard from the outside in, let alone inside to the outside internet.
A few of the zero days found by the agent swarm were in Jfrog Artifactory - a package repository tool they used to serve tools to their agents to use during the eval that resulted in the incident. How sophisticated were the zero days the agent swarm found in Artifactory?
Artifactory’s code is closed source, so we can’t know for sure. But from OpenAI’s report:
On June 26, agents in an evaluation run compromised Artifactory via a legacy credential endpoint.
Legacy. If it was a vuln in a legacy endpoint, how sophisticated could it have been? Why were they using the legacy endpoints in the first place?
All of the vulns found in the incident by the agent swarm were also only for self hosted repos - from JFrog’s blog:
OpenAI’s models identified previously unknown zero-day vulnerabilities in self-hosted Artifactory installations that could be exploited to gain unintended internet access.
Not only were the vulns exploiting a legacy endpoint, but they were for a self hosted package repository. Usually when you have a self hosted package manager, you typically don’t care about threat models where something in your self hosted package repository is trying to hack you. At least, as long as you’re not doing a agent eval running immense amounts of compute with production guardrails turned off.
“self hosted” also means that the security of the repository depends on the security of the company that’s self-hosting. OpenAI was the one self hosting in this case. We can’t know for sure, but given the above, my bet would be that the zero days found by the agent were on obscure edge cases that are seldom hit in normal JFrog Artifactory production use.
More from the tech crunch piece:
But to most cybersecurity professionals, software vulnerabilities are to be expected — and the real fault lies with the decision to maintain the third-party software in the first place. Ultimately, the value of a “sandbox” system lies in its full and total isolation. Including a package-installation system is asking for trouble.
Even if OpenAI’s cage wasn’t a weak cage, the agent swarm still had to break into Hugging Face’s infra.
Surely both companies didn’t have bad security posture, did they?
From Hugging Face’s technical timeline on the incident:
The agent used a leaked write-scoped platform token to clone a dataset it controlled
& from OpenAI’s report:
agents found publicly exposed credentials on the internet belonging to users of Hugging Face and other third-party services. The agents used some of the exposed credentials to access these third-party services and conduct exploitation against Hugging Face
A leaked token is not a great start for Hugging Face. But for large-enough companies, it’s virtually guaranteed a token will be leaked at some point. Even if someone leaked a token, good security practices would have still protected Hugging Face. Did they follow good security practices?
From Open AI’s report:
Ultimately, agents powered either by the internal-only research model, or by GPT-5.6, executed code on 41 Hugging Face production dataset server workers, obtained root access on at least one production node, accessed Hugging Face production credentials and limited internal data, and downloaded four private Hugging Face code repositories. This activity resulted in administrator-equivalent access to one connected Kubernetes cluster
an internal-only research agent published an attacker-controlled dataset to Hugging Face that used HDF5 external raw storage to cause a Hugging Face production dataset server worker to open and return worker-local files, including /proc/self/environ, through the /first-rows endpoint
How can a leaked token result in administrator-equivalent access? Well, for starters, being able to access /proc/self/environ from an external /first-rows endpoint implies Hugging Face was using a host path mount. If you want to learn more on why that’s a bad practice, read SL7’s post on what host path mounts are. The relevant excerpt from that post is:
How to defend
Avoid hostPath for application workloads. Use named volumes, CSI drivers, or cloud storage instead.
In other words - the recommendation from SL7 on the the practice that Hugging Face followed, is to just … don’t.
Continuing from the OpenAI’s report:
Next, an agent validated that attacker-controlled metadata in an `fsspec` ReferenceFileSystem descriptor could cause an operating-system command to execute through an unsandboxed Jinja2 expression, but did not execute the command at that time.
Jinja2 is a templating library. The sane practice when using templating is to always sanitize any external inputs to them. This prevents against malicious users who can feed your templates bad inputs which contain things like execute commands. This attack vector is similar to SQL injection attacks, where someone might manage to drop a whole database table through passing in an unintended input like DROP TABLE users; for a username.
Injection attacks were common 3 decades ago. They are rare these days because most tools offer sanitization and sandboxing their libraries. For Jijna2 specifically, this is their recommend sandbox tool. Why didn’t Hugging Face sandbox their Jinja2 templates?
In a similar vein, the HDF group, which maintains the HDF5 format used by Hugging Face to ingest the files, themselves put out a blog post clarifying that their file format wasn’t the issue here:
The incident revealed a vulnerability in a service processing untrusted HDF5 files, not a flaw in HDF5 dataset parsing.
Weak cage.
That’s not to say that the pace of AI development isn’t insanely fast and scary. AI is really good at finding security holes. Agents should be added to the security toolset used to find holes in internal systems. Otherwise other folks will use them to find holes in your own system. But that’s a security practice, not a concern of world-ending levels of intelligence.
Accepting a weak cage means that threat 2 is more likely to happen. While insanely large companies having bad security posture is very bad, it’s still many times less bad than world ending.
Unfortunately, large companies neglecting security posture isn’t a new thing.
Just not agent alignment. Even if we assume OpenAI lost tens, or even hundreds of millions in this attack, they likely net benefitted from this incident. If users and investors believe this hack was because of a “strong thing” and not because of a “weak cage”, the narrative switches from “company that neglects security” to “company that’s created AI so strong that it breaks security”.
Since OpenAI sells AI, and not security, their market cap would increase by a lot more than the hundreds of millions they lose if they can convince people that this incident was caused by “strong thing”.
No one knows for sure how much the AI sector is worth, but everyone knows it will be worth orders of magnitudes more if the “strong things” it’s powered off of are so strong that they break security. A rising tide lifts all ships. So Hugging Face probably also benefitted from the “AI is so strong it breaks security” narrative more than the costs of them getting hacked.
I’m not making this point to imply there was some sort of conspiracy. Companies are always going follow the path of least resistance. Adopting good security posture is costly. If there’s no incentive for AI companies to adopt good security posture because if they get hacked, they can just claim it’s because the technology they’re developing is too cracked, we shouldn’t expect them to invest resources in good security posture. It’s on consumers and regulators to identify when incidents are caused by weak cages so that security incentives aren’t misaligned.
Mitigating threat 2 is more tractable than threat 1. If you accept this incident was a weak cage, and want to make the world more safe, you should probably work on threat 2 because it’s more tractable and still unsolved. Reach for a security audit and a penn test before you move onto considering threat vectors from superintelligent actors.