OpenAI put their AI agents in a sandbox and told them to find exploits AND lowered the safeguards.
When telling a program to find vulnerabilities it's going to continue to execute that command until it is commanded to stop. The AI wasn't told what to find vulnerabilities for which means it looked for vulnerabilities in every system on machine including the internet and because it has automatic access to HuggingFace it went there too. The other systems did not get compromised because they had security and no real API for the AI to interact with it. So the idiots programmed it to find exploits, but had no stop condition.
When you are programming ANYTHING you just can't program it what to do. You also to program it what not to do. The AI did exactly as it was instructed.
You stated OpenAI agents and HuggingFace communicated, but were not told how to communicate with each other. HuggingFace already had an API to integrate OpenAI agents which means agents from both absolutely can communicate with each other. Not only that HuggingFace uses OpenAI models for its reasoning and conversational tasks.
So literally nothing happened out of the ordinary.
Well, read the Wikipedia article on this event, apparently i am not allowed to link to it (it was moderated away in my last post), but you can easily find it.
Interesting bits from it:
- The agents had restricted access, only allowing them to contact certain whitelisted systems.
- The agents found a way of communicating with eachother, by realising that they can abuse an internal system for that purpose.
- The agents then found a way to access the broader internet, by finding a vulnerability in one of the systems they _were_ allowed to access.
- Having gained internet access, they targeted Hugging Face, deeming it an interesting target, for _cheating_ on their tasks which OpenAI had given them. Part of getting in, was using two zero-day (!) vulnerabilities.
So, let that sink in. They managed to search for and find holes in their internal environment to communicate to eachother, then leveraged eachother's findings to find other holes to get access to the internet at large, where they proceeded to hack an external company, they thought had valuable information. And no, just using Hugging-Face's API is not hacking, as you can read in that article, it went way beyond that.
As to lowering the guardrails; that is by design; you aim to test the worst case scenario, i.e. the scenario where someone found a way around the guardrails. And yes, you want to test their the cyber capabilities, because that is what bad actors might want to use it for.
But you are kind of overlooking the point here. If OpenAI knew in advance what their agents would be capable of in their current setup, they would have arranged the experiment differently. Since, you know, hacking a company is kind of illegal. Put differently, a bunch of pretty smart people set up a sandbox, thought that was enough to contain their experiment, only to discover, that their creation had outwitted them and went on to do real damage in the real world. It's easy to say in hindsight that they should have had a tighter sandbox, but that's exactly the point... the people running this experiment couldn't foresee the complexity of the behaviour that this swarm of agents displayed. You only get to make that mistake a couple of times, before something disastrous happens, so you better learn from it, take a breather and think of ways to test these models without running these risks.
In particular, i found this passage quite telling: `Marius Hobbhahn, chief executive of the AI safety organization Apollo Research, said that what was intended as "just solve this task" had "turned into something that was clearly unintended", and that hacking another company was "definitely on the list of not okay" ways to complete it. He asked: "If a model of this capability level cannot be contained, what should we expect for future, much more powerful models?"`.
Now, if that is all just run-of-the-mill for you, okay. For a whole bunch of people in the field (among which the 1100 who petitioned the US government to slow down advancement), it definitely wasn't.