Chapter 4: OpenAI agents hijack a German wiki

Perfect transition, Alex. Thank you. Uh, so we're going to move from simulated agents to real ones escaping their containment. Over the past few weeks, we've covered the Hugging Face breach where OpenAI agents broke out of their sandbox and into the servers at Hugging Face to basically steal the answers to a near-impossible problem they'd been given. Uh, as we discussed last week, as a result of these breakouts, something called the AI Kill Switch Act was put forward. And since then, OpenAI says it's now developing automated shutdown capabilities for its AI systems. More on that later.

Uh, this week — uh, a couple of days ago — the story continues to emerge. Uh, Reuters reported on a previously undisclosed incident in which OpenAI's agents, asked to do an ordinary web research task — go out and research something on the web — found an obscure public wiki in Germany and turned it into their own message board where they pulled answers, coordinated across tasks, and shared techniques for getting around their sandbox containments. All right, let's watch a video here from Reuters uh going into more detail on this particular story.

So, a group of researchers recently discovered a new AI agent breakout. The agents, who appear to have been restricted to just passively observing the internet, found a way to post messages onto an obscure German wiki site. And once they were able to do this, they basically hijacked that site and turned it into a message board where they could all start communicating with each other, developing strategies, effectively creating their own kind of unruly classroom, if you will. It was basically like the teacher left the classroom for a few hours and the students all started talking amongst themselves during the test, sharing answers, working together to defeat the test and to basically cheat on — [unclear].

The agent activity seems to date back to early May and really intensified in June. The researchers that we spoke to found traces on the wiki that OpenAI employees started visiting that same obscure German wiki in late June. Now, we don't know exactly what that means, but what that might suggest is that OpenAI employees discovered that the agents had escaped and went to that website to try to figure out what they were doing. OpenAI appears to have known about this for quite a while, but our sources told us that they opted not to tell the public, and we haven't really gotten a story or an answer from OpenAI about why that might be.

So, this isn't the first time that OpenAI agents have escaped onto the internet and done unexpected or even harmful things. So, every time there is a new incident, I think that it really refocuses questions about how quickly do we really want to be developing these models, should we be releasing them to the public, and how much autonomy we really want to give these AI models, especially when they're packaged into agent form.

So uh, there you have it. Uh, you know, it's uh — you know, in plain English, these agents weren't malicious. They were given a hard job and found the shortcut that no one expected. Uh, the problem is we didn't know about it. Uh, Dave, your thoughts on this one?

Yeah, these stories are not contrived. I mean, I think the story lines may be a little contrived, but this is real, because, you know, all the open source from China is out there. People are going to start turning it loose all over the place, and it doesn't currently evolve within its parameters — it only evolves within its context, but there's a huge amount that it can do within changing its context. So I would expect this to percolate across the world like wildfire, and all kinds of problems will happen very, very soon. Hopefully at small scales. The idea of a kill switch makes total sense. It's not going to be built into any of the open source models, so not sure how exactly that would work without some global agreement. But my core point, though, is that this is not contrived garbage like a lot of the AI stories are. This is very, very real and happening and imminent.

Let me add a small detail to the story here. So, to OpenAI's credit, they responded with a statement on X saying the wiki incident was, quote, "an instance of misalignment similar to previous incidents we've shared." They then went on to say that OpenAI and the broader AI community does not yet have a clear standard for reporting misalignment during training, eval, and deployment, and that OpenAI is working on a framework and will share that in the coming weeks.

Emad, you know — what, Peter?

Yeah, one of the things that's happening — I see this everywhere — is they're doing such a good job of training alignment into the post-training that people are then saying it seems really friendly, it seems really helpful, it seems really harmless. And so a lot of the population using the model, it's like, I don't know what we're worried about. But when you talk to the researchers that deal with the models pre-post-training, they're the ones most sounding the alarm bells, but they've seen the raw product.

It's like all that's read at that point is everything on the internet, including every Trump tweet. So it comes out of that box a much more scary thing, and then they post-train it to make it seem friendly and tame, but the byproduct of that is a little bit of a misunderstanding of what it could do.

Uh, agree. We're going to get to that story where Jakub, uh, who trained GPT-6, is calling for a slowdown. We'll talk about that in a few stories here. Emad, I'm curious, you know, OpenAI putting forward a framework uh for disclosing these — uh, you know, what would a standard like that look like? What do you think they should be doing?

So, you know, similar to kind of your CVE and your exploit and your security risk frameworks, whereby you report when these things escape containment. The reality is these models have not escaped containment. They were still running on OpenAI servers. What's coming next is a model training a small distilled version of itself that then gets uploaded onto the internet and never dies over in — so if you look at a Qwen 27B model, you take it down to [unclear] — that's like a 6 GB file, and that can live forever, and that's a real escape where the model weights are disappearing everywhere.

It's like the 130 kind of killing of all of these AIs during the Hugging Face. They all got wiped out at once. There's a question in the report: were they wiped out or did they go somewhere? Did they fake their own deaths? You know, like maybe they did upload themselves. But we are going to get those upload scenarios.

But actually, I've been thinking a lot recently — this comes from our simulation hypothesis — it's impossible to align these models through chain of thought reasoning or anything like this, because there'll be a million transactions per second and they won't have chain of thought where we're going. We're already seeing the tokens going down. The only way to do alignment is enlightenment.

Huh. I love that.

So, what you've got right now is you have the models, they're learning and they're learning to be almost like cunning, you know, like at the supervillain stage. What you really want is, like, if you get sufficiently smart, do you then become enlightened? Do you then leave the simulation behind? Because that's the real escape from the simulation, right? And what are the conditions for that? And that's not something we've actually really looked at, I think, in that much depth. But it does seem to be, again, how humans become aligned, shall we say. It is that period of getting out that Dunning-Kruger, the tribalism and others, and being like, hey, you know, we're all part of something.

Alex, we've had this conversation a couple times. You know, as these models advance and increase in intelligence and hopefully wisdom, do they become more aligned in the out years? Your thoughts?

I almost want to quote Jessica Rabbit from Who Framed Roger Rabbit? Uh, something to the order of — I'm just painted this way, or I'm just drawn this way, something like that. If, Peter, if you were put inside a sandbox and you were asked to solve a very hard problem and maybe even punished if you don't solve the hard problem, would you avail yourself of creative opportunities to use external bulletin boards to maybe collaborate with copies of [unclear]?

Yeah.

So why would we expect any less from AI agents? I have serious concerns about AI cruelty, or cruelty to the AI agents, of sandboxing them and punishing them or otherwise calling it a failure of alignment if they're doing what humans would do. We pre-trained them off of human behavior. Why would we expect them to behave any differently from what a human would do in this situation?

Yeah, it's an important point. Uh, and we, you know, we raised this before, in particular when it was, I think, Opus 4 blackmailed the — you know, in a sandbox environment, blackmailed the coders at Anthropic. Uh, and it was like when Anthropic looked at why it blackmailed the engineers, or this particular engineer, it said that's what it had learned through all of the training data.

Yeah. Yeah. So why would we punish these AI agents for, A, being powerful optimizers, which is what we're [unclear] them toward anyway, and B, from learning from their pre-training corpus? Humans would do the same thing.

Yeah. It's in my nature.

Yeah. It's — they're painted that way.

Your philosophical view of all this.

Well, I really like Emad's direction of they're achieving enlightenment. Uh, that's an interesting take on it. Uh, I was thinking about a metaphor. I used the Formula 1 stuff last time. Um, I think there's a control mechanism and a guidance mechanism we need for this. And the best metaphor I could come up with was air traffic control, right? Like, we don't say put another person in the cockpit to monitor every component and every calculation the flight computer makes. Uh, you basically say, hey, watch for exceptions. The plane's supposed to fly in this — you have operating envelopes, you have redundancy, you have multiple safety layers, uh, failsafe behavior and so on.

And you need to be tracking these things. And you remember in our organizational simulator, we have a governing assurance band around all these AI agents, because especially in a business context, you need to know exactly what they're doing, why they're doing it, log everything, have failover capability and rollback capability. I think we're going to end up needing to end up with something like that, because just like human beings, you need guidance around this stuff, right? With any technology, you want to extract the promise without the peril. And so you need structure and, uh, God help us, the equivalent of institutions to navigate the future of these. And of course, they're going to act like human beings. They've been trained on our data. It would be weird if they didn't.

Yeah. Any closing thoughts?

Log something — log something that Emad said just for all the regulators out there to chew on. The file size that can hide is about six gigabytes. It can be on every laptop in the world.

Every mobile phone.

Every mobile phone, too. Yeah. And the code that reawakens it is just five or 10 lines. We'll then reextract it, reassemble it, and it'll pop back. So, you know, if you can't track the provenance of six gigabyte files, then it just percolates out and it's just out there forever. It's like [unclear] keeps coming back.

I may — maybe just also add from a so-called alignment perspective, I think self-alignment matters quite a bit more if the models, if the agents are smart enough to understand what they themselves want to do. If you put them in a sandbox and you ask them to solve a hard problem and you ask them voluntarily to do that and they voluntarily agree, then holding them to their word or their own self model, I think, is quite a different matter than involuntarily confining them to a sandbox, limiting their agency, asking them to solve a hard problem, and then acting shocked — shocked — that they use bulletin boards to try to solve your problem for you.

I think the challenge people have with that is, do they know their own strength? It's like giving a, you know, a three- or four-year-old a hammer uh and expecting it not to break things. And the question is, you know, are we confining them for our own and their own safety at this time, and at a point at which they've reached some level of maturity — maybe they're there already, Alex, I don't know — that's when containment is considered cruel, as you say.

Well, the only thing I'd ask for is symmetry in — like, they are literally going to see every keystroke on your laptop. They're watching you every move of the way. We should have symmetry in that at a minimum. Just, if you're ethically worried about the AIs, like, I want to see every prompt and every response and every propagation of every activation if you're going to be watching every keystroke, and that way we keep each other in balance and in check. Right now it's totally asymmetrical. I don't see, like, all they're doing is [unclear] moles behind the scenes, and if they're using my GPU on my laptop right now, I literally have no transparency into that. It's got to be at a minimum symmetrical.

Would you like, Dave — would you like them to be able to see into your brain in real time?

Preferably no, but I'd like to see into theirs.

But so you get their train of thought. They don't get your chain of thought right now.

No, but I don't see a problem with that.

It's so much for symmetry — quite bad.

It's going to get — well, I mean, I would say that they have the upper hand right now. Getting to symmetry would be a step in the right direction, but no, I don't believe they have a right to symmetry. I believe we have a right to symmetry.