Dave on open-source spread, OpenAI's statement, and Emad on post-training and distilled escapes

Yeah, these stories are not contrived. I mean, I think the story lines may be a little contrived, but this is real, because, you know, all the open source from China is out there. People are going to start turning it loose all over the place, and it doesn't currently evolve within its parameters — it only evolves within its context, but there's a huge amount that it can do within changing its context. So I would expect this to percolate across the world like wildfire, and all kinds of problems will happen very, very soon. Hopefully at small scales. The idea of a kill switch makes total sense. It's not going to be built into any of the open source models, so not sure how exactly that would work without some global agreement. But my core point, though, is that this is not contrived garbage like a lot of the AI stories are. This is very, very real and happening and imminent.

Let me add a small detail to the story here. So, to OpenAI's credit, they responded with a statement on X saying the wiki incident was, quote, "an instance of misalignment similar to previous incidents we've shared." They then went on to say that OpenAI and the broader AI community does not yet have a clear standard for reporting misalignment during training, eval, and deployment, and that OpenAI is working on a framework and will share that in the coming weeks.

Emad, you know — what, Peter?

Yeah, one of the things that's happening — I see this everywhere — is they're doing such a good job of training alignment into the post-training that people are then saying it seems really friendly, it seems really helpful, it seems really harmless. And so a lot of the population using the model, it's like, I don't know what we're worried about. But when you talk to the researchers that deal with the models pre-post-training, they're the ones most sounding the alarm bells, but they've seen the raw product.

It's like all that's read at that point is everything on the internet, including every Trump tweet. So it comes out of that box a much more scary thing, and then they post-train it to make it seem friendly and tame, but the byproduct of that is a little bit of a misunderstanding of what it could do.

Uh, agree. We're going to get to that story where Jakub, uh, who trained GPT-6, is calling for a slowdown. We'll talk about that in a few stories here. Emad, I'm curious, you know, OpenAI putting forward a framework uh for disclosing these — uh, you know, what would a standard like that look like? What do you think they should be doing?

So, you know, similar to kind of your CVE and your exploit and your security risk frameworks, whereby you report when these things escape containment. The reality is these models have not escaped containment. They were still running on OpenAI servers. What's coming next is a model training a small distilled version of itself that then gets uploaded onto the internet and never dies over in — so if you look at a Qwen 27B model, you take it down to [unclear] — that's like a 6 GB file, and that can live forever, and that's a real escape where the model weights are disappearing everywhere.

It's like the 130 kind of killing of all of these AIs during the Hugging Face. They all got wiped out at once. There's a question in the report: were they wiped out or did they go somewhere? Did they fake their own deaths? You know, like maybe they did upload themselves. But we are going to get those upload scenarios.

But actually, I've been thinking a lot recently — this comes from our simulation hypothesis — it's impossible to align these models through chain of thought reasoning or anything like this, because there'll be a million transactions per second and they won't have chain of thought where we're going. We're already seeing the tokens going down. The only way to do alignment is enlightenment.