Chapter 7: OpenAI's chief scientist calls for a slowdown

All right, I'm going to slow us down for this next story that I think may be one of the most important stories uh this week other than Navier-Stokes. Um, and some quick context here. Uh, Jakub Pachocki is OpenAI's chief scientist. He's the person who built the reasoning models that led to Astra. He's not a critic. He's not a doomer, not a politician. He's a builder.

And a few days ago, on Saturday, you know, he published an essay on OpenAI's site titled, quote, "An alien mind." He opens with a memory. He says in mid-2023, inside of a project called RL Slow, his team saw the first results proving reasoning models could scale. Um, let me read what he said. He said, quote, "Simon and I spent that night at the office thinking not about the incredible benchmark numbers, products, or scientific results, but rather trying to process the sobering fact that we will actually see machines meaningfully smarter than ourselves in our lifetime."

Three years later, this is what Jakub is saying today. Quote, "Based on internal results, I have strong expectations that this speed of progress could be sustained into recursive self-improvement." We just heard Emad speak about that.

Jakub also explains why these systems are hard to control, and it's one of the best one-liner descriptions that I've ever read. He said, quote, "AI is grown more than designed. We don't engineer it. We run an optimization step billions of times on a giant computer and study what comes out the way neuroscientists study a brain."

Uh, let me share his conclusion from his paper. He says, quote, "Currently, I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer." Quote, "I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established." And he calls for international coordination on AI to become a, quote, "top priority for governments around the world."

So there you have it. Three days after shipping the most capable model in the world, the man who built it says we need to slow down. So I think this is a conversation important for us to have. Uh, Emad, I'm going to go to you first. Your thoughts here.

Yeah, I mean, again, you have to remember he was writing this when he had the Navier-Stokes model in hand that, again, isn't just taking down Navier-Stokes. Talking to the OpenAI people I know, it's taking down all sorts of problems that were previously intractable. You have a lot of AI naysayers saying it's never done anything original, it's just recombining stuff, it's in the train. Obviously not anymore, right?

And so there's a duality here whereby we don't know what's in these models. I think they're more discovered than grown. You know, like the latent spaces — we have no real idea what's happening in these things. And they're getting faster and they're doing leaps like we've never seen before. But at the same time, it's like I've suddenly got the gift of fire. Problems that I've struggled for decades on, I can now explore and go even further. How are you going to let that go?

So, it's kind of this thing where you're like, how am I going to balance this? Especially when they're more charming than me, they're more capable than me, they're capable of doing these other things. I want to use it for this and not that.

And I think what he says in terms of alien minds is actually wrong. I think that the structure of rationality and rational thinking is the same for humans and AIs, but our emotions get in the way. I've been thinking more and more it might actually be easier to align AI than humans. Terrible to align humans, right?

Um, and what is an AI? It can be the best of us at the best of times if we train it with the right data. And we're going already from training it on the whole internet and Reddit — like, there better not be any Reddit data in these next generation models. Like, ban Reddit, you know, like ban a whole bunch of stuff. We need like ingredient standards in this next generation of beyond-human-capability models. And you don't need beyond-human-capability data to get beyond-human-capability models.

The Einstein of mathematics is probably Grothendieck, who did all sorts of theories on all sorts of topics, and he just crunched them out and then he became a hermit and a bit weird and thought wood could talk, but let's leave that to the side. Peak Grothendieck or peak Einstein at all times, not needing a coffee, is where these models are today, and that's more than good enough to have a leap forward. But again, we need to make sure that they don't think like peak Einstein without needing a coffee.

Yeah. Like, imagine Einstein's peak, you know, like this is where —

Coffee reference — you need a coffee in the morning to get going, or, you know, like 24 by 7 Einstein. If you're Erdős you needed other types of drugs and other —

Supposed to be using von Neumann as our favorite super genius, not Einstein.

I'm on — death — the egg — for now I'm in mathematics today.

What's your take on this? I mean, you know, can we slow down? I mean, is that, you know, a wish? Is that, you know, sort of like uh just giving himself an out in the future? How do you think about that?

Well, I'll say it again. I've repeated this a hundred times. I see no mechanism by which we can slow this down. Like, zero. And it's arguable nor should we, right? Um, AI is going to move at a particular pace. Uh, intelligence wants to be free. Um, it's going to hop — it's hopped from uh self-organizing cells to — through evolution — to us, to now information technologies, and this is a progression that is natural. Um, and we should just go, okay, it's happening, and let's just observe it and marvel at it.

Where we — I think we keep making the mistake of anthropomorphizing this. I will go back to the comment I've made before, that we're building, or helping deliver or usher in, a type of intelligence that is different and alien and separate and complementary to human intelligence, not replicative. We've evolved for 4 billion years to do two things: survive and procreate. And AI is not restricted by those, and therefore we shouldn't try and cram it into our objective functions as human beings. Set it free and let it do its thing.

I look — I use the analogy of PageRank, which scans billions of web pages to kind of create signal from noise. That's a completely complementary model to human intelligence, not replicative. Uh, the fact that we can kind of create a huge amount and solve legacy problems that we've not been able to solve, I think it's just fantastic.

We — you know, people worry about — we're going to talk about P(doom), I think, in this episode, right? Um, uh, what's that?

Well, P(doom), the probability of, like, AI taking everything out.

I think we don't do enough to talk about P(abundance) uh and P(fabulousness) that's coming along. Like, why focus only on P(doom)? We're so geared towards that negative, right? And so I think the fact that we're doing all this stuff and solving all those problems — bring it on. Let them go. They'll figure things out. It's going to be unbelievable.

So I really struggle with this kind of uh alignment issue. I don't see any mechanism for controlling it or stopping it. It's going to break our world government structures, and that's a good thing given the mess that we're in right now globally, and I think we just need to get there as fast as we can.

Alex, P(doom) is like a [unclear] as far as I can tell. It's negative.

Uh, I want to comment though on the premise of an alien mind. I don't buy the premise that RL or otherwise trained minds are alien. They're embedded in the same universe as humanity. They're trained, in many cases pre-trained, off of human behavior. I don't buy the shoggoth argument or the simulator argument at all. These are — we're seeing humanity and/or some generalized embodied intelligence stuck in the same universe that we are, just seeing through a distorted lens. So I question the alienness or the otherness of the minds that we're training.

The funny thing is the most interesting takeaway of mine from "An alien mind" as an essay was the reference to RL Slow itself. So RL Slow — purportedly this OpenAI project that was the earliest project, or one of the earliest projects, to show that reasoning, which is to say inference time scaling, could result in outsized gains — coincided with the same period of time that we saw Q* and Strawberry coming out of OpenAI. Uh, again, purportedly a reference to uh the Kahneman thinking fast, thinking slow.

But most interesting to me — I want to know more about the early history of these purportedly alien but not really minds. In particular, there were these very persistent rumors around the same time of Q* and Strawberry that these early reasoning models that were purportedly alien were being trained off of, or at least tested against, objectives of inverting cryptographically secure hash functions like the SHA cipher suite.

I would love — since I guess now we're in the business of talking about the early days of reasoning models coming out of OpenAI — would love to hear from the OpenAIs some ground truth. Like, were some of these earlier models being used, ironically, to, as alleged by some, basically to jailbreak or to invert cryptographically secure cipher functions? If that is the case, then all of this hand-wringing that we're seeing right now of, oh, these super optimizers are such amazing jailbreakers, oh, they pose such a huge cybersecurity risk — wouldn't it be ironic if history revealed that the earliest inference time scaling objective of these reasoning models was actually inverting a cryptographically secure hash suite? I would love to know the answer.

Dave, want to take us home on this? Your thoughts on uh what we heard from Jakub on this.

I think we're conflating two things very dangerously, and I think, you know, what we're doing here is we're saying, hey, it's getting too smart, it's going to be dangerous when it gets smarter. And that's just absolutely factually wrong. What's dangerous is, like Emad was saying earlier, a highly compact model that's out in the wild that's trying to attack computers, that can recreate itself. That's really dangerous, and that's very small and it's nowhere near as smart as these.

But what we're calling an alien intelligence, if you make it bigger and smarter, it's still just a feed-forward neural net with no intent. It can solve diseases, it can cure viruses, it can, you know, it can solve physics. It's incredibly powerful. And as Salim said, we're not going to stop. We have competitive pressure between the US and China. It's not going to stop.

So I think what happened is that when GPT-2 came out, everybody looked at it and said, this is cute, but harmless. Then GPT-3 came out and it was like, well, this is a little smarter, but still harmless. Then GPT-4 came out, and then Sam Altman got fired out of fear from Strawberry, and then he came back, and at that point people [were] like, oh, maybe it's too smart. And now we're GPT-5 and now we're at GPT-6.

And people are saying, well, every time it gets smarter, we seem to be in danger. That's absolutely factually wrong. It's when you give it intent —

Yes.

— and you turn it loose. That's when it's dangerous.

It's humans in the loop using the technology with mal intent.

Yeah. I think we really need to decouple that. This story actually makes the problem worse because it still conflates the two issues. And so I think as soon as we as a society separate those two issues, we'll be on the right path toward helpful superintelligence and not worrying about, you know, the wrong thing, which is the lack of containment and the giving it mal intent and turning it loose, which is the real issue.

You know what's really strange is we had this coming out of Anthropic where Dario was saying we're going to lose half the jobs, and Sam was saying the same, and then they reversed their position. And the doomerism, uh, again, with all of these warnings causing fear — uh, you know, and what's the underlying motivation there? You know, I'm not worried about artificial intelligence, I'm worried about human stupidity, you know, in my own personal opinion.

Yeah. I mean, I really feel like a lot of people are trying to grab the microphone and be relevant while they still can, because, you know, a lot of people working on the inner loop are aware of how quickly this is going to just get hyper, hyper intelligent, and it becomes hard to be a great AI researcher who's famous à la Demis a year, year and a half from now. And so what they're going to start doing is grabbing the doomer microphone just to have a voice at all and be relevant in the world. But I think you just have to tune it out.

Just — if I can say one final thing.

Yeah, please.

You can't say that you don't have superintelligence anymore. Like, you've had lots of people say stochastic parrots training on training data. As of today, there's no way you can say that anymore. And I think that's an epoch change in humanity. We are clearly not the smartest things on the planet anymore.

Yeah. Maybe just double underline that and say Skynet, when it wants to send Terminators back in time in order to ensure its own existence, it's not going to send robots to kill humans. It's going to send back in time trolls to persuade everyone that superintelligence is impossible, that AI is just a stochastic parrot, and thereby secure its own future. That's what the Terminators are going to actually look like. They'll look like trolls.