Jakub Pachocki's essay "An alien mind" and its call for voluntary slowdowns

All right, I'm going to slow us down for this next story that I think may be one of the most important stories uh this week other than Navier-Stokes. Um, and some quick context here. Uh, Jakub Pachocki is OpenAI's chief scientist. He's the person who built the reasoning models that led to Astra. He's not a critic. He's not a doomer, not a politician. He's a builder.

And a few days ago, on Saturday, you know, he published an essay on OpenAI's site titled, quote, "An alien mind." He opens with a memory. He says in mid-2023, inside of a project called RL Slow, his team saw the first results proving reasoning models could scale. Um, let me read what he said. He said, quote, "Simon and I spent that night at the office thinking not about the incredible benchmark numbers, products, or scientific results, but rather trying to process the sobering fact that we will actually see machines meaningfully smarter than ourselves in our lifetime."

Three years later, this is what Jakub is saying today. Quote, "Based on internal results, I have strong expectations that this speed of progress could be sustained into recursive self-improvement." We just heard Emad speak about that.

Jakub also explains why these systems are hard to control, and it's one of the best one-liner descriptions that I've ever read. He said, quote, "AI is grown more than designed. We don't engineer it. We run an optimization step billions of times on a giant computer and study what comes out the way neuroscientists study a brain."

Uh, let me share his conclusion from his paper. He says, quote, "Currently, I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer." Quote, "I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established." And he calls for international coordination on AI to become a, quote, "top priority for governments around the world."

So there you have it. Three days after shipping the most capable model in the world, the man who built it says we need to slow down. So I think this is a conversation important for us to have. Uh, Emad, I'm going to go to you first. Your thoughts here.