OpenAI's Greg Brockman - Compute & Scaling Laws Mystery

Compute as OpenAI's core business

00:00It Open AI has multiple stacks to the business, one of which is compute, and you guys have been very aggressive, very aggressive on securing compute. Why is that? >> Well, in many ways, we have a very simple business. We buy, rent, build compute, and we resell it at a margin. That's it. As long as the margin's positive, then you want to scale it because the demand for solving problems, the demand for intelligence, that's unlimited. And the AIs that we have right now really are able to rise to the challenge of effectively any kind of problem that you want to throw at them. >> Do you have enough compute? >> No. >> Really? >> Yeah, definitely not. >> was just with Matt Garman, and he says the GPU compute availability

Never enough compute

00:46in 2026 rounds to zero. Don't you guys have all of it? >> I mean, we have we we would love more. We're we're constantly out there hunting for more, honestly. And I'll tell you like when we first launched when we launched ChatGPT, I remember being on a call with my team, and they were like, "All right, how much compute should we buy?" And I said, "All of it." And they're like, "No, no, no, we're seriously like like come on, how much should we buy?" I'm like, "No matter how fast we try to ramp compute, I guarantee we're not going to be able to keep up with demand, and that has been true ever since." >> Um that's that's fascinating. Moving up from compute since I don't know if much of this audience can help you with

Scaling laws as a beautiful mystery

01:24securing more compute because most of them are founders of startups. About architecture and scaling laws, what are the what are the where are we in the scaling laws? Are they still doubling each year? Are you changing architecture? What's what is what are you guys pushing on the frontier on the research side? >> Well, I would say first of all, the scaling laws are a deep and very beautiful mystery, right? They feel deeply fundamental. It's like the scientific truth that just like you think about physics and, you know, Newton's laws and things like that, they're somehow this truth of the universe. And they're empirical. Like we don't necessarily have all the theory to explain exactly why it works, but to me

02:09the most beautiful thing is that neural networks were really designed like in the 1940s before there were computers. And somehow we've been able to take the exact ideas that were developed back then and apply increasing amounts of computation. And as you pour more compute into the models, they get correspondingly more capable. And it just keeps going. There's no wall. And that's I think that's a beautiful thing. >> That's pretty beautiful. Are there more researched or more algorithms that are in the works that because you know, in the past we had neural networks to your point in 1940s, but we couldn't we didn't have the compute for it. Now that we have the compute for it, you just or are we just pushing the same

New architectures and algorithms

02:50things or are there new architectures and new ideas coming out? >> Yeah, so I would I would think of it as we we absolutely have new ideas that are constantly powering what we do. It's very simplified to say, well, let's take a neural network from the 1940s and you know, put put it in a gigawatt data center, right? We have made tons of innovations and we constantly are improving things. And sometimes these are micro tweaks. Like you just realize that the way you've been formatting data was not quite right and that can actually be a a very big deal. Sometimes it's larger. You think about the shift from the LSTM to the transformer and I don't think the transformer is, you know, like everyone's moved past the

03:26transformer as described in the other 2018 paper. So there's there's constant innovation happening and I think of places that have been perhaps the most invested in long-term research on how to improve the architectures, how to improve the fundamental algorithms, and how to get the paradigm shifts. I think OpenAI has been leading the pack there and that's something we continue to invest in and I see lots of fruit on the horizon. >> Got it. And on the models, um, does OpenAI have a formal definition for AGI? Are we closed? Are we not close? Pat and Sonia published this thing that we are at AGI functionally. Do you agree with that? Do you not agree with that? >> Well, we do have a formal definition,

AGI definition and progress

04:02but to some extent I one thing I have learned is that everyone has their own intuitions about what AGI is and maybe you can view it as like according to my view of where we are. I think we're about 80% of the way there and that we have models that are smart. They're very capable. They're able to if you give >> Are they smarter than you? >> I mean I they're certainly more capable than I am at writing software, right? If you give it all the context, then yes, I I I think that they are they're just so capable. It's it's really remarkable. Like does anyone here feel better at writing software than GPD 5.4? >> Oh. >> All right, writing kernels. Um so even there we're seeing massive gains from Exactly. We're and for some of our

Autonomous overnight systems optimization

04:49internal results um that there we're really seeing if you pour the right kinds of of you know, if you have the right setup for your problem, then you're able to get really massive results out of very low-level um even low-level tasks. And just to give you one example of how things have been trending um one of my systems engineers also very similar was like, "Hey, like I haven't been able to get value out of the models for GPD 5 or 5.1 for 5.2 as well. For 5.3 he on a lark prepar- prepared this design document for a very complicated systems optimization he was about to do. He handed it over to the model, went to sleep waking up intending to like give this to his team to work on for the next week.

05:31And when he woke up it was done. That the model had actually implemented the initial spec, had seen that it was slow, had added instrumentation, had actually run the code, used a profiler to figure out where things were slow, and iterated multiple times until it gotten to an optimized result. And like like that is incredible. That's where we are.