Dario Amodei: AI Progress Is On Schedule—We're Just Blind to It

Lack of public recognition

00:00what the most surprising thing has been. The most surprising thing has been the lack of public recognition of how close we are to the end of the exponential. To me, it is absolutely wild that, you know, you have people, you know, within the bubble and outside the bubble, you know, but but you have people talking about these these, you know, just the same tired old hot button political issues and like, you know, or or around us we're like near the end of the exponential. I I want to understand what the exponential looks like right now because the first question I asked you when we recorded 3 years ago was, you know, what's up with scaling? How what does it work? >> [music] >> Um I have a similar question now, but I

What is the scaling hypothesis now?

00:41feel like it's a more complicated question because at least from the public's point of view, Yes. 3 years ago there were these, you know, well-known public trends [music] where across many orders of magnitude of compute you can see how the loss improves. And now we have RL scaling and there's no publicly known scaling law for it. It's not even clear what exactly the story is of Is this supposed to be teaching the model skills? Is it supposed to be teaching meta learning? Um what [music] is the scaling hypothesis at this point? >> Yeah, so so I have actually the same hypothesis that I had even all the way back in 2017. So in 2017, I think I talked about it last time, but I wrote a doc called the the big blob of compute

The big blob of compute hypothesis

01:18hypothesis. And and you know, it it wasn't about the scaling of language models in particular. When I when I wrote it, GPT-1 had had just come out, right? So that was, you know, one among many things, right? There [music] was back in those days there was robotics, people tried to work on reasoning as a separate thing from language models, there was [music] scaling of the kind of RL that happened that, you know, kind of happened in AlphaGo and uh you know, that that happened at Dota at OpenAI and um you know, people remember StarCraft at DeepMind, you know, AlphaStar. Um so uh it was written as a more general document. And and the specific thing I said was the following, that and you know, it's it's

01:57very, you know, it sudden put out the bit bit of a lesson a couple years later. Um uh but, you know, the the hypothesis is is basically the same. So so what it says is all the cleverness, all the techniques, all all the kind of we need a new method to to do something like that doesn't matter very much. There are only a few things that matter and I think I listed seven of them. One is like how much raw compute you have. The other is the quantity of data that you have. Then the third is kind of the quality and distribution of data, right? It needs to be a broad broad distribution of data. The fourth is, I think, how long you train for. Um the fifth is you need an objective function that can scale to the moon. So the

02:40pre-training objective function is one such objective function, right? An another objective function is, you know, the the kind of RL objective function that says like you have a goal, you're going to go out and reach the goal. Within that, of course, there's objective rewards like, you know, like you see in math and coding and there's more subjective rewards like you see in RL from human feedback or kind of higher-order higher-order versions of that. And and then the sixth and seventh were things around kind of like normalization or conditioning like, you know, just getting the numerical stability so that kind of the big blob of compute flows in this laminar way instead of instead of running into

03:18problems. So that was the hypothesis and it's a hypothesis I still hold. I I don't think I've seen very much that is not in line with that hypothesis. And so >> [music] >> the pre-training scaling laws were one example of what of of of of kind of what we see there. And indeed those have continued going. Like, you know, uh you know, I think I think now it's been it's been widely reported like, you know, we feel good about pre-training. Like pre-training is continuing to give us gains. What has changed is that now we're also seeing the same thing for RL, right? So we're seeing a pre-training phase and then we're seeing like an RL phase on top of that. Um and with RL it's it's actually just the same like, you know, even even other

RL scaling mirrors pre-training

04:03companies have have published um uh um like um you know, in some of their in some of their releases have published things that say, "Look, you know, we trained the model on math contests, you know, AIME or or the kind of other things and, you know, how well how well the model does is log-linear and how long we've trained it." And we see that as well and it's not just math contest, it's a wide variety of RL tasks. And so we're seeing the same scaling >> [music] >> in RL that we saw for pre-training. Um you mentioned Rich Richard Sutton and the bitter lesson. I interviewed him uh last year and he is actually very non-LLM-pilled. And if I'm if I I don't know if this is his perspective, but one way to paraphrase

The Sutton objection

04:48this objection is something like, "Look, something which possesses the true core of human learning would not require all these billions of dollars of data and compute and these bespoke environments to learn how to use Excel or how does an account, you know, how to how to use PowerPoint, how to navigate a web browser. And the fact that we have to build in these skills using these RL environments hints that we're actually lacking this core human learning algorithm. Uh and so we're scaling the wrong thing. And so yeah, that that that is a good question. Why are we doing all this RL scaling if we do think there's something that's going to be human-like in its ability to learn on the fly? Yeah, yeah. So I think

Why the puzzle may not matter

05:28I think this kind of puts together several things that should be kind of thought of thought of differently. Yeah. I think there is a genuine puzzle here, but it it may not matter. Um in fact, I would guess it probably it probably doesn't matter. So let's take the RL out of it for a second cuz I actually think RL and it's a red herring to say that RL is any different from pre-training in this matter. Um so if we if we look at pre-training scaling, um it it was very interesting back in, you know, 2017 when Alec Radford was doing GPT-1. If you look at the models before GPT-1, they were trained on these data sets that didn't represent a wide, you know, distribution of text, right? You had like, you know,

06:13these very standard, you know, kind of language modeling benchmarks. And GPT-1 itself was trained on a bunch of I think it was fan fiction, actually. Um but, you know, it was it was like literary, you know, it was like literary text, which is a very small fraction of the text that you get. And what we found with that, you know, and in those days it was like a billion words or something. So small data sets and represented a pretty narrow distribution, right? Like a narrow >> [music]