WEBVTT

00:00:00.000 --> 00:00:01.917
what the most surprising thing has been.

00:00:02.000 --> 00:00:03.917
The most surprising thing has been the

00:00:04.000 --> 00:00:07.557
lack of public recognition of how close

00:00:07.640 --> 00:00:09.957
we are to the end of the exponential. To

00:00:10.040 --> 00:00:14.717
me, it is absolutely wild
that, you know, you have

00:00:14.800 --> 00:00:17.837
people, you know, within the bubble
and outside the bubble, you know, but but

00:00:17.920 --> 00:00:19.917
you have people talking about these

00:00:20.000 --> 00:00:22.317
these, you know, just the same tired old

00:00:22.400 --> 00:00:25.277
hot button political issues and like,

00:00:25.360 --> 00:00:30.037
you know, or or around us we're like
near the end of the exponential. I I

00:00:30.120 --> 00:00:32.677
want to understand what
the exponential looks like right

00:00:32.760 --> 00:00:36.597
now because the first question I asked
you when we recorded 3 years ago was,

00:00:36.680 --> 00:00:38.567
you know, what's up with scaling?
How what does it work?

00:00:38.650 --> 00:00:41.397
>> [music] >> Um I have
a similar question now, but I

00:00:41.480 --> 00:00:43.597
feel like it's a more complicated
question because at least from the

00:00:43.680 --> 00:00:47.757
public's point of view, Yes. 3 years ago
there were these, you know, well-known

00:00:47.840 --> 00:00:51.317
public trends [music] where across many
orders of magnitude of compute you can

00:00:51.400 --> 00:00:56.037
see how the loss improves. And now we
have RL scaling and there's no publicly

00:00:56.120 --> 00:01:00.317
known scaling law for it. It's not even
clear what exactly the story is of Is

00:01:00.400 --> 00:01:02.597
this supposed to be teaching the model
skills? Is it supposed to be teaching

00:01:02.680 --> 00:01:06.757
meta learning? Um what [music]
is the scaling hypothesis at this point?

00:01:06.840 --> 00:01:09.157
>> Yeah, so so I have actually the same

00:01:09.240 --> 00:01:14.597
hypothesis that I had even all the way
back in 2017. So in 2017, I think I

00:01:14.680 --> 00:01:18.397
talked about it last time, but I wrote
a doc called the the big blob of compute

00:01:18.480 --> 00:01:22.557
hypothesis. And and you know, it it
wasn't about the scaling of language

00:01:22.640 --> 00:01:26.997
models in particular. When I when I
wrote it, GPT-1 had had just come out,

00:01:27.080 --> 00:01:30.557
right? So that was, you know, one among
many things, right? There [music] was

00:01:30.640 --> 00:01:33.997
back in those days there was robotics,
people tried to work on reasoning as a

00:01:34.080 --> 00:01:37.597
separate thing from language models,
there was [music] scaling of the kind of

00:01:37.680 --> 00:01:39.717
RL that happened that, you know, kind of

00:01:39.800 --> 00:01:45.477
happened in AlphaGo and uh you know,
that that happened at Dota at OpenAI and

00:01:45.560 --> 00:01:48.597
um you know, people remember
StarCraft at DeepMind,

00:01:48.680 --> 00:01:53.597
you know, AlphaStar. Um so uh it was
written as a more general document. And

00:01:53.680 --> 00:01:57.637
and the specific thing I said was the
following, that and you know, it's it's

00:01:57.720 --> 00:02:00.797
very, you know, it sudden put out
the bit bit of a lesson a couple years

00:02:00.880 --> 00:02:02.997
later. Um uh but, you know, the the

00:02:03.080 --> 00:02:05.157
hypothesis is is basically the same. So

00:02:05.240 --> 00:02:07.957
so what it says is all the cleverness,

00:02:08.040 --> 00:02:12.877
all the techniques, all all the kind
of we need a new method to to do something

00:02:12.960 --> 00:02:14.757
like that doesn't matter very much.

00:02:14.840 --> 00:02:18.437
There are only a few things that matter
and I think I listed seven of them. One

00:02:18.520 --> 00:02:21.477
is like how much raw compute you have.

00:02:21.560 --> 00:02:25.957
The other is the quantity of data that
you have. Then the third is kind of the

00:02:26.040 --> 00:02:28.077
quality and distribution of data, right?

00:02:28.160 --> 00:02:30.117
It needs to be a broad broad

00:02:30.200 --> 00:02:32.397
distribution of data. The fourth is, I

00:02:32.480 --> 00:02:35.197
think, how long you train for. Um the

00:02:35.280 --> 00:02:38.077
fifth is you need an objective function

00:02:38.160 --> 00:02:42.317
that can scale to the moon. So
the pre-training objective function is one

00:02:42.400 --> 00:02:47.157
such objective function, right? An another
objective function is, you know,

00:02:47.240 --> 00:02:50.597
the the kind of RL objective function
that says like you have a goal, you're

00:02:50.680 --> 00:02:52.397
going to go out and reach the goal.

00:02:52.480 --> 00:02:55.997
Within that, of course, there's
objective rewards like, you know, like

00:02:56.080 --> 00:02:59.917
you see in math and coding and there's
more subjective rewards like you see in

00:03:00.000 --> 00:03:03.557
RL from human feedback or kind
of higher-order higher-order versions of

00:03:03.640 --> 00:03:06.077
that. And and then the sixth and seventh

00:03:06.160 --> 00:03:10.677
were things around kind of like
normalization or conditioning like, you

00:03:10.760 --> 00:03:13.717
know, just getting the numerical
stability so that kind of the big blob

00:03:13.800 --> 00:03:16.317
of compute flows in this laminar way

00:03:16.400 --> 00:03:18.077
instead of instead of running into

00:03:18.160 --> 00:03:21.117
problems. So that was the hypothesis and

00:03:21.200 --> 00:03:23.757
it's a hypothesis I still hold. I I

00:03:23.840 --> 00:03:28.867
don't think I've seen very much that is
not in line with that hypothesis. And so

00:03:28.950 --> 00:03:31.237
>> [music] >> the pre-training
scaling laws were one

00:03:31.320 --> 00:03:36.037
example of what of of of of kind of what
we see there. And indeed those have

00:03:36.120 --> 00:03:39.917
continued going. Like, you know, uh you
know, I think I think now it's been it's

00:03:40.000 --> 00:03:43.477
been widely reported like, you know,
we feel good about pre-training. Like

00:03:43.560 --> 00:03:45.917
pre-training is continuing to give us

00:03:46.000 --> 00:03:49.357
gains. What has changed is that now

00:03:49.440 --> 00:03:53.637
we're also seeing the same thing for RL,
right? So we're seeing a pre-training

00:03:53.720 --> 00:03:55.677
phase and then we're seeing like an RL

00:03:55.760 --> 00:03:58.477
phase on top of that. Um

00:03:58.560 --> 00:04:01.517
and with RL it's it's actually just the

00:04:01.600 --> 00:04:03.597
same like, you know, even even other

00:04:03.680 --> 00:04:05.877
companies have have published um

00:04:05.960 --> 00:04:10.917
uh um like um you know, in some of their
in some of their releases have published

00:04:11.000 --> 00:04:14.637
things that say, "Look, you know,
we trained the model on math contests, you

00:04:14.720 --> 00:04:17.357
know, AIME or or the kind of other

00:04:17.440 --> 00:04:20.477
things and, you know, how well how well

00:04:20.560 --> 00:04:24.997
the model does is log-linear and how
long we've trained it." And we see that

00:04:25.080 --> 00:04:29.597
as well and it's not just math contest,
it's a wide variety of RL tasks. And so

00:04:29.680 --> 00:04:35.157
we're seeing the same scaling >> [music]
>> in RL that we saw for pre-training. Um

00:04:35.240 --> 00:04:36.437
you mentioned Rich Richard Sutton and

00:04:36.520 --> 00:04:38.597
the bitter lesson. I interviewed him uh

00:04:38.680 --> 00:04:42.317
last year and he is actually very

00:04:42.400 --> 00:04:46.317
non-LLM-pilled. And if I'm if I I
don't know if this is his perspective,

00:04:46.400 --> 00:04:50.557
but one way to paraphrase this
objection is something like, "Look,

00:04:50.640 --> 00:04:55.157
something which possesses the true core
of human learning would not require all

00:04:55.240 --> 00:04:59.237
these billions of dollars of data
and compute and these bespoke environments

00:04:59.320 --> 00:05:03.357
to learn how to use Excel or how does
an account, you know, how to how to use

00:05:03.440 --> 00:05:06.637
PowerPoint, how to navigate a web browser.
And the fact that we have to

00:05:06.720 --> 00:05:10.197
build in these skills using these RL

00:05:10.280 --> 00:05:12.557
environments hints that we're actually

00:05:12.640 --> 00:05:14.597
lacking this

00:05:14.680 --> 00:05:20.157
core human learning algorithm. Uh and so
we're scaling the wrong thing. And so

00:05:20.240 --> 00:05:21.077
yeah, that that that is a good question.

00:05:21.160 --> 00:05:24.197
Why are we doing all this RL scaling
if we do think there's something that's

00:05:24.280 --> 00:05:28.157
going to be human-like in its ability to
learn on the fly? Yeah, yeah. So I think

00:05:28.240 --> 00:05:32.077
I think this kind of puts together
several things that should be kind of

00:05:32.160 --> 00:05:34.317
thought of thought of differently. Yeah.

00:05:34.400 --> 00:05:36.997
I think there is a genuine puzzle here,

00:05:37.080 --> 00:05:41.477
but it it may not matter. Um in fact,
I would guess it probably it probably

00:05:41.560 --> 00:05:44.877
doesn't matter. So let's take
the RL out of it for a second

00:05:44.960 --> 00:05:49.037
cuz I actually think RL and it's a red
herring to say that RL is any different

00:05:49.120 --> 00:05:52.957
from pre-training in this matter.
Um so if we if we look at pre-training

00:05:53.040 --> 00:05:55.397
scaling, um it it was very interesting

00:05:55.480 --> 00:05:58.957
back in, you know, 2017 when Alec

00:05:59.040 --> 00:06:02.277
Radford was doing GPT-1. If you look at

00:06:02.360 --> 00:06:04.797
the models before GPT-1, they were

00:06:04.880 --> 00:06:07.677
trained on these data sets that didn't

00:06:07.760 --> 00:06:10.997
represent a wide, you know, distribution

00:06:11.080 --> 00:06:13.237
of text, right? You had like, you know,

00:06:13.320 --> 00:06:18.197
these very standard, you know, kind
of language modeling benchmarks. And GPT-1

00:06:18.280 --> 00:06:22.077
itself was trained on a bunch of I think
it was fan fiction, actually. Um but,

00:06:22.160 --> 00:06:24.917
you know, it was it was like literary,
you know, it was like literary text,

00:06:25.000 --> 00:06:28.637
which is a very small fraction of the
text that you get. And what we found

00:06:28.720 --> 00:06:31.157
with that, you know, and in those
days it was like a billion words or

00:06:31.240 --> 00:06:34.957
something. So small data sets and

00:06:35.040 --> 00:06:39.840
represented a pretty narrow
distribution, right? Like a narrow

00:06:40.860 --> 00:06:42.880
>> [music]