WEBVTT

00:00:00.000 --> 00:00:04.077
So, we talked 3 years ago. I'm curious
in your view, what has been the biggest

00:00:04.160 --> 00:00:06.717
update of the last 3 years? What has
been the biggest difference between what

00:00:06.800 --> 00:00:08.117
it felt like last 3 years versus now?

00:00:08.200 --> 00:00:12.317
>> Yeah, I would say actually
the underlying technology, like the

00:00:12.400 --> 00:00:14.837
exponential of the technology, has has

00:00:14.920 --> 00:00:19.557
gone, broadly speaking, I would say
about about as I expected it to go. I

00:00:19.640 --> 00:00:23.437
mean, there's like plus or minus, you
know, a couple. There's plus or minus a

00:00:23.520 --> 00:00:26.957
year or two here, there's plus or minus
a year or two there. I don't know that I

00:00:27.040 --> 00:00:29.077
would predict the specific direction of

00:00:29.160 --> 00:00:32.157
code. Um, but but actually when I look

00:00:32.240 --> 00:00:35.717
at the exponential, it it is roughly

00:00:35.800 --> 00:00:39.197
what I expected in terms of the march
of the models from like, you know, smart

00:00:39.280 --> 00:00:42.677
high school student to smart college
student to like, you know, beginning to

00:00:42.760 --> 00:00:47.037
do PhD and professional stuff and in the
case of code reaching beyond that. So,

00:00:47.120 --> 00:00:51.197
you know, the frontier is a little bit
uneven. It's roughly what I expected. I

00:00:51.280 --> 00:00:54.277
will tell you though what the most
surprising thing has been. The most

00:00:54.360 --> 00:00:56.717
surprising thing has been the lack of

00:00:56.800 --> 00:01:00.477
public recognition of how close we are

00:01:00.560 --> 00:01:05.237
to the end of the exponential. To me,
it is absolutely wild that, you know, you

00:01:05.320 --> 00:01:09.837
have people, you know, within the bubble
and outside the bubble. You know, but but

00:01:09.920 --> 00:01:11.917
you have people talking about these

00:01:12.000 --> 00:01:14.317
these, you know, just the same tired old

00:01:14.400 --> 00:01:17.277
hot button political issues and like,

00:01:17.360 --> 00:01:21.997
you know, or or or around us we're like
near the end of the exponential. I I I

00:01:22.080 --> 00:01:24.717
want to understand what that
exponential looks like right

00:01:24.800 --> 00:01:28.597
now because the first question I asked
you when we recorded 3 years ago was,

00:01:28.680 --> 00:01:32.037
you know, what's up with scaling?
How what is it work? Um, I have a similar

00:01:32.120 --> 00:01:35.277
question now, but I feel like it's
a more complicated question because at

00:01:35.360 --> 00:01:37.197
least from the public's point of view Yes.

00:01:37.280 --> 00:01:41.997
>> 3 years ago there were these, you know,
well-known public trends where across

00:01:42.080 --> 00:01:45.397
many orders of magnitude of computer you
could see how the loss improves. And now

00:01:45.480 --> 00:01:49.677
we have RL scaling and there's no
publicly known scaling law for it. It's

00:01:49.760 --> 00:01:53.277
not even clear what exactly the story
is of is it supposed to be teaching the

00:01:53.360 --> 00:01:57.237
model skills? Is it supposed to be
teaching metalearning? Um, what is the

00:01:57.320 --> 00:01:58.757
scaling hypothesis at this point?

00:01:58.840 --> 00:02:01.197
>> Yeah, so so I have actually the same

00:02:01.280 --> 00:02:06.597
hypothesis that I had even all the way
back in 2017. So, in 2017, I think I

00:02:06.680 --> 00:02:10.437
talked about it last time, but I wrote
a doc called the the big blob of compute

00:02:10.520 --> 00:02:14.557
hypothesis. And and you know, it it
wasn't about the scaling of language

00:02:14.640 --> 00:02:18.997
models in particular. When I when I
wrote it, GPT-1 had had just come out,

00:02:19.080 --> 00:02:22.597
right? So, that was you know,
one among many things, right? There was

00:02:22.680 --> 00:02:25.997
back in those days, there was robotics,
people trying to work on reasoning as a

00:02:26.080 --> 00:02:28.037
separate thing from language models.

00:02:28.120 --> 00:02:32.037
There was scaling of the kind of RL that
happened that you know, kind of happened

00:02:32.120 --> 00:02:37.717
in AlphaGo, and you know, that that
happened at Dota at OpenAI, and you

00:02:37.800 --> 00:02:40.597
know, people remember
StarCraft at DeepMind,

00:02:40.680 --> 00:02:44.917
you know, the AlphaStar. Um,
so it was written as a more general

00:02:45.000 --> 00:02:49.037
document. And and the specific thing
I said was the following, that and you

00:02:49.120 --> 00:02:52.277
know, it's it's very you know,
Rich Sutton put out the bitter lesson a

00:02:52.360 --> 00:02:56.197
couple years later, but you
know, the the hypothesis is

00:02:56.280 --> 00:02:57.957
basically the same. So, so what it says

00:02:58.040 --> 00:03:00.437
is all the cleverness, all the

00:03:00.520 --> 00:03:05.077
techniques, all all the kind of we need
a new method to to do something like

00:03:05.160 --> 00:03:08.397
that, doesn't matter very much.
There only a few things that matter, and I

00:03:08.480 --> 00:03:10.557
think I listed seven of them. One is

00:03:10.640 --> 00:03:13.637
like how much raw compute you have. The

00:03:13.720 --> 00:03:17.957
other is the quantity of data that you
have. Then the third is kind of the

00:03:18.040 --> 00:03:20.077
quality and distribution of data, right?

00:03:20.160 --> 00:03:22.117
It needs to be a broad broad

00:03:22.200 --> 00:03:24.357
distribution of data. The fourth is I

00:03:24.440 --> 00:03:27.197
think how long you train for. Um, the

00:03:27.280 --> 00:03:30.077
fifth is you need an objective function

00:03:30.160 --> 00:03:34.317
that can scale to the moon. So,
the pre-training objective function is one

00:03:34.400 --> 00:03:39.157
such objective function, right? An another
objective function is, you know,

00:03:39.240 --> 00:03:42.597
the the kind of RL objective function
that says like you have a goal, you're

00:03:42.680 --> 00:03:44.397
going to go out and reach the goal.

00:03:44.480 --> 00:03:48.397
Within that of course, there's objective
rewards like, you know, like you see in

00:03:48.480 --> 00:03:52.317
math and coding, and there's more
subjective rewards like you see in RL

00:03:52.400 --> 00:03:56.597
from human feedback or kind of higher
order higher order versions of that. And

00:03:56.680 --> 00:04:01.717
and then the sixth and seventh were
things around kind of like normalization

00:04:01.800 --> 00:04:04.917
or conditioning like you know just
getting the numerical stability so that

00:04:05.000 --> 00:04:09.677
kind of the big blob of compute flows
in this laminar way instead of instead of

00:04:09.760 --> 00:04:11.837
running into problems. So that was the

00:04:11.920 --> 00:04:14.637
hypothesis and this hypothesis I still

00:04:14.720 --> 00:04:16.997
hold. I I don't think I've seen very

00:04:17.080 --> 00:04:19.397
much that is not in line with that

00:04:19.480 --> 00:04:21.717
hypothesis. And so the pre-training

00:04:21.800 --> 00:04:24.197
scaling laws were one example of what of

00:04:24.280 --> 00:04:29.317
of of of kind of what we see there.
And indeed those have continued going like

00:04:29.400 --> 00:04:33.237
you know you know I think I think now
it's been it's been widely reported like

00:04:33.320 --> 00:04:35.277
you know we feel good about pre-training

00:04:35.360 --> 00:04:37.717
like pre-training is continuing to give

00:04:37.800 --> 00:04:41.317
us gains. What has changed is that now

00:04:41.400 --> 00:04:45.637
we're also seeing the same thing for RL
right so we're seeing a pre-training

00:04:45.720 --> 00:04:47.637
phase and then we're seeing like an RL

00:04:47.720 --> 00:04:50.437
phase on top of that.

00:04:50.520 --> 00:04:53.477
And with RL it's it's actually just the

00:04:53.560 --> 00:04:58.837
same like you know even even
other companies have have published

00:04:58.920 --> 00:05:03.197
like you know in some of their in some
of their releases have published things

00:05:03.280 --> 00:05:04.957
that say look you know we train the

00:05:05.040 --> 00:05:07.997
model on math contests you know AIME or

00:05:08.080 --> 00:05:11.277
or the kind of other things and you know

00:05:11.360 --> 00:05:15.677
how well how well the model does is log
linear and how long we've trained it.

00:05:15.760 --> 00:05:19.997
And we see that as well and it's not
just math contests it's a wide variety

00:05:20.080 --> 00:05:22.397
of RL tasks. And so we're seeing the

00:05:22.480 --> 00:05:25.637
same scaling in RL that we saw for

00:05:25.720 --> 00:05:27.557
pre-training. If you enjoyed this clip

00:05:27.640 --> 00:05:33.680
you can watch the full episode here
and subscribe for more clips. Thanks.