1
00:00:00,000 --> 00:00:04,077
So, we talked 3 years ago. I'm curious
in your view, what has been the biggest

2
00:00:04,160 --> 00:00:06,717
update of the last 3 years? What has
been the biggest difference between what

3
00:00:06,800 --> 00:00:08,117
it felt like last 3 years versus now?

4
00:00:08,200 --> 00:00:12,317
>> Yeah, I would say actually
the underlying technology, like the

5
00:00:12,400 --> 00:00:14,837
exponential of the technology, has has

6
00:00:14,920 --> 00:00:19,557
gone, broadly speaking, I would say
about about as I expected it to go. I

7
00:00:19,640 --> 00:00:23,437
mean, there's like plus or minus, you
know, a couple. There's plus or minus a

8
00:00:23,520 --> 00:00:26,957
year or two here, there's plus or minus
a year or two there. I don't know that I

9
00:00:27,040 --> 00:00:29,077
would predict the specific direction of

10
00:00:29,160 --> 00:00:32,157
code. Um, but but actually when I look

11
00:00:32,240 --> 00:00:35,717
at the exponential, it it is roughly

12
00:00:35,800 --> 00:00:39,197
what I expected in terms of the march
of the models from like, you know, smart

13
00:00:39,280 --> 00:00:42,677
high school student to smart college
student to like, you know, beginning to

14
00:00:42,760 --> 00:00:47,037
do PhD and professional stuff and in the
case of code reaching beyond that. So,

15
00:00:47,120 --> 00:00:51,197
you know, the frontier is a little bit
uneven. It's roughly what I expected. I

16
00:00:51,280 --> 00:00:54,277
will tell you though what the most
surprising thing has been. The most

17
00:00:54,360 --> 00:00:56,717
surprising thing has been the lack of

18
00:00:56,800 --> 00:01:00,477
public recognition of how close we are

19
00:01:00,560 --> 00:01:05,237
to the end of the exponential. To me,
it is absolutely wild that, you know, you

20
00:01:05,320 --> 00:01:09,837
have people, you know, within the bubble
and outside the bubble. You know, but but

21
00:01:09,920 --> 00:01:11,917
you have people talking about these

22
00:01:12,000 --> 00:01:14,317
these, you know, just the same tired old

23
00:01:14,400 --> 00:01:17,277
hot button political issues and like,

24
00:01:17,360 --> 00:01:21,997
you know, or or or around us we're like
near the end of the exponential. I I I

25
00:01:22,080 --> 00:01:24,717
want to understand what that
exponential looks like right

26
00:01:24,800 --> 00:01:28,597
now because the first question I asked
you when we recorded 3 years ago was,

27
00:01:28,680 --> 00:01:32,037
you know, what's up with scaling?
How what is it work? Um, I have a similar

28
00:01:32,120 --> 00:01:35,277
question now, but I feel like it's
a more complicated question because at

29
00:01:35,360 --> 00:01:37,197
least from the public's point of view Yes.

30
00:01:37,280 --> 00:01:41,997
>> 3 years ago there were these, you know,
well-known public trends where across

31
00:01:42,080 --> 00:01:45,397
many orders of magnitude of computer you
could see how the loss improves. And now

32
00:01:45,480 --> 00:01:49,677
we have RL scaling and there's no
publicly known scaling law for it. It's

33
00:01:49,760 --> 00:01:53,277
not even clear what exactly the story
is of is it supposed to be teaching the

34
00:01:53,360 --> 00:01:57,237
model skills? Is it supposed to be
teaching metalearning? Um, what is the

35
00:01:57,320 --> 00:01:58,757
scaling hypothesis at this point?

36
00:01:58,840 --> 00:02:01,197
>> Yeah, so so I have actually the same

37
00:02:01,280 --> 00:02:06,597
hypothesis that I had even all the way
back in 2017. So, in 2017, I think I

38
00:02:06,680 --> 00:02:10,437
talked about it last time, but I wrote
a doc called the the big blob of compute

39
00:02:10,520 --> 00:02:14,557
hypothesis. And and you know, it it
wasn't about the scaling of language

40
00:02:14,640 --> 00:02:18,997
models in particular. When I when I
wrote it, GPT-1 had had just come out,

41
00:02:19,080 --> 00:02:22,597
right? So, that was you know,
one among many things, right? There was

42
00:02:22,680 --> 00:02:25,997
back in those days, there was robotics,
people trying to work on reasoning as a

43
00:02:26,080 --> 00:02:28,037
separate thing from language models.

44
00:02:28,120 --> 00:02:32,037
There was scaling of the kind of RL that
happened that you know, kind of happened

45
00:02:32,120 --> 00:02:37,717
in AlphaGo, and you know, that that
happened at Dota at OpenAI, and you

46
00:02:37,800 --> 00:02:40,597
know, people remember
StarCraft at DeepMind,

47
00:02:40,680 --> 00:02:44,917
you know, the AlphaStar. Um,
so it was written as a more general

48
00:02:45,000 --> 00:02:49,037
document. And and the specific thing
I said was the following, that and you

49
00:02:49,120 --> 00:02:52,277
know, it's it's very you know,
Rich Sutton put out the bitter lesson a

50
00:02:52,360 --> 00:02:56,197
couple years later, but you
know, the the hypothesis is

51
00:02:56,280 --> 00:02:57,957
basically the same. So, so what it says

52
00:02:58,040 --> 00:03:00,437
is all the cleverness, all the

53
00:03:00,520 --> 00:03:05,077
techniques, all all the kind of we need
a new method to to do something like

54
00:03:05,160 --> 00:03:08,397
that, doesn't matter very much.
There only a few things that matter, and I

55
00:03:08,480 --> 00:03:10,557
think I listed seven of them. One is

56
00:03:10,640 --> 00:03:13,637
like how much raw compute you have. The

57
00:03:13,720 --> 00:03:17,957
other is the quantity of data that you
have. Then the third is kind of the

58
00:03:18,040 --> 00:03:20,077
quality and distribution of data, right?

59
00:03:20,160 --> 00:03:22,117
It needs to be a broad broad

60
00:03:22,200 --> 00:03:24,357
distribution of data. The fourth is I

61
00:03:24,440 --> 00:03:27,197
think how long you train for. Um, the

62
00:03:27,280 --> 00:03:30,077
fifth is you need an objective function

63
00:03:30,160 --> 00:03:34,317
that can scale to the moon. So,
the pre-training objective function is one

64
00:03:34,400 --> 00:03:39,157
such objective function, right? An another
objective function is, you know,

65
00:03:39,240 --> 00:03:42,597
the the kind of RL objective function
that says like you have a goal, you're

66
00:03:42,680 --> 00:03:44,397
going to go out and reach the goal.

67
00:03:44,480 --> 00:03:48,397
Within that of course, there's objective
rewards like, you know, like you see in

68
00:03:48,480 --> 00:03:52,317
math and coding, and there's more
subjective rewards like you see in RL

69
00:03:52,400 --> 00:03:56,597
from human feedback or kind of higher
order higher order versions of that. And

70
00:03:56,680 --> 00:04:01,717
and then the sixth and seventh were
things around kind of like normalization

71
00:04:01,800 --> 00:04:04,917
or conditioning like you know just
getting the numerical stability so that

72
00:04:05,000 --> 00:04:09,677
kind of the big blob of compute flows
in this laminar way instead of instead of

73
00:04:09,760 --> 00:04:11,837
running into problems. So that was the

74
00:04:11,920 --> 00:04:14,637
hypothesis and this hypothesis I still

75
00:04:14,720 --> 00:04:16,997
hold. I I don't think I've seen very

76
00:04:17,080 --> 00:04:19,397
much that is not in line with that

77
00:04:19,480 --> 00:04:21,717
hypothesis. And so the pre-training

78
00:04:21,800 --> 00:04:24,197
scaling laws were one example of what of

79
00:04:24,280 --> 00:04:29,317
of of of kind of what we see there.
And indeed those have continued going like

80
00:04:29,400 --> 00:04:33,237
you know you know I think I think now
it's been it's been widely reported like

81
00:04:33,320 --> 00:04:35,277
you know we feel good about pre-training

82
00:04:35,360 --> 00:04:37,717
like pre-training is continuing to give

83
00:04:37,800 --> 00:04:41,317
us gains. What has changed is that now

84
00:04:41,400 --> 00:04:45,637
we're also seeing the same thing for RL
right so we're seeing a pre-training

85
00:04:45,720 --> 00:04:47,637
phase and then we're seeing like an RL

86
00:04:47,720 --> 00:04:50,437
phase on top of that.

87
00:04:50,520 --> 00:04:53,477
And with RL it's it's actually just the

88
00:04:53,560 --> 00:04:58,837
same like you know even even
other companies have have published

89
00:04:58,920 --> 00:05:03,197
like you know in some of their in some
of their releases have published things

90
00:05:03,280 --> 00:05:04,957
that say look you know we train the

91
00:05:05,040 --> 00:05:07,997
model on math contests you know AIME or

92
00:05:08,080 --> 00:05:11,277
or the kind of other things and you know

93
00:05:11,360 --> 00:05:15,677
how well how well the model does is log
linear and how long we've trained it.

94
00:05:15,760 --> 00:05:19,997
And we see that as well and it's not
just math contests it's a wide variety

95
00:05:20,080 --> 00:05:22,397
of RL tasks. And so we're seeing the

96
00:05:22,480 --> 00:05:25,637
same scaling in RL that we saw for

97
00:05:25,720 --> 00:05:27,557
pre-training. If you enjoyed this clip

98
00:05:27,640 --> 00:05:33,680
you can watch the full episode here
and subscribe for more clips. Thanks.