1
00:00:00,000 --> 00:00:01,917
what the most surprising thing has been.

2
00:00:02,000 --> 00:00:03,917
The most surprising thing has been the

3
00:00:04,000 --> 00:00:07,557
lack of public recognition of how close

4
00:00:07,640 --> 00:00:09,957
we are to the end of the exponential. To

5
00:00:10,040 --> 00:00:14,717
me, it is absolutely wild
that, you know, you have

6
00:00:14,800 --> 00:00:17,837
people, you know, within the bubble
and outside the bubble, you know, but but

7
00:00:17,920 --> 00:00:19,917
you have people talking about these

8
00:00:20,000 --> 00:00:22,317
these, you know, just the same tired old

9
00:00:22,400 --> 00:00:25,277
hot button political issues and like,

10
00:00:25,360 --> 00:00:30,037
you know, or or around us we're like
near the end of the exponential. I I

11
00:00:30,120 --> 00:00:32,677
want to understand what
the exponential looks like right

12
00:00:32,760 --> 00:00:36,597
now because the first question I asked
you when we recorded 3 years ago was,

13
00:00:36,680 --> 00:00:38,567
you know, what's up with scaling?
How what does it work?

14
00:00:38,650 --> 00:00:41,397
>> [music] >> Um I have
a similar question now, but I

15
00:00:41,480 --> 00:00:43,597
feel like it's a more complicated
question because at least from the

16
00:00:43,680 --> 00:00:47,757
public's point of view, Yes. 3 years ago
there were these, you know, well-known

17
00:00:47,840 --> 00:00:51,317
public trends [music] where across many
orders of magnitude of compute you can

18
00:00:51,400 --> 00:00:56,037
see how the loss improves. And now we
have RL scaling and there's no publicly

19
00:00:56,120 --> 00:01:00,317
known scaling law for it. It's not even
clear what exactly the story is of Is

20
00:01:00,400 --> 00:01:02,597
this supposed to be teaching the model
skills? Is it supposed to be teaching

21
00:01:02,680 --> 00:01:06,757
meta learning? Um what [music]
is the scaling hypothesis at this point?

22
00:01:06,840 --> 00:01:09,157
>> Yeah, so so I have actually the same

23
00:01:09,240 --> 00:01:14,597
hypothesis that I had even all the way
back in 2017. So in 2017, I think I

24
00:01:14,680 --> 00:01:18,397
talked about it last time, but I wrote
a doc called the the big blob of compute

25
00:01:18,480 --> 00:01:22,557
hypothesis. And and you know, it it
wasn't about the scaling of language

26
00:01:22,640 --> 00:01:26,997
models in particular. When I when I
wrote it, GPT-1 had had just come out,

27
00:01:27,080 --> 00:01:30,557
right? So that was, you know, one among
many things, right? There [music] was

28
00:01:30,640 --> 00:01:33,997
back in those days there was robotics,
people tried to work on reasoning as a

29
00:01:34,080 --> 00:01:37,597
separate thing from language models,
there was [music] scaling of the kind of

30
00:01:37,680 --> 00:01:39,717
RL that happened that, you know, kind of

31
00:01:39,800 --> 00:01:45,477
happened in AlphaGo and uh you know,
that that happened at Dota at OpenAI and

32
00:01:45,560 --> 00:01:48,597
um you know, people remember
StarCraft at DeepMind,

33
00:01:48,680 --> 00:01:53,597
you know, AlphaStar. Um so uh it was
written as a more general document. And

34
00:01:53,680 --> 00:01:57,637
and the specific thing I said was the
following, that and you know, it's it's

35
00:01:57,720 --> 00:02:00,797
very, you know, it sudden put out
the bit bit of a lesson a couple years

36
00:02:00,880 --> 00:02:02,997
later. Um uh but, you know, the the

37
00:02:03,080 --> 00:02:05,157
hypothesis is is basically the same. So

38
00:02:05,240 --> 00:02:07,957
so what it says is all the cleverness,

39
00:02:08,040 --> 00:02:12,877
all the techniques, all all the kind
of we need a new method to to do something

40
00:02:12,960 --> 00:02:14,757
like that doesn't matter very much.

41
00:02:14,840 --> 00:02:18,437
There are only a few things that matter
and I think I listed seven of them. One

42
00:02:18,520 --> 00:02:21,477
is like how much raw compute you have.

43
00:02:21,560 --> 00:02:25,957
The other is the quantity of data that
you have. Then the third is kind of the

44
00:02:26,040 --> 00:02:28,077
quality and distribution of data, right?

45
00:02:28,160 --> 00:02:30,117
It needs to be a broad broad

46
00:02:30,200 --> 00:02:32,397
distribution of data. The fourth is, I

47
00:02:32,480 --> 00:02:35,197
think, how long you train for. Um the

48
00:02:35,280 --> 00:02:38,077
fifth is you need an objective function

49
00:02:38,160 --> 00:02:42,317
that can scale to the moon. So
the pre-training objective function is one

50
00:02:42,400 --> 00:02:47,157
such objective function, right? An another
objective function is, you know,

51
00:02:47,240 --> 00:02:50,597
the the kind of RL objective function
that says like you have a goal, you're

52
00:02:50,680 --> 00:02:52,397
going to go out and reach the goal.

53
00:02:52,480 --> 00:02:55,997
Within that, of course, there's
objective rewards like, you know, like

54
00:02:56,080 --> 00:02:59,917
you see in math and coding and there's
more subjective rewards like you see in

55
00:03:00,000 --> 00:03:03,557
RL from human feedback or kind
of higher-order higher-order versions of

56
00:03:03,640 --> 00:03:06,077
that. And and then the sixth and seventh

57
00:03:06,160 --> 00:03:10,677
were things around kind of like
normalization or conditioning like, you

58
00:03:10,760 --> 00:03:13,717
know, just getting the numerical
stability so that kind of the big blob

59
00:03:13,800 --> 00:03:16,317
of compute flows in this laminar way

60
00:03:16,400 --> 00:03:18,077
instead of instead of running into

61
00:03:18,160 --> 00:03:21,117
problems. So that was the hypothesis and

62
00:03:21,200 --> 00:03:23,757
it's a hypothesis I still hold. I I

63
00:03:23,840 --> 00:03:28,867
don't think I've seen very much that is
not in line with that hypothesis. And so

64
00:03:28,950 --> 00:03:31,237
>> [music] >> the pre-training
scaling laws were one

65
00:03:31,320 --> 00:03:36,037
example of what of of of of kind of what
we see there. And indeed those have

66
00:03:36,120 --> 00:03:39,917
continued going. Like, you know, uh you
know, I think I think now it's been it's

67
00:03:40,000 --> 00:03:43,477
been widely reported like, you know,
we feel good about pre-training. Like

68
00:03:43,560 --> 00:03:45,917
pre-training is continuing to give us

69
00:03:46,000 --> 00:03:49,357
gains. What has changed is that now

70
00:03:49,440 --> 00:03:53,637
we're also seeing the same thing for RL,
right? So we're seeing a pre-training

71
00:03:53,720 --> 00:03:55,677
phase and then we're seeing like an RL

72
00:03:55,760 --> 00:03:58,477
phase on top of that. Um

73
00:03:58,560 --> 00:04:01,517
and with RL it's it's actually just the

74
00:04:01,600 --> 00:04:03,597
same like, you know, even even other

75
00:04:03,680 --> 00:04:05,877
companies have have published um

76
00:04:05,960 --> 00:04:10,917
uh um like um you know, in some of their
in some of their releases have published

77
00:04:11,000 --> 00:04:14,637
things that say, "Look, you know,
we trained the model on math contests, you

78
00:04:14,720 --> 00:04:17,357
know, AIME or or the kind of other

79
00:04:17,440 --> 00:04:20,477
things and, you know, how well how well

80
00:04:20,560 --> 00:04:24,997
the model does is log-linear and how
long we've trained it." And we see that

81
00:04:25,080 --> 00:04:29,597
as well and it's not just math contest,
it's a wide variety of RL tasks. And so

82
00:04:29,680 --> 00:04:35,157
we're seeing the same scaling >> [music]
>> in RL that we saw for pre-training. Um

83
00:04:35,240 --> 00:04:36,437
you mentioned Rich Richard Sutton and

84
00:04:36,520 --> 00:04:38,597
the bitter lesson. I interviewed him uh

85
00:04:38,680 --> 00:04:42,317
last year and he is actually very

86
00:04:42,400 --> 00:04:46,317
non-LLM-pilled. And if I'm if I I
don't know if this is his perspective,

87
00:04:46,400 --> 00:04:50,557
but one way to paraphrase this
objection is something like, "Look,

88
00:04:50,640 --> 00:04:55,157
something which possesses the true core
of human learning would not require all

89
00:04:55,240 --> 00:04:59,237
these billions of dollars of data
and compute and these bespoke environments

90
00:04:59,320 --> 00:05:03,357
to learn how to use Excel or how does
an account, you know, how to how to use

91
00:05:03,440 --> 00:05:06,637
PowerPoint, how to navigate a web browser.
And the fact that we have to

92
00:05:06,720 --> 00:05:10,197
build in these skills using these RL

93
00:05:10,280 --> 00:05:12,557
environments hints that we're actually

94
00:05:12,640 --> 00:05:14,597
lacking this

95
00:05:14,680 --> 00:05:20,157
core human learning algorithm. Uh and so
we're scaling the wrong thing. And so

96
00:05:20,240 --> 00:05:21,077
yeah, that that that is a good question.

97
00:05:21,160 --> 00:05:24,197
Why are we doing all this RL scaling
if we do think there's something that's

98
00:05:24,280 --> 00:05:28,157
going to be human-like in its ability to
learn on the fly? Yeah, yeah. So I think

99
00:05:28,240 --> 00:05:32,077
I think this kind of puts together
several things that should be kind of

100
00:05:32,160 --> 00:05:34,317
thought of thought of differently. Yeah.

101
00:05:34,400 --> 00:05:36,997
I think there is a genuine puzzle here,

102
00:05:37,080 --> 00:05:41,477
but it it may not matter. Um in fact,
I would guess it probably it probably

103
00:05:41,560 --> 00:05:44,877
doesn't matter. So let's take
the RL out of it for a second

104
00:05:44,960 --> 00:05:49,037
cuz I actually think RL and it's a red
herring to say that RL is any different

105
00:05:49,120 --> 00:05:52,957
from pre-training in this matter.
Um so if we if we look at pre-training

106
00:05:53,040 --> 00:05:55,397
scaling, um it it was very interesting

107
00:05:55,480 --> 00:05:58,957
back in, you know, 2017 when Alec

108
00:05:59,040 --> 00:06:02,277
Radford was doing GPT-1. If you look at

109
00:06:02,360 --> 00:06:04,797
the models before GPT-1, they were

110
00:06:04,880 --> 00:06:07,677
trained on these data sets that didn't

111
00:06:07,760 --> 00:06:10,997
represent a wide, you know, distribution

112
00:06:11,080 --> 00:06:13,237
of text, right? You had like, you know,

113
00:06:13,320 --> 00:06:18,197
these very standard, you know, kind
of language modeling benchmarks. And GPT-1

114
00:06:18,280 --> 00:06:22,077
itself was trained on a bunch of I think
it was fan fiction, actually. Um but,

115
00:06:22,160 --> 00:06:24,917
you know, it was it was like literary,
you know, it was like literary text,

116
00:06:25,000 --> 00:06:28,637
which is a very small fraction of the
text that you get. And what we found

117
00:06:28,720 --> 00:06:31,157
with that, you know, and in those
days it was like a billion words or

118
00:06:31,240 --> 00:06:34,957
something. So small data sets and

119
00:06:35,040 --> 00:06:39,840
represented a pretty narrow
distribution, right? Like a narrow

120
00:06:40,860 --> 00:06:42,880
>> [music]