1
00:00:00,640 --> 00:00:04,212
We talked three years ago. In your view,
what has been the biggest update over

2
00:00:04,296 --> 00:00:05,197
the last three years?

3
00:00:05,280 --> 00:00:09,230
What has been the biggest difference
between what it felt like then versus now?

4
00:00:10,880 --> 00:00:17,880
Broadly speaking, the exponential of the
underlying technology has gone about as

5
00:00:18,800 --> 00:00:23,437
I expected it to go. There's plus or minus

6
00:00:23,520 --> 00:00:27,357
a year or two here and there.
I don't know that I would've

7
00:00:27,440 --> 00:00:34,237
predicted the specific direction of code.
But when I look at the exponential,

8
00:00:34,320 --> 00:00:39,517
it is roughly what I expected in terms
of the march of the models from smart high

9
00:00:39,600 --> 00:00:44,201
school student to smart college student
to beginning to do PhD and professional

10
00:00:44,284 --> 00:00:47,037
stuff, and in the case
of code reaching beyond that.

11
00:00:47,120 --> 00:00:49,517
The frontier is a little
bit uneven, but it's

12
00:00:49,600 --> 00:00:55,117
roughly what I expected.
What has been the most surprising

13
00:00:55,200 --> 00:01:00,694
thing is the lack of public recognition
of how close we are to the end of

14
00:01:00,778 --> 00:01:01,917
the exponential.

15
00:01:02,000 --> 00:01:08,957
To me, it is absolutely wild that you have
people — within the bubble and outside the

16
00:01:09,040 --> 00:01:16,040
bubble — talking about the same tired, old
hot-button political issues, when we are

17
00:01:19,920 --> 00:01:23,917
near the end of the exponential.
I want to understand what that

18
00:01:24,000 --> 00:01:26,957
exponential looks like right now.
The first question I asked you when

19
00:01:27,040 --> 00:01:30,990
we recorded three years ago was, "what’s
up with scaling and why does it work?"

20
00:01:31,440 --> 00:01:34,640
I have a similar question now,
but it feels more complicated.

21
00:01:35,280 --> 00:01:40,830
At least from the public's point of view,
three years ago there were well-known

22
00:01:40,913 --> 00:01:41,757
public trends

23
00:01:41,840 --> 00:01:45,037
across many orders of magnitude of compute
where you could see how the loss improves.

24
00:01:45,120 --> 00:01:49,597
Now we have RL scaling and there's
no publicly known scaling law for it.

25
00:01:49,680 --> 00:01:53,643
It's not even clear what the story is.
Is this supposed to be teaching the model

26
00:01:53,726 --> 00:01:56,449
skills? Is it supposed
to be teaching meta-learning?

27
00:01:56,532 --> 00:01:59,040
What is the scaling
hypothesis at this point?

28
00:01:59,840 --> 00:02:06,397
I actually have the same hypothesis
I had even all the way back in 2017.

29
00:02:06,480 --> 00:02:10,437
I think I talked about it last time,
but I wrote a doc called "The Big Blob

30
00:02:10,521 --> 00:02:11,760
of Compute Hypothesis".

31
00:02:12,880 --> 00:02:15,517
It wasn't about the scaling
of language models in particular.

32
00:02:15,600 --> 00:02:21,680
When I wrote it GPT-1 had just come out.
That was one among many things.

33
00:02:22,720 --> 00:02:26,077
Back in those days there was robotics.
People tried to work on reasoning as

34
00:02:26,160 --> 00:02:30,157
a separate thing from language models,
and there was scaling of the kind of RL

35
00:02:30,240 --> 00:02:35,992
that happened in AlphaGo
and in Dota at OpenAI.

36
00:02:36,076 --> 00:02:42,160
People remember StarCraft
at DeepMind, AlphaStar.

37
00:02:43,520 --> 00:02:50,520
It was written as a more general document.
Rich Sutton put out "The Bitter

38
00:02:52,160 --> 00:02:57,437
Lesson" a couple years later.
The hypothesis is basically the same.

39
00:02:57,520 --> 00:03:02,035
What it says is that all
the cleverness, all the techniques,

40
00:03:02,118 --> 00:03:04,717
all the "we need a new method to do

41
00:03:04,800 --> 00:03:08,477
something", that doesn't matter very much.
There are only a few things that matter.

42
00:03:08,560 --> 00:03:13,597
I think I listed seven of them.
One is how much raw compute you have.

43
00:03:13,680 --> 00:03:19,612
The second is the quantity of data.
The third is the quality and distribution

44
00:03:19,695 --> 00:03:20,237
of data.

45
00:03:20,320 --> 00:03:26,560
It needs to be a broad distribution.
The fourth is how long you train for.

46
00:03:27,200 --> 00:03:31,997
The fifth is that you need an objective
function that can scale to the moon.

47
00:03:32,080 --> 00:03:35,840
The pre-training objective function
is one such objective function.

48
00:03:36,720 --> 00:03:42,557
Another is the RL objective
function that says you have a goal,

49
00:03:42,640 --> 00:03:46,128
you're going to go out
and reach the goal. Within that,

50
00:03:46,212 --> 00:03:48,077
there's objective rewards like

51
00:03:48,160 --> 00:03:52,077
you see in math and coding, and there's
more subjective rewards like you see in

52
00:03:52,160 --> 00:03:59,117
RLHF or higher-order versions of that.
Then the sixth and seventh were things

53
00:03:59,200 --> 00:04:04,637
around normalization or conditioning,
just getting the numerical stability

54
00:04:04,720 --> 00:04:10,181
so that the big blob of compute flows in
this laminar way instead of running into

55
00:04:10,264 --> 00:04:15,597
problems. That was the hypothesis,
and it's a hypothesis I still hold.

56
00:04:15,680 --> 00:04:19,600
I don't think I've seen very
much that is not in line with it.

57
00:04:21,040 --> 00:04:26,740
The pre-training scaling laws were
one example of what we see there.

58
00:04:26,824 --> 00:04:29,120
Those have continued going.

59
00:04:31,120 --> 00:04:35,837
Now it's been widely reported,
we feel good about pre-training.

60
00:04:35,920 --> 00:04:41,837
It’s continuing to give us gains.
What has changed is that now we're

61
00:04:41,920 --> 00:04:46,077
also seeing the same thing for RL.
We're seeing a pre-training phase

62
00:04:46,160 --> 00:04:53,160
and then an RL phase on top of that.
With RL, it’s actually just the same.

63
00:04:55,280 --> 00:05:02,280
Even other companies have published
things in some of their releases that say,

64
00:05:03,851 --> 00:05:10,031
"We train the model on math
contests — AIME or other things —

65
00:05:10,114 --> 00:05:14,557
and how well the model
does is log-linear in

66
00:05:14,640 --> 00:05:17,437
how long we've trained it."
We see that as well,

67
00:05:17,520 --> 00:05:21,757
and it's not just math contests.
It's a wide variety of RL tasks.

68
00:05:21,840 --> 00:05:27,197
We're seeing the same scaling
in RL that we saw for pre-training.

69
00:05:27,280 --> 00:05:31,757
You mentioned Rich Sutton and "The Bitter
Lesson". I interviewed him last year,

70
00:05:31,840 --> 00:05:38,477
and he's actually very non-LLM-pilled.
I don’t know if this is his perspective,

71
00:05:38,560 --> 00:05:41,425
but one way to paraphrase
his objection is:

72
00:05:41,509 --> 00:05:44,717
Something which possesses
the true core of human

73
00:05:44,800 --> 00:05:49,816
learning would not require all these
billions of dollars of data and compute

74
00:05:49,899 --> 00:05:55,617
and these bespoke environments, to learn
how to use Excel, how to use PowerPoint,

75
00:05:55,701 --> 00:05:57,837
how to navigate a web browser.

76
00:05:57,920 --> 00:06:03,533
The fact that we have to build in these
skills using these RL environments hints

77
00:06:03,617 --> 00:06:08,637
that we are actually lacking
a core human learning algorithm.

78
00:06:08,720 --> 00:06:12,957
So we're scaling the wrong thing.
That does raise

79
00:06:13,040 --> 00:06:15,895
the question. Why are we doing all this RL
scaling if we think there's something

80
00:06:15,979 --> 00:06:20,037
that's going to be human-like
in its ability to learn on the fly?

81
00:06:20,120 --> 00:06:23,517
I think this puts together
several things that

82
00:06:23,600 --> 00:06:29,037
should be thought of differently.
There is a genuine puzzle here,

83
00:06:29,120 --> 00:06:33,040
but it may not matter.
In fact, I would guess it probably

84
00:06:33,760 --> 00:06:35,404
doesn't matter.
There is an interesting thing.

85
00:06:35,487 --> 00:06:37,357
Let me take the RL out
of it for a second, because I

86
00:06:37,440 --> 00:06:42,032
actually think it's a red herring to say
that RL is any different from pre-training

87
00:06:42,115 --> 00:06:42,960
in this matter.

88
00:06:43,760 --> 00:06:47,677
If we look at pre-training scaling,
it was very interesting

89
00:06:47,760 --> 00:06:53,297
back in 2017 when Alec
Radford was doing GPT-1.

90
00:06:53,380 --> 00:06:59,037
The models before GPT-1
were trained on datasets

91
00:06:59,120 --> 00:07:06,120
that didn't represent a wide distribution
of text. You had very standard language

92
00:07:08,080 --> 00:07:11,277
modeling benchmarks.
GPT-1 itself was trained on

93
00:07:11,360 --> 00:07:17,757
a bunch of fanfiction, I think actually.
It was literary text, which is a very

94
00:07:17,840 --> 00:07:23,197
small fraction of the text you can get.
In those days it was like a billion words

95
00:07:23,280 --> 00:07:30,280
or something, so small datasets
representing a pretty narrow distribution

96
00:07:31,244 --> 00:07:37,997
of what you can see in the world.
It didn't generalize well.

97
00:07:38,080 --> 00:07:43,280
If you did better
on some fanfiction corpus,

98
00:07:43,920 --> 00:07:47,277
it wouldn't generalize that well to other
tasks. We had all these measures. We had

99
00:07:47,360 --> 00:07:53,840
all these measures of how well it did at
predicting all these other kinds of texts.

100
00:07:55,840 --> 00:07:59,602
It was only when you trained over
all the tasks on the internet —

101
00:07:59,686 --> 00:08:01,437
when you did a general internet

102
00:08:01,520 --> 00:08:05,458
scrape from something like Common
Crawl or scraping links in Reddit,

103
00:08:05,541 --> 00:08:11,566
which is what we did for GPT-2 —
that you started to get generalization.

104
00:08:11,649 --> 00:08:15,757
I think we're seeing the same thing on RL.

105
00:08:15,840 --> 00:08:21,917
We're starting first with simple RL
tasks like training on math competitions,

106
00:08:22,000 --> 00:08:23,120
then moving to

107
00:08:24,640 --> 00:08:28,577
broader training that
involves things like code.

108
00:08:28,661 --> 00:08:31,677
Now we're moving to many other tasks.

109
00:08:31,760 --> 00:08:35,277
I think then we're going
to increasingly get generalization.

110
00:08:35,360 --> 00:08:39,837
So that kind of takes out the RL vs.
pre-training side of it.

111
00:08:39,920 --> 00:08:46,920
But there is a puzzle either way, which
is that in pre-training we use trillions

112
00:08:48,524 --> 00:08:49,600
of tokens.

113
00:08:50,640 --> 00:08:54,477
Humans don't see trillions of words.
So there is an actual sample

114
00:08:54,560 --> 00:08:59,837
efficiency difference here. There
is actually something different here.

115
00:08:59,920 --> 00:09:06,317
The models start from scratch
and they need much more training.

116
00:09:06,400 --> 00:09:13,400
But we also see that once they're trained,
if we give them a long context length of

117
00:09:15,600 --> 00:09:16,792
a million — the only thing
blocking long context is inference —

118
00:09:16,875 --> 00:09:17,197
they're very good at

119
00:09:17,280 --> 00:09:23,360
learning and adapting within that context.
So I don’t know the full answer to this.

120
00:09:24,080 --> 00:09:28,237
I think there's something going
on where pre-training is not like

121
00:09:28,320 --> 00:09:32,397
the process of humans learning, but it's
somewhere between the process of humans

122
00:09:32,480 --> 00:09:35,751
learning and the process
of human evolution.

123
00:09:35,834 --> 00:09:38,877
We get many of our priors from evolution.

124
00:09:38,960 --> 00:09:43,117
Our brain isn't just a blank slate.
Whole books have been written about this.

125
00:09:43,200 --> 00:09:45,357
The language models are
much more like blank slates.

126
00:09:45,440 --> 00:09:49,616
They literally start as random weights,
whereas the human brain starts with all

127
00:09:49,699 --> 00:09:53,321
these regions connected to all
these inputs and outputs.

128
00:09:53,404 --> 00:09:56,397
Maybe we should think
of pre-training — and

129
00:09:56,480 --> 00:10:02,477
for that matter, RL as well — as something
that exists in the middle space between

130
00:10:02,560 --> 00:10:06,641
human evolution and human
on-the-spot learning.

131
00:10:06,724 --> 00:10:10,717
And we should think
of the in-context learning

132
00:10:10,800 --> 00:10:16,297
that the models do as something between
long-term human learning and short-term

133
00:10:16,380 --> 00:10:19,928
human learning. So there's
this hierarchy. There’s evolution,

134
00:10:20,011 --> 00:10:22,557
there's long-term learning,
there's short-term

135
00:10:22,640 --> 00:10:28,637
learning, and there's just human reaction.
The LLM phases exist along this spectrum,

136
00:10:28,720 --> 00:10:31,860
but not necessarily
at exactly the same points.

137
00:10:31,943 --> 00:10:34,877
There’s no analog
to some of the human modes

138
00:10:34,960 --> 00:10:39,917
of learning the LLMs are falling in
between the points. Does that make sense?

139
00:10:40,000 --> 00:10:42,317
Yes, although some things
are still a bit confusing.

140
00:10:42,400 --> 00:10:45,757
For example, if the analogy is that this
is like evolution so it's fine that it's

141
00:10:45,840 --> 00:10:51,437
not sample efficient, then if we're
going to get super sample-efficient

142
00:10:51,520 --> 00:10:56,064
agent from in-context learning,
why are we bothering to build all these RL

143
00:10:56,147 --> 00:10:56,877
environments?

144
00:10:56,960 --> 00:11:00,317
There are companies whose work seems
to be teaching models how to use this API,

145
00:11:00,400 --> 00:11:04,094
how to use Slack, how to use whatever.
It's confusing to me why there's so much

146
00:11:04,177 --> 00:11:04,477
emphasis

147
00:11:04,560 --> 00:11:09,522
on that if the kind of agent that can
just learn on the fly is emerging or has

148
00:11:09,605 --> 00:11:10,640
already emerged.

149
00:11:11,600 --> 00:11:14,338
I can't speak for the
emphasis of anyone else.

150
00:11:14,421 --> 00:11:17,120
I can only talk about
how we think about it.

151
00:11:20,720 --> 00:11:25,037
The goal is not to teach the model
every possible skill within RL,

152
00:11:25,120 --> 00:11:28,251
just as we don't do that within
pre-training. Within pre-training,

153
00:11:28,334 --> 00:11:29,517
we're not trying to expose

154
00:11:29,600 --> 00:11:35,600
the model to every possible way
that words could be put together.

155
00:11:37,280 --> 00:11:41,997
Rather, the model trains on a lot
of things and then reaches generalization

156
00:11:42,080 --> 00:11:43,360
across pre-training.

157
00:11:43,920 --> 00:11:50,183
That was the transition from GPT-1
to GPT-2 that I saw up close.

158
00:11:50,267 --> 00:11:53,357
The model reaches a point. I had

159
00:11:53,440 --> 00:12:00,440
these moments where I was like, "Oh yeah,
you just give the model a list of numbers

160
00:12:00,657 --> 00:12:04,203
— this is the cost of the house,
this is the square feet of the house —

161
00:12:04,287 --> 00:12:05,837
and the model completes the pattern

162
00:12:05,920 --> 00:12:08,397
and does linear regression."
Not great, but it does it,

163
00:12:08,480 --> 00:12:12,200
and it's never seen
that exact thing before.

164
00:12:12,283 --> 00:12:15,917
So to the extent that
we are building these

165
00:12:16,000 --> 00:12:23,000
RL environments, the goal is very similar
to what was done five or ten years ago

166
00:12:23,118 --> 00:12:24,720
with pre-training.

167
00:12:26,560 --> 00:12:28,988
We're trying to get a whole bunch of data,

168
00:12:29,071 --> 00:12:32,157
not because we want to cover
a specific document or a

169
00:12:32,240 --> 00:12:35,917
specific skill, but because
we want to generalize.

170
00:12:36,000 --> 00:12:39,677
I think the framework
you're laying down obviously

171
00:12:39,760 --> 00:12:42,831
makes sense.
We're making progress toward AGI.

172
00:12:42,914 --> 00:12:45,917
Nobody at this point
disagrees we're going to

173
00:12:46,000 --> 00:12:49,437
achieve AGI this century.
The crux is you say we're

174
00:12:49,520 --> 00:12:54,320
hitting the end of the exponential.
Somebody else looks at this and says,

175
00:12:55,920 --> 00:12:59,642
"We've been making progress since 2012,
and by 2035 we'll have a human-like

176
00:12:59,725 --> 00:13:00,558
agent."

177
00:13:04,800 --> 00:13:07,277
Obviously we’re seeing in these models
the kinds of things that evolution did,

178
00:13:07,360 --> 00:13:09,528
or that learning within
a human lifetime does.

179
00:13:09,611 --> 00:13:11,437
I want to understand what you’re seeing

180
00:13:11,520 --> 00:13:14,670
that makes you think it's one
year away and not ten years away.

181
00:13:17,680 --> 00:13:24,680
There are two claims you could make
here, one stronger and one weaker.

182
00:13:26,560 --> 00:13:33,560
Starting with the weaker claim,
when I first saw the scaling back in 2019,

183
00:13:35,120 --> 00:13:41,133
I wasn’t sure. This was a 50/50 thing.
I thought I saw something.

184
00:13:41,217 --> 00:13:43,197
My claim was that this

185
00:13:43,280 --> 00:13:50,280
was much more likely than anyone thinks.
Maybe there's a 50% chance this happens.

186
00:13:51,600 --> 00:13:55,362
On the basic hypothesis of, as you put it,

187
00:13:55,445 --> 00:14:00,397
within ten years we'll get
to what I call a "country of

188
00:14:00,480 --> 00:14:06,877
geniuses in a data center", I'm at 90% on
that. It's hard to go much higher than 90%

189
00:14:06,960 --> 00:14:12,835
because the world is so unpredictable.
Maybe the irreducible uncertainty puts us

190
00:14:12,919 --> 00:14:13,357
at 95%,

191
00:14:13,440 --> 00:14:20,440
where you get to things like multiple
companies having internal turmoil,

192
00:14:20,953 --> 00:14:23,040
Taiwan gets invaded,

193
00:14:24,160 --> 00:14:29,997
all the fabs get blown up by missiles.
Now you've jinxed us, Dario.

194
00:14:30,080 --> 00:14:37,080
You could construct a 5% world where
things get delayed for ten years.

195
00:14:43,920 --> 00:14:49,200
There's another 5% which is that I'm very
confident on tasks that can be verified.

196
00:14:50,160 --> 00:14:53,200
With coding, except
for that irreducible uncertainty,

197
00:14:54,400 --> 00:14:56,093
I think we'll be there
in one or two years.

198
00:14:56,177 --> 00:14:58,077
There's no way we will
not be there in ten years

199
00:14:58,160 --> 00:15:02,041
in terms of being able
to do end-to-end coding.

200
00:15:02,124 --> 00:15:05,837
My one little bit
of fundamental uncertainty,

201
00:15:05,920 --> 00:15:10,205
even on long timescales,
is about tasks that aren't verifiable:

202
00:15:10,288 --> 00:15:12,160
planning a mission to Mars;

203
00:15:14,320 --> 00:15:21,320
doing some fundamental scientific
discovery like CRISPR; writing a novel.

204
00:15:21,440 --> 00:15:27,517
It’s hard to verify those tasks.
I am almost certain we have a

205
00:15:27,600 --> 00:15:33,120
reliable path to get there, but if there's
a little bit of uncertainty it's there.

206
00:15:34,320 --> 00:15:39,600
On the ten-year timeline I'm at 90%,
which is about as certain as you can be.

207
00:15:40,400 --> 00:15:46,157
I think it's crazy to say
that this won't happen by 2035.

208
00:15:46,240 --> 00:15:48,717
In some sane world, it would
be outside the mainstream.

209
00:15:48,800 --> 00:15:55,800
But the emphasis on verification hints to
me a lack of belief that these models are

210
00:15:57,467 --> 00:15:58,637
generalized.

211
00:15:58,720 --> 00:16:03,251
If you think about humans, we're both
good at things for which we get verifiable

212
00:16:03,334 --> 00:16:07,837
reward and things for which we don't.
No, this is why I’m almost sure.

213
00:16:07,920 --> 00:16:12,557
We already see substantial generalization
from things that verify to things that

214
00:16:12,640 --> 00:16:15,357
don't. We're already seeing that.
But it seems like you were emphasizing

215
00:16:15,440 --> 00:16:20,221
this as a spectrum which will split
apart which domains in which we see more

216
00:16:20,304 --> 00:16:21,137
progress.

217
00:16:21,600 --> 00:16:24,359
That doesn't seem like
how humans get better.

218
00:16:24,442 --> 00:16:27,517
The world in which we
don't get there is the world

219
00:16:27,600 --> 00:16:34,600
in which we do all the verifiable
things. Many of them generalize,

220
00:16:34,800 --> 00:16:39,997
but we don't fully get there.
We don’t fully color in the other side

221
00:16:40,080 --> 00:16:47,080
of the box. It's not a binary thing. Even
if generalization is weak and you can only

222
00:16:47,347 --> 00:16:49,237
do verifiable domains, it's not clear to
me you could automate software engineering

223
00:16:49,320 --> 00:16:53,548
in such a world. You are "a
software engineer" in some sense,

224
00:16:53,631 --> 00:16:58,397
but part of being a software
engineer for you involves

225
00:16:58,480 --> 00:17:00,888
writing long memos
about your grand vision.

226
00:17:00,971 --> 00:17:03,437
I don’t think that’s
part of the job of SWE.

227
00:17:03,520 --> 00:17:04,877
That's part of the job of the
company, not SWE specifically.

228
00:17:04,960 --> 00:17:09,280
But SWE does involve design
documents and other things like that.

229
00:17:10,400 --> 00:17:13,150
The models are already pretty
good at writing comments.

230
00:17:14,800 --> 00:17:20,780
Again, I’m making much weaker
claims here than I believe,

231
00:17:20,863 --> 00:17:24,397
to distinguish between two things.

232
00:17:24,480 --> 00:17:27,080
We're already almost there
for software engineering.

233
00:17:28,000 --> 00:17:31,334
By what metric? There's one metric which
is how many lines of code are written

234
00:17:31,417 --> 00:17:32,250
by AI.

235
00:17:32,800 --> 00:17:36,227
If you consider other productivity
improvements in the history of software

236
00:17:36,311 --> 00:17:38,813
engineering, compilers write
all the lines of software.

237
00:17:38,896 --> 00:17:40,877
There's a difference
between how many lines

238
00:17:40,960 --> 00:17:44,566
are written and how big
the productivity improvement is.

239
00:17:44,649 --> 00:17:46,560
"We’re almost there" meaning…

240
00:17:47,120 --> 00:17:51,270
How big is the productivity improvement,
not just how many lines are written by AI?

241
00:17:52,000 --> 00:17:57,037
I actually agree with you on this.
I've made a series of predictions on

242
00:17:57,120 --> 00:18:02,000
code and software engineering. I think
people have repeatedly misunderstood them.

243
00:18:03,920 --> 00:18:09,437
Let me lay out the spectrum.
About eight or nine months ago,

244
00:18:09,520 --> 00:18:16,520
I said the AI model will be writing 90% of
the lines of code in three to six months.

245
00:18:16,800 --> 00:18:21,597
That happened, at least at some places.
It happened at Anthropic, happened with

246
00:18:21,680 --> 00:18:27,437
many people downstream using our models.
But that's actually a very weak criterion.

247
00:18:27,520 --> 00:18:31,564
People thought I was saying that we
won't need 90% of the software engineers.

248
00:18:31,647 --> 00:18:32,797
Those things are worlds

249
00:18:32,880 --> 00:18:37,943
apart. The spectrum is:
90% of code is written by the model,

250
00:18:38,026 --> 00:18:41,117
100% of code is written by the model.

251
00:18:41,200 --> 00:18:46,823
That's a big difference in productivity.
90% of the end-to-end SWE tasks —

252
00:18:46,906 --> 00:18:47,517
including

253
00:18:47,600 --> 00:18:54,077
things like compiling, setting up clusters
and environments, testing features,

254
00:18:54,160 --> 00:19:01,160
writing memos — are done by the models.
100% of today's SWE tasks are done by

255
00:19:01,440 --> 00:19:02,397
the models.

256
00:19:02,480 --> 00:19:05,997
Even when that happens, it doesn't mean
software engineers are out of a job.

257
00:19:06,080 --> 00:19:10,157
There are new higher-level things
they can do, where they can manage.

258
00:19:10,240 --> 00:19:15,677
Then further down the spectrum, there's
90% less demand for SWEs, which I think

259
00:19:15,760 --> 00:19:21,357
will happen but this is a spectrum.
I wrote about it in "The Adolescence

260
00:19:21,440 --> 00:19:25,040
of Technology" where I went through
this kind of spectrum with farming.

261
00:19:26,720 --> 00:19:29,120
I actually totally agree with you on that.

262
00:19:29,920 --> 00:19:32,077
These are very different
benchmarks from each other,

263
00:19:32,160 --> 00:19:34,786
but we're proceeding
through them super fast.

264
00:19:34,870 --> 00:19:37,760
Part of your vision
is that going from 90 to 100

265
00:19:38,670 --> 00:19:45,357
is going to happen fast, and that it
leads to huge productivity improvements.

266
00:19:45,440 --> 00:19:48,214
But what I notice is that even
in greenfield projects people start

267
00:19:48,297 --> 00:19:49,037
with Claude Code or

268
00:19:49,120 --> 00:19:54,557
something, people report starting a lot of
projects… Do we see in the world out there

269
00:19:54,640 --> 00:19:58,157
a renaissance of software, all these new
features that wouldn't exist otherwise?

270
00:19:58,240 --> 00:20:01,997
At least so far, it doesn't seem like
we see that. So that does make me wonder.

271
00:20:02,080 --> 00:20:09,080
Even if I never had to intervene with
Claude Code, the world is complicated.

272
00:20:09,200 --> 00:20:14,557
Jobs are complicated. Closing the loop on
self-contained systems, whether it’s just

273
00:20:14,640 --> 00:20:19,760
writing software or something, how much
broader gains would we see just from that?

274
00:20:20,400 --> 00:20:24,557
Maybe that should dilute our estimation
of the "country of geniuses".

275
00:20:24,640 --> 00:20:31,640
I simultaneously agree with you that it's
a reason why these things don't happen

276
00:20:34,311 --> 00:20:40,000
instantly, but at the same time,
I think the effect is gonna be very fast.

277
00:20:41,360 --> 00:20:46,957
You could have these two poles.
One is that AI is not going to make

278
00:20:47,040 --> 00:20:52,397
progress. It's slow. It's going to take
forever to diffuse within the economy.

279
00:20:52,480 --> 00:20:56,477
Economic diffusion has become one
of these buzzwords that's a reason why

280
00:20:56,560 --> 00:21:00,110
we're not going to make AI progress,
or why AI progress doesn't matter.

281
00:21:00,560 --> 00:21:04,560
The other axis is that we'll get recursive
self-improvement, the whole thing.

282
00:21:05,120 --> 00:21:07,770
Can't you just draw
an exponential line on the curve?

283
00:21:08,400 --> 00:21:15,400
We're going to have Dyson spheres around
the sun so many nanoseconds after we get

284
00:21:16,803 --> 00:21:17,757
recursive.

285
00:21:17,840 --> 00:21:23,277
I'm completely caricaturing the view here,
but there are these two extremes.

286
00:21:23,360 --> 00:21:29,198
But what we've seen from the beginning,
at least if you look within Anthropic,

287
00:21:29,282 --> 00:21:35,627
there's this bizarre 10x per year growth
in revenue that we've seen. So in 2023,

288
00:21:35,710 --> 00:21:42,424
it was zero to $100 million. In 2024, it
was $100 million to $1 billion. In 2025,

289
00:21:42,507 --> 00:21:45,600
it was $1 billion to $ 9-10 billion.

290
00:21:46,320 --> 00:21:48,502
You guys should have just bought a billion
dollars of your own products so you could

291
00:21:48,585 --> 00:21:49,419
just…

292
00:21:50,640 --> 00:21:53,997
And the first month of this year,
that exponential is...

293
00:21:54,080 --> 00:22:01,080
You would think it would slow down, but
we added another few billion to revenue

294
00:22:02,858 --> 00:22:04,080
in January.

295
00:22:05,600 --> 00:22:10,157
Obviously that curve can't go on forever.
The GDP is only so large.

296
00:22:10,240 --> 00:22:15,703
I would even guess that it
bends somewhat this year,

297
00:22:15,787 --> 00:22:20,077
but that is a fast curve. That's a really

298
00:22:20,160 --> 00:22:25,173
fast curve. I would bet it stays pretty
fast even as the scale goes to the entire

299
00:22:25,257 --> 00:22:32,003
economy. So I think we should be thinking
about this middle world where things are

300
00:22:32,087 --> 00:22:33,997
extremely fast, but not

301
00:22:34,080 --> 00:22:37,523
instant, where they take time
because of economic diffusion,

302
00:22:37,607 --> 00:22:39,757
because of the need to close the loop.

303
00:22:39,840 --> 00:22:46,840
Because it's fiddly: "I have to do
change management within my enterprise…

304
00:22:48,518 --> 00:22:50,077
I set this up,

305
00:22:50,160 --> 00:22:54,037
but I have to change the security
permissions on this in order to make it

306
00:22:54,120 --> 00:22:54,954
actually work…

307
00:22:55,600 --> 00:23:01,437
I had this old piece of software that
checks the model before it's compiled

308
00:23:01,520 --> 00:23:05,357
and released and I have to rewrite it.
Yes, the model can do that, but I have

309
00:23:05,440 --> 00:23:09,200
to tell the model to do that.
It has to take time to do that."

310
00:23:10,080 --> 00:23:17,080
So I think everything we've seen so far is
compatible with the idea that there's one

311
00:23:17,258 --> 00:23:20,202
fast exponential that's
the capability of the model.

312
00:23:20,286 --> 00:23:22,317
Then there's another fast exponential

313
00:23:22,400 --> 00:23:26,157
that's downstream of that, which is the
diffusion of the model into the economy.

314
00:23:26,240 --> 00:23:32,051
Not instant, not slow, much faster
than any previous technology,

315
00:23:32,134 --> 00:23:34,160
but it has its limits.

316
00:23:37,120 --> 00:23:41,809
When I look inside Anthropic, when I
look at our customers: fast adoption,

317
00:23:41,892 --> 00:23:45,437
but not infinitely fast.
Can I try a hot take on you? Yeah.

318
00:23:45,520 --> 00:23:48,573
I feel like diffusion
is cope that people say.

319
00:23:48,656 --> 00:23:51,437
When the model isn't able to do something,

320
00:23:51,520 --> 00:23:54,100
they're like, "oh,
but it's a diffusion issue."

321
00:23:54,183 --> 00:23:56,877
But then you should use
the comparison to humans.

322
00:23:56,960 --> 00:24:00,477
You would think that the inherent
advantages that AIs have would make

323
00:24:00,560 --> 00:24:01,677
diffusion a much easier

324
00:24:01,760 --> 00:24:05,997
problem for new AIs getting onboarded
than new humans getting onboarded.

325
00:24:06,080 --> 00:24:08,637
An AI can read your entire
Slack and your drive in minutes.

326
00:24:08,720 --> 00:24:12,077
They can share all the knowledge that the
other copies of the same instance have.

327
00:24:12,160 --> 00:24:14,397
You don't have this adverse selection
problem when you're hiring AI, so you

328
00:24:14,480 --> 00:24:17,203
can just hire copies of a vetted AI model.

329
00:24:17,287 --> 00:24:20,077
Hiring a human is so
much more of a hassle.

330
00:24:20,160 --> 00:24:23,357
People hire humans all the time.
We pay humans upwards of $50 trillion

331
00:24:23,440 --> 00:24:28,513
in wages because they're useful,
even though in principle it would be much

332
00:24:28,596 --> 00:24:32,612
easier to integrate AIs into
the economy than it is to hire humans.

333
00:24:32,695 --> 00:24:34,797
The diffusion doesn't really explain.

334
00:24:34,880 --> 00:24:41,757
I think diffusion is very real
and doesn't exclusively have

335
00:24:41,840 --> 00:24:48,840
to do with limitations on the AI models.
Again, there are people who use diffusion

336
00:24:49,760 --> 00:24:54,157
as kind of a buzzword to say this isn't a
big deal. I'm not talking about that. I'm

337
00:24:54,240 --> 00:24:57,997
not talking about how AI will diffuse
at the speed of previous technologies.

338
00:24:58,080 --> 00:25:02,491
I think AI will diffuse much faster
than previous technologies have,

339
00:25:02,574 --> 00:25:07,550
but not infinitely fast. I'll just give
an example of this. There's Claude Code.

340
00:25:07,634 --> 00:25:10,160
Claude Code is extremely easy to set up.

341
00:25:10,960 --> 00:25:14,077
If you're a developer, you can
just start using Claude Code.

342
00:25:14,160 --> 00:25:19,437
There is no reason why a developer at a
large enterprise should not be adopting

343
00:25:19,520 --> 00:25:25,197
Claude Code as quickly as an individual
developer or developer at a startup.

344
00:25:25,280 --> 00:25:31,037
We do everything we can to promote it.
We sell Claude Code to enterprises.

345
00:25:31,120 --> 00:25:36,478
Big enterprises, big financial companies,
big pharmaceutical companies,

346
00:25:36,561 --> 00:25:38,317
all of them are adopting

347
00:25:38,400 --> 00:25:45,400
Claude Code much faster than enterprises
typically adopt new technology. But again,

348
00:25:46,320 --> 00:25:51,421
it takes time. Any given
feature or any given product,

349
00:25:51,504 --> 00:25:54,877
like Claude Code or Cowork, will get

350
00:25:54,960 --> 00:26:00,632
adopted by the individual developers
who are on Twitter all the time,

351
00:26:00,715 --> 00:26:02,717
by the Series A startups,

352
00:26:02,800 --> 00:26:09,800
many months faster than they will get
adopted by a large enterprise that does

353
00:26:10,080 --> 00:26:11,120
food sales.

354
00:26:11,840 --> 00:26:14,557
There are just a number of factors.
You have to go through legal,

355
00:26:14,640 --> 00:26:20,877
you have to provision it for everyone.
It has to pass security and compliance.

356
00:26:20,960 --> 00:26:25,613
The leaders of the company who are
further away from the AI revolution are

357
00:26:25,696 --> 00:26:26,637
forward-looking,

358
00:26:26,720 --> 00:26:31,037
but they have to say, "Oh, it makes
sense for us to spend 50 million.

359
00:26:31,120 --> 00:26:35,037
This is what this Claude Code thing is.
This is why it helps our company.

360
00:26:35,120 --> 00:26:37,597
This is why it makes us more productive."
Then they have to explain

361
00:26:37,680 --> 00:26:42,717
to the people two levels below. They have
to say, "Okay, we have 3,000 developers.

362
00:26:42,800 --> 00:26:45,117
Here's how we're going to roll
it out to our developers."

363
00:26:45,200 --> 00:26:50,397
We have conversations like this every day.
We are doing everything we can to make

364
00:26:50,480 --> 00:26:55,760
Anthropic's revenue grow 20 or 30x
a year instead of 10x a year.

365
00:26:57,440 --> 00:27:01,440
Again, many enterprises are just
saying, "This is so productive.

366
00:27:02,160 --> 00:27:05,037
We're going to take shortcuts
in our usual procurement process."

367
00:27:05,120 --> 00:27:08,477
They're moving much faster than
when we tried to sell them just

368
00:27:08,560 --> 00:27:13,677
the ordinary API, which many of them use.
Claude Code is a more compelling product,

369
00:27:13,760 --> 00:27:16,544
but it's not an infinitely
compelling product.

370
00:27:16,627 --> 00:27:19,037
I don't think even AGI or powerful AI or

371
00:27:19,120 --> 00:27:22,877
"country of geniuses in a data center"
will be an infinitely compelling product.

372
00:27:22,960 --> 00:27:26,777
It will be a compelling product
enough maybe to get 3-5x, or 10x,

373
00:27:26,860 --> 00:27:30,158
a year of growth, even when you're
in the hundreds of billions of dollars,

374
00:27:30,242 --> 00:27:31,997
which is extremely hard
to do and has never been

375
00:27:32,080 --> 00:27:34,456
done in history before,
but not infinitely fast.

376
00:27:34,539 --> 00:27:36,557
I buy that it would be a slight slowdown.

377
00:27:36,640 --> 00:27:39,357
Maybe this is not your claim,
but sometimes people talk about this like,

378
00:27:39,440 --> 00:27:43,033
"Oh, the capabilities are there,
but because of diffusion...

379
00:27:43,117 --> 00:27:45,200
otherwise we're basically at AGI".

380
00:27:46,160 --> 00:27:49,277
I don't believe we're basically at AGI.
I think if you had the "country

381
00:27:49,360 --> 00:27:53,757
of geniuses in a data center"...
If we had the "country of geniuses

382
00:27:53,840 --> 00:27:57,277
in a data center", we would know it.
We would know it if you had the

383
00:27:57,360 --> 00:28:01,517
"country of geniuses in a data center".
Everyone in this room would know it.

384
00:28:01,600 --> 00:28:07,677
Everyone in Washington would know it.
People in rural parts might not know it,

385
00:28:07,760 --> 00:28:14,760
but we would know it. We don't
have that now. That is very clear.

386
00:29:42,400 --> 00:29:46,432
Coming back to concrete prediction…
Because there are so many different things

387
00:29:46,516 --> 00:29:47,277
to disambiguate,

388
00:29:47,360 --> 00:29:50,797
it can be easy to talk past each other
when we're talking about capabilities.

389
00:29:50,880 --> 00:29:53,092
For example, when I
interviewed you three years ago,

390
00:29:53,175 --> 00:29:54,637
I asked you a prediction about what

391
00:29:54,720 --> 00:29:58,370
we should expect three years from now.
You were right. You said,

392
00:29:58,453 --> 00:30:00,237
"We should expect systems which,

393
00:30:00,320 --> 00:30:04,077
if you talk to them for the course of an
hour, it's hard to tell them apart from

394
00:30:04,160 --> 00:30:07,037
a generally well-educated human."
I think you were right about that.

395
00:30:07,120 --> 00:30:11,869
I think spiritually I feel unsatisfied
because my internal expectation was

396
00:30:11,953 --> 00:30:15,507
that such a system could automate
large parts of white-collar work.

397
00:30:15,591 --> 00:30:17,677
So it might be more
productive to talk about

398
00:30:17,760 --> 00:30:20,560
the actual end capabilities
you want from such a system.

399
00:30:21,520 --> 00:30:24,988
I will basically tell
you where I think we are.

400
00:30:25,071 --> 00:30:28,237
Let me ask a very
specific question so that

401
00:30:28,320 --> 00:30:32,077
we can figure out exactly what kinds of
capabilities we should think about soon.

402
00:30:32,160 --> 00:30:35,268
Maybe I'll ask about it in the
context of a job I understand well,

403
00:30:35,351 --> 00:30:36,477
not because it's the most

404
00:30:36,560 --> 00:30:40,437
relevant job, but just because I
can evaluate the claims about it.

405
00:30:40,520 --> 00:30:41,997
Take video editors. I have

406
00:30:42,080 --> 00:30:47,837
video editors. Part of their job involves
learning about our audience's preferences,

407
00:30:47,920 --> 00:30:50,717
learning about my preferences and tastes,
and the different trade-offs we have.

408
00:30:50,800 --> 00:30:54,950
They’re, over the course of many months,
building up this understanding of context.

409
00:30:55,600 --> 00:30:58,877
The skill and ability they have six
months into the job, a model that can

410
00:30:58,960 --> 00:31:03,760
pick up that skill on the job on the fly,
when should we expect such an AI system?

411
00:31:04,400 --> 00:31:08,031
I guess what you're talking about is
that we're doing this interview for three

412
00:31:08,114 --> 00:31:08,948
hours.

413
00:31:09,920 --> 00:31:11,837
Someone's going to come in,
someone's going to edit it.

414
00:31:11,920 --> 00:31:14,885
They're going to be like,
"Oh, I don't know,

415
00:31:14,968 --> 00:31:18,640
Dario scratched his head
and we could edit that out."

416
00:31:19,200 --> 00:31:22,397
"Magnify that." "There was this long

417
00:31:22,480 --> 00:31:24,863
discussion that is less
interesting to people.

418
00:31:24,947 --> 00:31:27,277
There's another thing
that's more interesting

419
00:31:27,360 --> 00:31:33,517
to people, so let's make this edit."
I think the "country of geniuses in

420
00:31:33,600 --> 00:31:38,088
a data center" will be able to do that.
The way it will be able to do that is it

421
00:31:38,171 --> 00:31:43,517
will have general control of a computer
screen. You'll be able to feed this in.

422
00:31:43,600 --> 00:31:46,896
It'll be able to also use
the computer screen to go on the web,

423
00:31:46,979 --> 00:31:48,320
look at all your previous

424
00:31:49,360 --> 00:31:53,917
interviews, look at what people are saying
on Twitter in response to your interviews,

425
00:31:54,000 --> 00:31:59,037
talk to you, ask you questions, talk to
your staff, look at the history of edits

426
00:31:59,120 --> 00:32:04,560
that you did, and from that, do the job. I
think that's dependent on several things.

427
00:32:06,720 --> 00:32:10,157
I think this is one of the things
that's actually blocking deployment:

428
00:32:10,240 --> 00:32:14,431
getting to the point on computer use where
the models are really masters at using

429
00:32:14,514 --> 00:32:15,347
the computer.

430
00:32:16,320 --> 00:32:20,797
We've seen this climb in benchmarks, and
benchmarks are always imperfect measures.

431
00:32:20,880 --> 00:32:27,880
But I think when we first released
computer use a year and a quarter ago,

432
00:32:30,474 --> 00:32:33,677
OSWorld was at maybe 15%.

433
00:32:33,760 --> 00:32:38,400
I don't remember exactly,
but we've climbed from that to 65-70%.

434
00:32:40,240 --> 00:32:45,296
There may be harder measures as well, but
I think computer use has to pass a point

435
00:32:45,380 --> 00:32:49,480
of reliability. Can I just follow up on
that before you move on to the next point?

436
00:32:50,000 --> 00:32:54,077
For years, I've been trying to build
different internal LLM tools for myself.

437
00:32:54,160 --> 00:32:59,517
Often I have these text-in, text-out
tasks, which should be dead center

438
00:32:59,600 --> 00:33:03,120
in the repertoire of these models.
Yet I still hire humans to do them.

439
00:33:03,760 --> 00:33:07,597
If it's something like, "identify what the
best clips would be in this transcript",

440
00:33:07,680 --> 00:33:10,266
maybe the LLMs do
a seven-out-of-ten job on them.

441
00:33:10,349 --> 00:33:12,717
But there's not this
ongoing way I can engage

442
00:33:12,800 --> 00:33:16,317
with them to help them get better at the
job the way I could with a human employee.

443
00:33:16,400 --> 00:33:20,317
That missing ability, even if you
solve computer use, would still block

444
00:33:20,400 --> 00:33:24,117
my ability to offload
an actual job to them.

445
00:33:24,200 --> 00:33:27,917
This gets back to what
we were talking about

446
00:33:28,000 --> 00:33:31,498
before with learning on the job.
It's very interesting.

447
00:33:31,581 --> 00:33:33,600
I think with the coding agents,

448
00:33:34,160 --> 00:33:38,795
I don't think people would say that
learning on the job is what is preventing

449
00:33:38,878 --> 00:33:39,837
the coding agents

450
00:33:39,920 --> 00:33:46,077
from doing everything end to end. They
keep getting better. We have engineers

451
00:33:46,160 --> 00:33:51,437
at Anthropic who don't write any code.
When I look at the productivity, to your

452
00:33:51,520 --> 00:33:56,162
previous question, we have folks
who say, "This GPU kernel, this chip,

453
00:33:56,245 --> 00:33:57,917
I used to write it myself.

454
00:33:58,000 --> 00:34:03,040
I just have Claude do it." There's
this enormous improvement in productivity.

455
00:34:04,560 --> 00:34:11,560
When I see Claude Code, familiarity with
the codebase or a feeling that the model

456
00:34:13,120 --> 00:34:17,531
hasn't worked at the company for a year,
that's not high up on the list

457
00:34:17,614 --> 00:34:18,797
of complaints I see.

458
00:34:18,880 --> 00:34:22,557
I think what I'm saying is that we're
kind of taking a different path.

459
00:34:22,640 --> 00:34:23,917
Don't you think with coding
that's because there

460
00:34:24,000 --> 00:34:28,797
is an external scaffold of memory which
exists instantiated in the codebase?

461
00:34:28,880 --> 00:34:31,037
I don't know how many
other jobs have that.

462
00:34:31,120 --> 00:34:33,277
Coding made fast
progress precisely because

463
00:34:33,360 --> 00:34:37,517
it has this unique advantage
that other economic activity doesn't.

464
00:34:37,600 --> 00:34:43,386
But when you say that, what you're
implying is that by reading the codebase

465
00:34:43,470 --> 00:34:47,997
into the context, I have everything
that the human needed to learn on the job.

466
00:34:48,080 --> 00:34:52,436
So that would be an example
of—whether it's written or not,

467
00:34:52,519 --> 00:34:54,317
whether it's available or

468
00:34:54,400 --> 00:34:59,360
not—a case where everything you needed
to know you got from the context window.

469
00:35:00,640 --> 00:35:02,853
What we think of as learning—"I
started this job,

470
00:35:02,937 --> 00:35:05,197
it's going to take me six
months to understand the

471
00:35:05,280 --> 00:35:07,335
code base"—the model just
did it in the context.

472
00:35:07,418 --> 00:35:09,117
I honestly don't know how to think about

473
00:35:09,200 --> 00:35:13,920
this because there are people who
qualitatively report what you're saying.

474
00:35:16,080 --> 00:35:20,473
I'm sure you saw last year, there was
a major study where they had experienced

475
00:35:20,557 --> 00:35:26,547
developers try to close pull requests in
repositories that they were familiar with.

476
00:35:26,630 --> 00:35:28,717
Those developers reported an

477
00:35:28,800 --> 00:35:31,597
uplift. They reported that they felt more
productive with the use of these models.

478
00:35:31,680 --> 00:35:34,957
But in fact, if you look at their output
and how much was actually merged back in,

479
00:35:35,040 --> 00:35:37,117
there was a 20% downlift.
They were less productive

480
00:35:37,200 --> 00:35:40,317
as a result of using these models.
So I'm trying to square the qualitative

481
00:35:40,400 --> 00:35:44,557
feeling that people feel with these
models versus, 1) in a macro level,

482
00:35:44,640 --> 00:35:48,557
where is this renaissance of software? And
then 2) when people do these independent

483
00:35:48,640 --> 00:35:53,277
evaluations, why are we not seeing
the productivity benefits we would expect?

484
00:35:53,360 --> 00:35:56,235
Within Anthropic,
this is just really unambiguous.

485
00:35:56,318 --> 00:35:58,957
We're under an incredible
amount of commercial

486
00:35:59,040 --> 00:36:02,807
pressure and make it even harder for
ourselves because we have all this safety

487
00:36:02,890 --> 00:36:03,597
stuff we do that

488
00:36:03,680 --> 00:36:10,680
I think we do more than other companies.
The pressure to survive economically

489
00:36:11,280 --> 00:36:14,562
while also keeping
our values is just incredible.

490
00:36:14,646 --> 00:36:18,080
We're trying to keep
this 10x revenue curve going.

491
00:36:18,800 --> 00:36:23,277
There is zero time for bullshit.
There is zero time for feeling

492
00:36:23,360 --> 00:36:29,840
like we're productive when we're not.
These tools make us a lot more productive.

493
00:36:30,960 --> 00:36:33,917
Why do you think we're concerned
about competitors using the tools?

494
00:36:34,000 --> 00:36:38,677
Because we think we're
ahead of the competitors.

495
00:36:38,760 --> 00:36:43,437
We wouldn't be going
through all this trouble if

496
00:36:43,520 --> 00:36:46,773
this were secretly
reducing our productivity.

497
00:36:46,857 --> 00:36:49,517
We see the end productivity every few

498
00:36:49,600 --> 00:36:54,237
months in the form of model launches.
There's no kidding yourself about this.

499
00:36:54,320 --> 00:37:00,557
The models make you more productive. 1)
People feeling like they're productive is

500
00:37:00,640 --> 00:37:02,717
qualitatively predicted
by studies like this.

501
00:37:02,800 --> 00:37:04,637
But 2) if I just look at the end output,

502
00:37:04,720 --> 00:37:07,660
obviously you guys are
making fast progress.

503
00:37:07,743 --> 00:37:10,477
But the idea was supposed to be that with

504
00:37:10,560 --> 00:37:14,077
recursive self-improvement, you make
a better AI, the AI helps you build a

505
00:37:14,160 --> 00:37:18,797
better next AI, et cetera, et cetera.
What I see instead—if I look at you,

506
00:37:18,880 --> 00:37:22,537
OpenAI, DeepMind—is that people are
just shifting around the podium every few

507
00:37:22,620 --> 00:37:25,037
months. Maybe you think that stops
because you've won or whatever.

508
00:37:25,120 --> 00:37:31,517
But why are we not seeing the person with
the best coding model have this lasting

509
00:37:31,600 --> 00:37:35,631
advantage if in fact there are these
enormous productivity gains from the last

510
00:37:35,714 --> 00:37:36,548
coding model.

511
00:37:38,160 --> 00:37:44,324
I think my model of the situation is
that there's an advantage that's gradually

512
00:37:44,407 --> 00:37:50,957
growing. I would say right now the coding
models give maybe, I don't know,

513
00:37:51,040 --> 00:37:58,040
a 15-20% total factor speed up. That's
my view. Six months ago, it was maybe 5%.

514
00:38:01,280 --> 00:38:05,392
So it didn't matter. 5% doesn't register.
It's now just getting to the point where

515
00:38:05,475 --> 00:38:06,309
it's

516
00:38:06,400 --> 00:38:11,680
one of several factors that kind of
matters. That's going to keep speeding up.

517
00:38:12,400 --> 00:38:18,397
I think six months ago, there were several
companies that were at roughly the same

518
00:38:18,480 --> 00:38:21,764
point because this wasn't
a notable factor,

519
00:38:21,848 --> 00:38:25,837
but I think it's starting
to speed up more and more.

520
00:38:25,920 --> 00:38:31,188
I would also say there are multiple
companies that write models that are used

521
00:38:31,271 --> 00:38:32,717
for code and we're not

522
00:38:32,800 --> 00:38:39,398
perfectly good at preventing some of these
other companies from using our models

523
00:38:39,481 --> 00:38:40,400
internally.

524
00:38:41,120 --> 00:38:48,120
So I think everything we're seeing
is consistent with this kind of snowball

525
00:38:48,898 --> 00:38:49,731
model.

526
00:38:52,000 --> 00:38:58,778
Again, my theme in all of this is
all of this is soft takeoff, soft,

527
00:38:58,862 --> 00:39:03,290
smooth exponentials, although
the exponentials are relatively steep.

528
00:39:03,373 --> 00:39:05,677
So we're seeing this
snowball gather momentum

529
00:39:05,760 --> 00:39:12,760
where it's like 10%, 20%, 25%, 40%.
As you go, Amdahl's law, you have

530
00:39:13,120 --> 00:39:17,357
to get all the things that are preventing
you from closing the loop out of the way.

531
00:39:17,440 --> 00:39:20,960
But this is one of the biggest
priorities within Anthropic.

532
00:39:22,560 --> 00:39:27,565
Stepping back, before in the stack
we were talking about when do we get

533
00:39:27,648 --> 00:39:29,357
this on-the-job learning?

534
00:39:29,440 --> 00:39:32,797
It seems like the point you were making
on the coding thing is that we actually

535
00:39:32,880 --> 00:39:35,997
don't need on-the-job learning.
You can have tremendous productivity

536
00:39:36,080 --> 00:39:39,498
improvements, you can have potentially
trillions of dollars of revenue for AI

537
00:39:39,581 --> 00:39:40,317
companies, without

538
00:39:40,400 --> 00:39:45,727
this basic human ability to learn
on the job. Maybe that's not your claim,

539
00:39:45,811 --> 00:39:47,117
you should clarify.

540
00:39:47,200 --> 00:39:52,157
But in most domains of economic activity,
people say, "I hired somebody,

541
00:39:52,240 --> 00:39:53,837
they weren't that useful

542
00:39:53,920 --> 00:39:57,403
for the first few months, and then
over time they built up the context,

543
00:39:57,487 --> 00:40:00,077
understanding." It's actually hard
to define what we're talking about here.

544
00:40:00,160 --> 00:40:04,634
But they got something and then now
they're a powerhorse and they're so

545
00:40:04,717 --> 00:40:09,144
valuable to us. If AI doesn't develop
this ability to learn on the fly,

546
00:40:09,227 --> 00:40:11,917
I'm a bit skeptical
that we're going to see

547
00:40:12,000 --> 00:40:16,512
huge changes to the world without
that ability. I think two things here.

548
00:40:16,595 --> 00:40:21,917
There's the state of the technology right
now. Again, we have these two stages.

549
00:40:22,000 --> 00:40:27,399
We have the pre-training and RL stage
where you throw a bunch of data and tasks

550
00:40:27,482 --> 00:40:31,517
into the models and then they generalize.
So it's like learning, but it's like

551
00:40:31,600 --> 00:40:38,397
learning from more data and not learning
over one human or one model's lifetime.

552
00:40:38,480 --> 00:40:42,477
So again, this is situated between
evolution and human learning.

553
00:40:42,560 --> 00:40:45,597
But once you learn all
those skills, you have them.

554
00:40:45,680 --> 00:40:50,060
Just like with pre-training,
just how the models know more,

555
00:40:50,143 --> 00:40:52,557
if I look at a pre-trained model,

556
00:40:52,640 --> 00:40:55,837
it knows more about the history
of samurai in Japan than I do.

557
00:40:55,920 --> 00:41:02,920
It knows more about baseball than I do.
It knows more about low-pass filters

558
00:41:03,520 --> 00:41:08,877
and electronics, all of these things.
Its knowledge is way broader than mine.

559
00:41:08,960 --> 00:41:15,960
So I think even just that may get us
to the point where the models are better

560
00:41:16,745 --> 00:41:22,359
at everything. We also have, again, just
with scaling the kind of existing setup,

561
00:41:22,443 --> 00:41:23,917
the in-context learning.

562
00:41:24,000 --> 00:41:27,437
I would describe it as kind
of like human on-the-job learning,

563
00:41:27,520 --> 00:41:30,144
but a little weaker
and a little short term.

564
00:41:30,228 --> 00:41:33,037
You look at in-context
learning and if you give

565
00:41:33,120 --> 00:41:35,212
the model a bunch
of examples it does get it.

566
00:41:35,296 --> 00:41:37,596
There's real learning
that happens in context.

567
00:41:38,160 --> 00:41:42,160
A million tokens is a lot.
That can be days of human learning.

568
00:41:42,880 --> 00:41:49,880
If you think about the model reading
a million words, how long would it

569
00:41:50,400 --> 00:41:57,117
take me to read a million? Days or weeks
at least. So you have these two things.

570
00:41:57,200 --> 00:42:00,800
I think these two things within
the existing paradigm may just be enough

571
00:42:00,884 --> 00:42:01,517
to get you the

572
00:42:01,600 --> 00:42:04,717
"country of geniuses in a data center".
I don't know for sure, but I think

573
00:42:04,800 --> 00:42:08,819
they're going to get you a large
fraction of it. There may be gaps,

574
00:42:08,902 --> 00:42:10,717
but I certainly think that just

575
00:42:10,800 --> 00:42:16,308
as things are, this is enough to generate
trillions of dollars of revenue.

576
00:42:16,391 --> 00:42:17,517
That's one. Two,

577
00:42:17,600 --> 00:42:23,011
is this idea of continual learning,
this idea of a single model learning on

578
00:42:23,094 --> 00:42:23,927
the job.

579
00:42:24,320 --> 00:42:29,197
I think we're working on that too.
There's a good chance that in the next

580
00:42:29,280 --> 00:42:36,280
year or two, we also solve that.
Again, I think you get most

581
00:42:36,800 --> 00:42:43,800
of the way there without it.
The trillions of dollars a year market,

582
00:42:45,280 --> 00:42:48,447
maybe all of the national security
implications and the safety implications

583
00:42:48,530 --> 00:42:54,328
that I wrote about in "Adolescence of
Technology" can happen without it. But we,

584
00:42:54,411 --> 00:42:57,757
and I imagine others, are working on it.

585
00:42:57,840 --> 00:43:02,957
There's a good chance that we will
get there within the next year or two.

586
00:43:03,040 --> 00:43:06,540
There are a bunch of ideas.
I won't go into all of them in detail, but

587
00:43:07,040 --> 00:43:10,797
one is just to make the context longer.
There's nothing preventing

588
00:43:10,880 --> 00:43:14,317
longer contexts from working.
You just have to train at longer contexts

589
00:43:14,400 --> 00:43:16,451
and then learn to serve them at inference.

590
00:43:16,535 --> 00:43:18,637
Both of those are
engineering problems that

591
00:43:18,720 --> 00:43:22,320
we are working on and I would assume
others are working on them as well.

592
00:43:22,480 --> 00:43:25,997
This context length increase, it seemed
like there was a period from 2020 to 2023

593
00:43:26,080 --> 00:43:30,942
where from GPT-3 to GPT-4 Turbo, there
was an increase from 2000 context lengths

594
00:43:31,025 --> 00:43:31,437
to 128K.

595
00:43:31,520 --> 00:43:36,957
I feel like for the two-ish years since
then, we've been in the same-ish ballpark.

596
00:43:37,040 --> 00:43:41,357
When context lengths get much longer
than that, people report qualitative

597
00:43:41,440 --> 00:43:45,600
degradation in the ability of the
model to consider that full context.

598
00:43:47,120 --> 00:43:49,707
So I'm curious what you're internally
seeing that makes you think,

599
00:43:49,790 --> 00:43:50,557
"10 million contexts,

600
00:43:50,640 --> 00:43:54,397
100 million contexts to get six months
of human learning and building context".

601
00:43:54,480 --> 00:43:58,397
This isn't a research problem. This is
an engineering and inference problem.

602
00:43:58,480 --> 00:44:03,280
If you want to serve long context,
you have to store your entire KV cache.

603
00:44:06,560 --> 00:44:11,517
It's difficult to store all the memory
in the GPUs, to juggle the memory around.

604
00:44:11,600 --> 00:44:15,197
I don't even know the details. At this
point, this is at a level of detail

605
00:44:15,280 --> 00:44:20,031
that I'm no longer able to follow,
although I knew it in the GPT-3 era.

606
00:44:20,114 --> 00:44:24,318
"These are the weights, these are
the activations you have to store…"

607
00:44:24,401 --> 00:44:26,797
But these days the whole thing is flipped

608
00:44:26,880 --> 00:44:30,161
because we have MoE
models and all of that.

609
00:44:30,244 --> 00:44:34,000
Regarding this degradation
you're talking about,

610
00:44:34,640 --> 00:44:38,051
without getting too specific,
there's two things.

611
00:44:38,134 --> 00:44:41,117
There's the context
length you train at and

612
00:44:41,200 --> 00:44:45,357
there's a context length that you serve
at. If you train at a small context length

613
00:44:45,440 --> 00:44:49,437
and then try to serve at a long context
length, maybe you get these degradations.

614
00:44:49,520 --> 00:44:52,797
It's better than nothing, you might still
offer it, but you get these degradations.

615
00:44:52,880 --> 00:44:55,480
Maybe it's harder to train
at a long context length.

616
00:44:56,880 --> 00:45:01,040
I want to, at the same time,
ask about maybe some rabbit holes.

617
00:45:01,600 --> 00:45:04,477
Wouldn't you expect that if you had
to train on longer context length,

618
00:45:04,560 --> 00:45:09,291
that would mean that you're able to get
less samples in for the same amount

619
00:45:09,374 --> 00:45:10,207
of compute?

620
00:45:10,880 --> 00:45:14,397
Maybe it's not worth diving deep on that.
I want to get an answer to the

621
00:45:14,480 --> 00:45:20,557
bigger picture question.
I don't feel a preference

622
00:45:20,640 --> 00:45:25,037
for a human editor that's been working for
me for six months versus an AI that's been

623
00:45:25,120 --> 00:45:31,200
working with me for six months, what year
do you predict that that will be the case?

624
00:45:33,840 --> 00:45:37,251
My guess for that is there's a lot
of problems where basically we can do

625
00:45:37,335 --> 00:45:38,077
this when we have

626
00:45:38,160 --> 00:45:45,160
the "country of geniuses in a data
center". My picture for that,

627
00:45:45,375 --> 00:45:47,997
if you made me guess, is

628
00:45:48,080 --> 00:45:52,460
one to two years, maybe one to three
years. It's really hard to tell.

629
00:45:52,543 --> 00:45:54,077
I have a strong view—99%,

630
00:45:54,160 --> 00:45:59,920
95%—that all this will happen in 10 years.
I think that's just a super safe bet.

631
00:46:00,560 --> 00:46:04,637
I have a hunch—this is more like a 50/50
thing—that it's going to be more like

632
00:46:04,720 --> 00:46:07,867
one to two, maybe more like one
to three. So one to three years.

633
00:46:07,950 --> 00:46:08,960
Country of geniuses,

634
00:46:10,160 --> 00:46:14,237
and the slightly less economically
valuable task of editing videos.

635
00:46:14,320 --> 00:46:16,957
It seems pretty economically
valuable, let me tell you.

636
00:46:17,040 --> 00:46:19,134
It's just there are a lot
of use cases like that.

637
00:46:19,218 --> 00:46:20,557
There are a lot of similar ones.

638
00:46:20,640 --> 00:46:23,597
So you're predicting that
within one to three years.

639
00:46:23,680 --> 00:46:27,788
And then, generally, Anthropic has
predicted that by late '26 or early '27 we

640
00:46:27,871 --> 00:46:28,877
will have AI systems

641
00:46:28,960 --> 00:46:32,979
that "have the ability to navigate
interfaces available to humans doing

642
00:46:33,062 --> 00:46:34,077
digital work today,

643
00:46:34,160 --> 00:46:37,958
intellectual capabilities matching
or exceeding that of Nobel Prize winners,

644
00:46:38,041 --> 00:46:38,877
and the ability to

645
00:46:38,960 --> 00:46:42,317
interface with the physical world".
You gave an interview two months ago

646
00:46:42,400 --> 00:46:48,717
with DealBook where you were emphasizing
your company's more responsible compute

647
00:46:48,800 --> 00:46:52,700
scaling as compared to your competitors.
I'm trying to square these two views.

648
00:46:52,960 --> 00:46:57,517
If you really believe that we're going
to have a country of geniuses, you want as

649
00:46:57,600 --> 00:47:00,557
big a data center as you can get.
There's no reason to slow down.

650
00:47:00,640 --> 00:47:04,877
The TAM of a Nobel Prize winner, that can
actually do everything a Nobel Prize

651
00:47:04,960 --> 00:47:10,477
winner can do, is trillions of dollars.
So I'm trying to square this conservatism,

652
00:47:10,560 --> 00:47:13,933
which seems rational if you
have more moderate timelines,

653
00:47:14,016 --> 00:47:16,320
with your stated views about progress.

654
00:47:16,880 --> 00:47:20,170
It actually all fits together.
We go back to this fast,

655
00:47:20,253 --> 00:47:22,400
but not infinitely fast, diffusion.

656
00:47:23,120 --> 00:47:26,271
Let's say that we're making
progress at this rate.

657
00:47:26,354 --> 00:47:29,200
The technology is making
progress this fast.

658
00:47:29,920 --> 00:47:36,920
I have very high conviction that we're
going to get there within a few years.

659
00:47:39,040 --> 00:47:41,357
I have a hunch that we're going
to get there within a year or two.

660
00:47:41,440 --> 00:47:46,717
So there’s a little uncertainty
on the technical side, but pretty strong

661
00:47:46,800 --> 00:47:51,517
confidence that it won't be off by much.
What I'm less certain about is, again,

662
00:47:51,600 --> 00:47:56,557
the economic diffusion side.
I really do believe that we could

663
00:47:56,640 --> 00:48:03,640
have models that are a country of geniuses
in the data center in one to two years.

664
00:48:03,920 --> 00:48:10,920
One question is: How many years after that
do the trillions in revenue start rolling

665
00:48:13,034 --> 00:48:13,868
in?

666
00:48:14,160 --> 00:48:18,080
I don't think it's guaranteed
that it's going to be immediate.

667
00:48:19,440 --> 00:48:26,440
It could be one year, it could be two
years, I could even stretch it to five

668
00:48:27,120 --> 00:48:32,482
years although I'm skeptical of that.
So we have this uncertainty.

669
00:48:32,565 --> 00:48:34,957
Even if the technology goes as

670
00:48:35,040 --> 00:48:41,006
fast as I suspect that it will, we don't
know exactly how fast it's going to drive

671
00:48:41,090 --> 00:48:45,699
revenue. We know it's coming, but with
the way you buy these data centers,

672
00:48:45,782 --> 00:48:49,997
if you're off by a couple years,
that can be ruinous. It is just like how I

673
00:48:50,080 --> 00:48:54,850
wrote in "Machines of Loving Grace".
I said I think we might get this powerful

674
00:48:54,933 --> 00:48:57,837
AI, this "country of genius in the data
center". That description you gave comes

675
00:48:57,920 --> 00:49:02,717
from "Machines of Loving Grace". I said
we'll get that in 2026, maybe 2027. Again,

676
00:49:02,800 --> 00:49:06,840
that is my hunch. I wouldn't be
surprised if I'm off by a year or two,

677
00:49:06,923 --> 00:49:10,381
but that is my hunch. Let's say
that happens. That's the starting gun.

678
00:49:10,464 --> 00:49:12,814
How long does it take
to cure all the diseases?

679
00:49:13,360 --> 00:49:18,582
That's one of the ways that drives
a huge amount of economic value.

680
00:49:18,665 --> 00:49:21,037
You cure every disease. There's

681
00:49:21,120 --> 00:49:23,917
a question of how much of that goes to the
pharmaceutical company or the AI company,

682
00:49:24,000 --> 00:49:28,425
but there's an enormous consumer surplus
because —assuming we can get access

683
00:49:28,509 --> 00:49:29,197
for everyone,

684
00:49:29,280 --> 00:49:34,157
which I care about greatly—we cure all of
these diseases. How long does it take? You

685
00:49:34,240 --> 00:49:40,877
have to do the biological discovery,
you have to manufacture the new drug,

686
00:49:40,960 --> 00:49:43,255
you have to go through
the regulatory process.

687
00:49:43,339 --> 00:49:45,200
We saw this with vaccines and COVID.

688
00:49:47,680 --> 00:49:51,120
We got the vaccine out to everyone,
but it took a year and a half.

689
00:49:52,000 --> 00:49:57,461
My question is: How long does it take
to get the cure for everything—which AI

690
00:49:57,544 --> 00:49:58,397
is the genius

691
00:49:58,480 --> 00:50:02,957
that can in theory invent—out to everyone?
How long from when that AI first exists

692
00:50:03,040 --> 00:50:08,080
in the lab to when diseases have
actually been cured for everyone?

693
00:50:09,920 --> 00:50:14,477
We've had a polio vaccine for 50 years.
We're still trying to eradicate it in the

694
00:50:14,560 --> 00:50:18,077
most remote corners of Africa.
The Gates Foundation is trying

695
00:50:18,160 --> 00:50:19,997
as hard as they can.
Others are trying as hard

696
00:50:20,080 --> 00:50:25,011
as they can. But that's difficult.
Again, I don't expect most of the economic

697
00:50:25,094 --> 00:50:30,005
diffusion to be as difficult as that.
That's the most difficult case.

698
00:50:30,089 --> 00:50:32,397
But there's a real dilemma here.

699
00:50:32,480 --> 00:50:39,037
Where I've settled on it is that it will
be faster than anything we've seen in the

700
00:50:39,120 --> 00:50:45,920
world, but it still has its limits.
So when we go to buying data centers,

701
00:50:47,120 --> 00:50:53,997
again, the curve I'm looking at is: we've
had a 10x a year increase every year.

702
00:50:54,080 --> 00:51:00,125
At the beginning of this year, we're
looking at $10 billion in annualized

703
00:51:00,208 --> 00:51:01,042
revenue.

704
00:51:02,320 --> 00:51:06,378
We have to decide how much compute to buy.

705
00:51:06,461 --> 00:51:10,717
It takes a year or two
to actually build out

706
00:51:10,800 --> 00:51:16,877
the data centers, to reserve the data
center. Basically I'm saying, "In 2027,

707
00:51:16,960 --> 00:51:23,960
how much compute do I get?"
I could assume that the

708
00:51:24,720 --> 00:51:31,117
revenue will continue growing 10x a year,
so it'll be $100 billion at the end of

709
00:51:31,200 --> 00:51:38,200
2026 and $1 trillion at the end of 2027.
Actually it would be $5 trillion dollars

710
00:51:39,520 --> 00:51:43,517
of compute because it would be
$1 trillion a year for five years.

711
00:51:43,600 --> 00:51:48,480
I could buy $1 trillion of compute
that starts at the end of 2027.

712
00:51:49,200 --> 00:51:54,267
If my revenue is not $1 trillion
dollars, if it's even $800 billion,

713
00:51:54,351 --> 00:51:56,237
there's no force on earth,

714
00:51:56,320 --> 00:52:01,417
there's no hedge on earth that could stop
me from going bankrupt if I buy that much

715
00:52:01,501 --> 00:52:02,334
compute.

716
00:52:03,200 --> 00:52:07,197
Even though a part of my brain wonders
if it's going to keep growing 10x,

717
00:52:07,280 --> 00:52:12,207
I can't buy $1 trillion
a year of compute in 2027.

718
00:52:12,290 --> 00:52:17,117
If I'm just off by a year
in that rate of growth,

719
00:52:17,200 --> 00:52:23,760
or if the growth rate is 5x a year instead
of 10x a year, then you go bankrupt.

720
00:52:25,360 --> 00:52:31,010
So you end up in a world where you're
supporting hundreds of billions,

721
00:52:31,093 --> 00:52:32,240
not trillions.

722
00:52:33,120 --> 00:52:38,477
You accept some risk that there's so much
demand that you can't support the revenue,

723
00:52:38,560 --> 00:52:43,597
and you accept some risk that you
got it wrong and it's still slow.

724
00:52:43,680 --> 00:52:48,618
When I talked about behaving responsibly,
what I meant actually was not the absolute

725
00:52:48,702 --> 00:52:49,535
amount.

726
00:52:51,920 --> 00:52:55,677
I think it is true we're spending somewhat
less than some of the other players.

727
00:52:55,760 --> 00:53:00,578
It's actually the other things, like
have we been thoughtful about it or are we

728
00:53:00,661 --> 00:53:05,117
YOLOing and saying, "We're going to do
$100 billion here or $100 billion there"?

729
00:53:05,200 --> 00:53:09,597
I get the impression that some of the
other companies have not written down

730
00:53:09,680 --> 00:53:12,557
the spreadsheet, that they don't really
understand the risks they're taking.

731
00:53:12,640 --> 00:53:16,176
They're just doing stuff
because it sounds cool.

732
00:53:16,259 --> 00:53:19,117
We've thought carefully about it. We're

733
00:53:19,200 --> 00:53:23,835
an enterprise business. Therefore,
we can rely more on revenue.

734
00:53:23,918 --> 00:53:26,157
It's less fickle than consumer.

735
00:53:26,240 --> 00:53:30,784
We have better margins, which is the
buffer between buying too much and buying

736
00:53:30,867 --> 00:53:31,437
too little.

737
00:53:31,520 --> 00:53:37,117
I think we bought an amount that allows
us to capture pretty strong upside worlds.

738
00:53:37,200 --> 00:53:42,157
It won't capture the full 10x a year.
Things would have to go pretty badly for

739
00:53:42,240 --> 00:53:46,397
us to be in financial trouble.
So we've thought carefully and

740
00:53:46,480 --> 00:53:48,797
we've made that balance.
That's what I mean when

741
00:53:48,880 --> 00:53:53,997
I say that we're being responsible.
So it seems like it's possible that we

742
00:53:54,080 --> 00:53:56,609
actually just have different definitions
of the "country of a genius in a data

743
00:53:56,692 --> 00:53:59,479
center". Because when I think
of actual human geniuses,

744
00:53:59,562 --> 00:54:02,477
an actual country of human
geniuses in a data center,

745
00:54:02,560 --> 00:54:08,237
I would happily buy $5 trillion worth
of compute to run an actual country of

746
00:54:08,320 --> 00:54:11,277
human geniuses in a data center.
Let's say JPMorgan or Moderna or

747
00:54:11,360 --> 00:54:14,557
whatever doesn't want to use them.
I've got a country of geniuses.

748
00:54:14,640 --> 00:54:17,558
They'll start their own company. If they
can't start their own company and they're

749
00:54:17,641 --> 00:54:20,770
bottlenecked by clinical trials… It is
worth stating that with clinical trials,

750
00:54:20,853 --> 00:54:24,724
most clinical trials fail because the drug
doesn't work. There's not efficacy.

751
00:54:24,808 --> 00:54:27,117
I make exactly that point in "Machines of

752
00:54:27,200 --> 00:54:30,077
Loving Grace", I say the clinical
trials are going to go much faster

753
00:54:30,160 --> 00:54:33,332
than we're used to,
but not infinitely fast. Okay,

754
00:54:33,416 --> 00:54:35,677
and then suppose it takes a year for

755
00:54:35,760 --> 00:54:38,410
the clinical trials to work out so that
you're getting revenue from that and can

756
00:54:38,493 --> 00:54:42,643
make more drugs. Okay, well, you've got a
country of geniuses and you're an AI lab.

757
00:54:44,000 --> 00:54:48,800
You could use many more AI researchers.
You also think there are these

758
00:54:48,884 --> 00:54:49,917
self-reinforcing

759
00:54:50,000 --> 00:54:55,997
gains from smart people working on AI
tech. You can have the data center

760
00:54:56,080 --> 00:55:01,517
working on AI progress.
Are there substantially

761
00:55:01,600 --> 00:55:06,783
more gains from buying $1 trillion a year
of compute versus $300 billion a year

762
00:55:06,867 --> 00:55:09,677
of compute? If your competitor
is buying a trillion, yes there is.

763
00:55:09,760 --> 00:55:12,705
Well, no, there's some
gain, but then again,

764
00:55:12,788 --> 00:55:16,160
there's this chance that
they go bankrupt before.

765
00:55:17,040 --> 00:55:21,700
Again, if you're off by only
a year, you destroy yourselves.

766
00:55:21,784 --> 00:55:26,997
That's the balance. We're buying a lot.
We're buying a hell of a lot.

767
00:55:27,080 --> 00:55:30,317
We're buying an amount
that's comparable to

768
00:55:30,400 --> 00:55:36,648
what the biggest players in the game
are buying. But if you're asking me,

769
00:55:36,732 --> 00:55:38,640
"Why haven't we signed

770
00:55:39,520 --> 00:55:43,277
$10 trillion of compute starting
in mid-2027?"... First of all,

771
00:55:43,360 --> 00:55:44,557
it can't be produced.

772
00:55:44,640 --> 00:55:50,157
There isn't that much in the world.
But second, what if the country of

773
00:55:50,240 --> 00:55:55,997
geniuses comes, but it comes in mid-2028
instead of mid-2027? You go bankrupt.

774
00:55:56,080 --> 00:56:00,397
So if your projection is one to three
years, it seems like you should want

775
00:56:00,480 --> 00:56:06,080
$10 trillion of compute
by 2029 at the latest?

776
00:56:06,164 --> 00:56:11,517
Even in the longest
version of the timelines

777
00:56:11,600 --> 00:56:16,237
you state, the compute you are ramping
up to build doesn't seem in accordance.

778
00:56:16,320 --> 00:56:21,357
What makes you think that?
Human wages, let's say,

779
00:56:21,440 --> 00:56:26,246
are on the order of $50 trillion a year—
So I won't talk about Anthropic

780
00:56:26,329 --> 00:56:27,197
in particular,

781
00:56:27,280 --> 00:56:34,280
but if you talk about the industry, the
amount of compute the industry is building

782
00:56:37,050 --> 00:56:38,397
this year is

783
00:56:38,480 --> 00:56:45,480
probably, call it, 10-15 gigawatts.
It goes up by roughly 3x a year.

784
00:56:48,480 --> 00:56:54,797
So next year's 30-40 gigawatts.
2028 might be 100 gigawatts.

785
00:56:54,880 --> 00:56:58,400
2029 might be like 300 gigawatts.

786
00:57:03,760 --> 00:57:07,277
I'm doing the math in my head,
but each gigawatt costs maybe $10 billion,

787
00:57:07,360 --> 00:57:13,997
on the order of $10-15 billion a year.
You put that all together and you're

788
00:57:14,080 --> 00:57:15,514
getting about what you described.
You’re getting exactly that.

789
00:57:15,597 --> 00:57:16,077
You're getting multiple

790
00:57:16,160 --> 00:57:22,480
trillions a year by 2028 or 2029.
You're getting exactly what you predict.

791
00:57:23,040 --> 00:57:25,917
That's for the industry.
That's for the industry, that’s right.

792
00:57:26,000 --> 00:57:31,983
Suppose Anthropic's compute keeps
3x-ing a year, and then by 2027-28,

793
00:57:32,066 --> 00:57:34,000
you have 10 gigawatts.

794
00:57:34,960 --> 00:57:40,557
Multiply that by, as you say, $10 billion.
So then it's like $100 billion a year.

795
00:57:40,640 --> 00:57:43,267
But then you're saying the TAM
by 2028 is $200 billion.

796
00:57:43,350 --> 00:57:46,328
Again, I don't want to give
exact numbers for Anthropic,

797
00:57:46,411 --> 00:57:48,077
but these numbers are too small.

798
00:57:48,160 --> 00:57:55,160
Okay, interesting. You've told investors

799
00:58:49,920 --> 00:58:52,666
that you plan to be
profitable starting in 2028.

800
00:58:52,749 --> 00:58:55,437
This is the year when
we're potentially getting

801
00:58:55,520 --> 00:59:01,073
the country of geniuses as a data center.
This is now going to unlock all

802
00:59:01,156 --> 00:59:06,190
this progress in medicine
and health and new technologies.

803
00:59:06,273 --> 00:59:10,480
Wouldn't this be exactly
the time where you'd

804
00:59:11,040 --> 00:59:14,661
want to reinvest in the business and build
bigger "countries" so they can make more

805
00:59:14,744 --> 00:59:15,578
discoveries?

806
00:59:16,320 --> 00:59:19,280
Profitability is this kind
of weird thing in this field.

807
00:59:21,760 --> 00:59:28,760
I don't think in this field profitability
is actually a measure of spending down

808
00:59:32,000 --> 00:59:35,997
versus investing in the business.
Let's just take a model of this.

809
00:59:36,080 --> 00:59:39,729
I actually think profitability happens
when you underestimated the amount

810
00:59:39,813 --> 00:59:40,957
of demand you were going

811
00:59:41,040 --> 00:59:44,977
to get and loss happens when you
overestimated the amount of demand you

812
00:59:45,061 --> 00:59:49,984
were going to get, because you're
buying the data centers ahead of time.

813
00:59:50,068 --> 00:59:52,317
Think about it this way. Again,

814
00:59:52,400 --> 00:59:54,818
these are stylized facts.
These numbers are not exact.

815
00:59:54,901 --> 00:59:56,717
I'm just trying to make a toy model here.

816
00:59:56,800 --> 01:00:00,686
Let's say half of your compute is for
training and half of your compute is

817
01:00:00,769 --> 01:00:01,602
for inference.

818
01:00:02,640 --> 01:00:06,480
The inference has some gross
margin that's more than 50%.

819
01:00:07,200 --> 01:00:10,223
So what that means is that if
you were in steady-state,

820
01:00:10,306 --> 01:00:16,882
you build a data center and if you knew
exactly the demand you were getting,

821
01:00:16,966 --> 01:00:21,600
you would get a certain amount of revenue.

822
01:00:23,120 --> 01:00:28,637
Let’s say you pay $100 billion a year for
compute. On $50 billion a year you support

823
01:00:28,720 --> 01:00:35,120
$150 billion of revenue. The other
$50 billion is used for training.

824
01:00:36,240 --> 01:00:39,997
Basically you’re profitable
and you make $50 billion of profit.

825
01:00:40,080 --> 01:00:45,517
Those are the economics of the industry
today, or not today but where we’re

826
01:00:45,600 --> 01:00:49,037
projecting forward in a year or two.
The only thing that makes that not the

827
01:00:49,120 --> 01:00:53,443
case is if you get less
demand than $50 billion.

828
01:00:53,526 --> 01:00:57,757
Then you have more than
50% of your data center

829
01:00:57,840 --> 01:01:01,357
for research and you're not profitable.
So you train stronger models,

830
01:01:01,440 --> 01:01:07,037
but you're not profitable. If you
get more demand than you thought, then

831
01:01:07,120 --> 01:01:13,152
research gets squeezed, but you're kind of
able to support more inference and you're

832
01:01:13,235 --> 01:01:14,400
more profitable.

833
01:01:16,000 --> 01:01:18,957
Maybe I'm not explaining it well, but
the thing I'm trying to say is that you

834
01:01:19,040 --> 01:01:24,317
decide the amount of compute first.
Then you have some target desire of

835
01:01:24,400 --> 01:01:28,637
inference versus training,
but that gets determined by demand.

836
01:01:28,720 --> 01:01:30,637
It doesn't get determined by you.
What I'm hearing is the reason

837
01:01:30,720 --> 01:01:35,920
you're predicting profit is that you are
systematically underinvesting in compute?

838
01:01:37,040 --> 01:01:40,132
No, no, no. I'm saying
it's hard to predict.

839
01:01:40,216 --> 01:01:43,597
These things about 2028
and when it will happen,

840
01:01:43,680 --> 01:01:46,477
that's our attempt to do
the best we can with investors.

841
01:01:46,560 --> 01:01:50,397
All of this stuff is really uncertain
because of the cone of uncertainty.

842
01:01:50,480 --> 01:01:55,440
We could be profitable in 2026
if the revenue grows fast enough.

843
01:01:58,960 --> 01:02:03,040
If we overestimate or underestimate
the next year, that could swing wildly.

844
01:02:04,400 --> 01:02:08,834
What I'm trying to get at is that you
have a model in your head of a business

845
01:02:08,917 --> 01:02:13,917
that invests, invests, invests,
gets scale and then becomes profitable.

846
01:02:14,000 --> 01:02:16,157
There's a single point
at which things turn around.

847
01:02:16,240 --> 01:02:19,277
I don't think the economics
of this industry work that way.

848
01:02:19,360 --> 01:02:23,306
I see. So if I'm understanding correctly,
you're saying that because of

849
01:02:23,389 --> 01:02:24,157
the discrepancy

850
01:02:24,240 --> 01:02:27,128
between the amount of compute we should
have gotten and the amount of compute we

851
01:02:27,211 --> 01:02:30,637
got, we were sort of forced to make
profit. But that doesn't mean we're going

852
01:02:30,720 --> 01:02:33,437
to continue making profit.
We're going to reinvest the money

853
01:02:33,520 --> 01:02:37,757
because now AI has made so much progress
and we want a bigger country of geniuses.

854
01:02:37,840 --> 01:02:42,560
So back into revenue is high,
but losses are also high.

855
01:02:44,160 --> 01:02:48,119
If every year we predict exactly
what the demand is going to be,

856
01:02:48,202 --> 01:02:50,077
we'll be profitable every year.

857
01:02:50,160 --> 01:02:56,215
Because spending 50% of your
compute on research, roughly,

858
01:02:56,298 --> 01:03:00,237
plus a gross margin that's higher than

859
01:03:00,320 --> 01:03:03,711
50% and correct demand
prediction leads to profit.

860
01:03:03,795 --> 01:03:07,117
That's the profitable
business model that I think

861
01:03:07,200 --> 01:03:13,437
is kind of there, but obscured by these
building ahead and prediction errors.

862
01:03:13,520 --> 01:03:20,520
I guess you're treating the 50% as a sort
of given constant, whereas in fact,

863
01:03:21,040 --> 01:03:23,680
if AI progress is fast and you can
increase the progress by scaling up more,

864
01:03:23,763 --> 01:03:26,717
you should just have more than 50% and not
make profit. But here's what I'll say. You

865
01:03:26,800 --> 01:03:33,040
might want to scale it up more.
Remember the log returns to scale.

866
01:03:34,080 --> 01:03:40,480
If 70% would get you a very little
bit of a smaller model through a factor

867
01:03:40,564 --> 01:03:41,440
of 1.4x...

868
01:03:42,320 --> 01:03:49,296
That extra $20 billion, each dollar
there is worth much less to you because of

869
01:03:49,379 --> 01:03:51,197
the log-linear setup.

870
01:03:51,280 --> 01:03:58,280
So you might find that it's better
to invest that $20 billion in serving

871
01:03:58,960 --> 01:04:05,037
inference or in hiring engineers who
are kind of better at what they're doing.

872
01:04:05,120 --> 01:04:08,453
So the reason I said 50%...
That's not exactly our target.

873
01:04:08,536 --> 01:04:10,397
It's not exactly going to be 50%.

874
01:04:10,480 --> 01:04:16,520
It’ll probably vary over time.
What I'm saying is the log-linear return,

875
01:04:16,604 --> 01:04:18,477
what it leads to is you

876
01:04:18,560 --> 01:04:25,403
spend of order one fraction
of the business. Like not 5%, not 95%.

877
01:04:25,487 --> 01:04:28,867
Then you get diminishing returns.

878
01:04:28,950 --> 01:04:34,160
I feel strange that I'm convincing Dario
to believe in AI progress or something.

879
01:04:34,960 --> 01:04:37,837
Okay, you don't invest in research
because it has diminishing returns,

880
01:04:37,920 --> 01:04:38,762
but you invest in the other
things you mentioned.

881
01:04:38,845 --> 01:04:39,517
I think profit at a sort of macro level—

882
01:04:39,600 --> 01:04:42,357
Again, I'm talking about
diminishing returns,

883
01:04:42,440 --> 01:04:45,280
but after you're spending
$50 billion a year.

884
01:04:46,240 --> 01:04:51,597
This is a point I'm sure you would make,
but diminishing returns on a genius could

885
01:04:51,680 --> 01:04:53,997
be quite high. More generally,

886
01:04:54,080 --> 01:04:58,237
what is profit in a market economy?
Profit is basically saying other

887
01:04:58,320 --> 01:05:01,997
companies in the market can do more
things with this money than I can.

888
01:05:02,080 --> 01:05:06,477
Put aside Anthropic. I don't want
to give information about Anthropic.

889
01:05:06,560 --> 01:05:09,997
That’s why I'm giving these stylized
numbers. But let's just derive the

890
01:05:10,080 --> 01:05:17,080
equilibrium of the industry.
Why doesn't everyone spend 100% of their

891
01:05:21,040 --> 01:05:23,294
compute on training
and not serve any customers?

892
01:05:23,377 --> 01:05:25,437
It's because if they
didn't get any revenue,

893
01:05:25,520 --> 01:05:27,357
they couldn't raise money,
they couldn't do compute deals,

894
01:05:27,440 --> 01:05:29,463
they couldn't buy more
compute the next year.

895
01:05:29,546 --> 01:05:31,757
So there's going to be
an equilibrium where every

896
01:05:31,840 --> 01:05:38,717
company spends less than 100% on training
and certainly less than 100% on inference.

897
01:05:38,800 --> 01:05:43,804
It should be clear why you don't just
serve the current models and never train

898
01:05:43,887 --> 01:05:48,143
another model, because then you don't have
any demand because you'll fall behind.

899
01:05:48,226 --> 01:05:52,590
So there's some equilibrium. It's not
gonna be 10%, it's not gonna be 90%.

900
01:05:52,673 --> 01:05:55,517
Let's just say as a
stylized fact, it's 50%.

901
01:05:55,600 --> 01:05:59,460
That's what I'm getting at.
I think we're gonna be in a position where

902
01:05:59,543 --> 01:06:01,037
that equilibrium of how much

903
01:06:01,120 --> 01:06:07,466
you spend on training is less than
the gross margins that you're able to get

904
01:06:07,549 --> 01:06:11,517
on compute. So the underlying
economics are profitable.

905
01:06:11,600 --> 01:06:14,637
The problem is you
have this hellish demand

906
01:06:14,720 --> 01:06:20,359
prediction problem when you're buying the
next year of compute and you might guess

907
01:06:20,443 --> 01:06:26,408
under and be very profitable
but have no compute for research.

908
01:06:26,492 --> 01:06:30,557
Or you might guess over and you are not

909
01:06:30,640 --> 01:06:34,805
profitable and you have all
the compute for research in the world.

910
01:06:34,888 --> 01:06:36,157
Does that make sense?

911
01:06:36,240 --> 01:06:40,625
Just as a dynamic model of the industry?
Maybe stepping back,

912
01:06:40,708 --> 01:06:42,237
I'm not saying I think

913
01:06:42,320 --> 01:06:45,631
the "country of geniuses" is going to come
in two years and therefore you should buy

914
01:06:45,715 --> 01:06:46,548
this compute.

915
01:06:47,120 --> 01:06:51,517
To me, the end conclusion you're
arriving at makes a lot of sense.

916
01:06:51,600 --> 01:06:56,929
But that's because it seems like "country
of geniuses" is hard and there's a long

917
01:06:57,012 --> 01:06:57,597
way to go.

918
01:06:57,680 --> 01:07:02,451
So stepping back, the thing I'm trying
to get at is more that it seems like

919
01:07:02,534 --> 01:07:03,357
your worldview

920
01:07:03,440 --> 01:07:07,140
is compatible with somebody who says,
"We're like 10 years away from a world

921
01:07:07,223 --> 01:07:07,837
in which we're

922
01:07:07,920 --> 01:07:14,637
generating trillions of dollars of value."
That's just not my view. So I'll make

923
01:07:14,720 --> 01:07:19,917
another prediction. It is hard for me
to see that there won't be trillions

924
01:07:20,000 --> 01:07:26,077
of dollars in revenue before 2030.
I can construct a plausible world.

925
01:07:26,160 --> 01:07:31,197
It takes maybe three years. That would
be the end of what I think it's plausible.

926
01:07:31,280 --> 01:07:36,317
Like in 2028, we get the real "country
of geniuses in the data center".

927
01:07:36,400 --> 01:07:43,400
The revenue's going into the low hundreds
of billions by 2028, and then the country

928
01:07:46,400 --> 01:07:51,331
of geniuses accelerates it to trillions.
We’re basically on the slow end

929
01:07:51,415 --> 01:07:55,714
of diffusion. It takes two
years to get to the trillions.

930
01:07:55,797 --> 01:07:59,757
That would be the world
where it takes until 2030.

931
01:07:59,840 --> 01:08:05,357
I suspect even composing the technical
exponential and diffusion exponential,

932
01:08:05,440 --> 01:08:10,397
we’ll get there before 2030. So you
laid out a model where Anthropic makes

933
01:08:10,480 --> 01:08:14,717
profit because it seems like fundamentally
we're in a compute-constrained world.

934
01:08:14,800 --> 01:08:20,160
So eventually we keep growing compute— I
think the way the profit comes is… Again,

935
01:08:21,040 --> 01:08:23,765
let's just abstract
the whole industry here.

936
01:08:23,849 --> 01:08:26,957
Let's just imagine we're
in an economics textbook.

937
01:08:27,040 --> 01:08:32,320
We have a small number of firms.
Each can invest a limited amount.

938
01:08:33,680 --> 01:08:38,477
Each can invest some fraction in R&D.
They have some marginal cost to serve.

939
01:08:38,560 --> 01:08:45,560
The gross profit margins on that marginal
cost are very high because inference

940
01:08:45,829 --> 01:08:50,160
is efficient. There's some competition,
but the models are also differentiated.

941
01:08:52,720 --> 01:08:55,517
Companies will compete to push
their research budgets up.

942
01:08:55,600 --> 01:09:00,557
But because there's a small number of
players, we have the... What is it called?

943
01:09:00,640 --> 01:09:05,837
The Cournot equilibrium, I think, is what
the small number of firm equilibrium is.

944
01:09:05,920 --> 01:09:12,920
The point is it doesn't equilibrate
to perfect competition with zero margins.

945
01:09:15,600 --> 01:09:20,158
If there's three firms in the economy
and all are kind of independently behaving

946
01:09:20,242 --> 01:09:24,797
rationally, it doesn't equilibrate
to zero. Help me understand that, because

947
01:09:24,880 --> 01:09:30,165
right now we do have three leading
firms and they're not making profit.

948
01:09:30,248 --> 01:09:31,760
So what is changing?

949
01:09:33,840 --> 01:09:38,077
Again, the gross margins
right now are very positive.

950
01:09:38,160 --> 01:09:41,106
What's happening is a
combination of two things.

951
01:09:41,189 --> 01:09:43,757
One is that we're still in the exponential

952
01:09:43,840 --> 01:09:50,840
scale-up phase of compute. A model
gets trained. Let's say a model got

953
01:09:53,600 --> 01:10:00,600
trained that costs $1 billion last year.
Then this year it produced $4 billion of

954
01:10:02,800 --> 01:10:08,129
revenue and cost $1 billion
to inference from. Again,

955
01:10:08,213 --> 01:10:12,317
I'm using stylized numbers here, but that

956
01:10:12,400 --> 01:10:17,842
would be 75% gross
margins and this 25% tax.

957
01:10:17,926 --> 01:10:23,117
So that model as a whole makes $2 billion.

958
01:10:23,200 --> 01:10:27,437
But at the same time, we're spending $10
billion to train the next model because

959
01:10:27,520 --> 01:10:31,117
there's an exponential scale-up.
So the company loses money. Each model

960
01:10:31,200 --> 01:10:35,597
makes money, but the company loses money.
The equilibrium I'm talking about is an

961
01:10:35,680 --> 01:10:41,440
equilibrium where we have the "country
of geniuses in a data center", but that

962
01:10:43,440 --> 01:10:47,954
model training scale-up has equilibrated
more. Maybe it's still going up.

963
01:10:48,037 --> 01:10:52,507
We're still trying to predict the demand,
but it's more leveled out.

964
01:10:52,590 --> 01:10:55,680
I'm confused about
a couple of things there.

965
01:10:56,720 --> 01:11:00,637
Let's start with the current world.
In the current world, you're right that,

966
01:11:00,720 --> 01:11:04,328
as you said before, if you treat
each individual model as a company,

967
01:11:04,411 --> 01:11:05,197
it's profitable.

968
01:11:05,280 --> 01:11:10,361
But of course, a big part of the
production function of being a frontier

969
01:11:10,444 --> 01:11:12,957
lab is training the next model,
right? Yes, that's right.

970
01:11:13,040 --> 01:11:16,157
If you didn't do that, then you'd
make profit for two months and then

971
01:11:16,240 --> 01:11:19,117
you wouldn't have margins because
you wouldn't have the best model.

972
01:11:19,200 --> 01:11:23,597
But at some point that reaches
the biggest scale that it can reach.

973
01:11:23,680 --> 01:11:26,708
And then in equilibrium,
we have algorithmic improvements,

974
01:11:26,791 --> 01:11:28,317
but we're spending roughly the

975
01:11:28,400 --> 01:11:32,720
same amount to train the next model
as we spend to train the current model.

976
01:11:37,520 --> 01:11:40,131
At some point you run out
of money in the economy.

977
01:11:40,215 --> 01:11:42,557
A fixed lump of labor
fallacy… The economy is

978
01:11:42,640 --> 01:11:44,317
going to grow, right? That's one
of your predictions. We're going

979
01:11:44,400 --> 01:11:47,357
to have the data centers in space.
Yes, but this is another example

980
01:11:47,440 --> 01:11:53,037
of the theme I was talking about.
The economy will grow much faster

981
01:11:53,120 --> 01:11:59,190
with AI than I think it ever has before.
Right now the compute is growing 3x

982
01:11:59,273 --> 01:12:03,597
a year. I don't believe the economy
is gonna grow 300% a year.

983
01:12:03,680 --> 01:12:08,157
I said this in "Machines of Loving
Grace", I think we may get 10-20%

984
01:12:08,240 --> 01:12:13,197
per year growth in the economy, but we're
not gonna get 300% growth in the economy.

985
01:12:13,280 --> 01:12:18,877
So I think in the end, if compute becomes
the majority of what the economy produces,

986
01:12:18,960 --> 01:12:22,557
it's gonna be capped by that.
So let's assume a model

987
01:12:22,640 --> 01:12:26,877
where compute stays capped. The world
where frontier labs are making money

988
01:12:26,960 --> 01:12:30,715
is one where they continue
to make fast progress.

989
01:12:30,798 --> 01:12:34,397
Because fundamentally
your margin is limited by

990
01:12:34,480 --> 01:12:36,957
how good the alternative is.
So you are able to make money

991
01:12:37,040 --> 01:12:39,437
because you have a frontier model.
If you didn't have a frontier model

992
01:12:39,520 --> 01:12:45,037
you wouldn't be making money.
So this model requires there

993
01:12:45,120 --> 01:12:47,997
never to be a steady state.
Forever and ever you keep

994
01:12:48,080 --> 01:12:51,037
making more algorithmic progress.
I don't think that's true. I mean,

995
01:12:51,120 --> 01:12:58,120
I feel like we're in an economics class.
Do you know the Tyler Cowen quote?

996
01:12:59,920 --> 01:13:03,677
We never stop talking about economics.
We never stop talking about economics.

997
01:13:03,760 --> 01:13:10,760
So no, I don't think this field's
going to be a monopoly.

998
01:13:12,560 --> 01:13:15,117
All my lawyers never want me
to say the word "monopoly".

999
01:13:15,200 --> 01:13:17,837
But I don't think this field's
going to be a monopoly.

1000
01:13:17,920 --> 01:13:21,437
You do get industries in which
there are a small number of players.

1001
01:13:21,520 --> 01:13:27,517
Not one, but a small number of players.
Ordinarily, the way you get monopolies

1002
01:13:27,600 --> 01:13:34,600
like Facebook or Meta—I always call
them Facebook—is these kinds of network

1003
01:13:36,275 --> 01:13:40,957
effects. The way you get industries in
which there are a small number of players,

1004
01:13:41,040 --> 01:13:45,538
is very high costs of entry.
Cloud is like this.

1005
01:13:45,622 --> 01:13:49,357
I think cloud is a good example of this.

1006
01:13:49,440 --> 01:13:52,197
There are three, maybe four,
players within cloud.

1007
01:13:52,280 --> 01:13:55,120
I think that's the same
for AI, three, maybe four.

1008
01:13:56,640 --> 01:14:02,157
The reason is that it's so expensive.
It requires so much expertise and so

1009
01:14:02,240 --> 01:14:08,157
much capital to run a cloud company.
You have to put up all this capital.

1010
01:14:08,240 --> 01:14:10,044
In addition to putting
up all this capital,

1011
01:14:10,128 --> 01:14:11,757
you have to get all of this other stuff

1012
01:14:11,840 --> 01:14:14,527
that requires a lot
of skill to make it happen.

1013
01:14:14,611 --> 01:14:17,357
So if you go to someone
and you're like, "I want

1014
01:14:17,440 --> 01:14:21,063
to disrupt this industry, here's
$100 billion." You're like, "okay,

1015
01:14:21,146 --> 01:14:22,557
I'm putting in $100 billion

1016
01:14:22,640 --> 01:14:25,941
and also betting that you can do all these
other things that these people have been

1017
01:14:26,025 --> 01:14:29,277
doing." Only to decrease the profit.
The effect of your entering

1018
01:14:29,360 --> 01:14:33,197
is that profit margins go down.
So, we have equilibria like this

1019
01:14:33,280 --> 01:14:37,147
all the time in the economy
where we have a few players.

1020
01:14:37,230 --> 01:14:39,757
Profits are not astronomical. Margins

1021
01:14:39,840 --> 01:14:46,059
are not astronomical, but they're
not zero. That's what we see on cloud.

1022
01:14:46,142 --> 01:14:50,800
Cloud is very undifferentiated.
Models are more differentiated than cloud.

1023
01:14:51,680 --> 01:14:57,076
Everyone knows Claude is good
at different things than GPT is good at,

1024
01:14:57,160 --> 01:14:58,877
than Gemini is good at.

1025
01:14:58,960 --> 01:15:03,360
It's not just that Claude's good at
coding, GPT is good at math and reasoning.

1026
01:15:05,120 --> 01:15:08,578
It's more subtle than that. Models
are good at different types of coding.

1027
01:15:08,661 --> 01:15:09,597
Models have different

1028
01:15:09,680 --> 01:15:14,270
styles. I think these things are
actually quite different from each other,

1029
01:15:14,354 --> 01:15:15,597
and so I would expect

1030
01:15:15,680 --> 01:15:21,688
more differentiation than
you see in cloud. Now,

1031
01:15:21,771 --> 01:15:26,637
there actually is one counter-argument.

1032
01:15:26,720 --> 01:15:30,320
That counter-argument is if
the process of producing models,

1033
01:15:32,960 --> 01:15:37,517
if AI models can do that themselves, then
that could spread throughout the economy.

1034
01:15:37,600 --> 01:15:41,197
But that is not an argument
for commoditizing AI models in general.

1035
01:15:41,280 --> 01:15:45,357
That's kind of an argument for
commoditizing the whole economy at once.

1036
01:15:45,440 --> 01:15:48,877
I don't know what quite happens
in that world where basically anyone

1037
01:15:48,960 --> 01:15:51,277
can do anything,
anyone can build anything,

1038
01:15:51,360 --> 01:15:53,677
and there's no moat
around anything at all.

1039
01:15:53,760 --> 01:15:58,477
I don't know, maybe we want that world.
Maybe that's the end state here.

1040
01:15:58,560 --> 01:16:05,560
Maybe when AI models can do everything, if
we've solved all the safety and security

1041
01:16:08,231 --> 01:16:09,197
problems,

1042
01:16:09,280 --> 01:16:16,280
that's one of the mechanisms for the
economy just flattening itself again.

1043
01:16:17,920 --> 01:16:22,480
But that's kind of far post-"country
of geniuses in the data center."

1044
01:16:23,440 --> 01:16:28,224
Maybe a finer way to put
that potential point is:

1045
01:16:28,308 --> 01:16:32,397
1) it seems like AI research is especially

1046
01:16:32,480 --> 01:16:37,252
loaded on raw intellectual power, which
will be especially abundant in the world

1047
01:16:37,335 --> 01:16:39,664
of AGI. And 2) if you just
look at the world today,

1048
01:16:39,748 --> 01:16:41,837
there are very few
technologies that seem to be

1049
01:16:41,920 --> 01:16:46,208
diffusing as fast as AI
algorithmic progress.

1050
01:16:46,291 --> 01:16:49,997
So that does hint that this industry is

1051
01:16:50,080 --> 01:16:54,157
sort of structurally diffusive.
I think coding is going fast, but

1052
01:16:54,240 --> 01:16:57,445
I think AI research is a superset
of coding and there are aspects of it

1053
01:16:57,528 --> 01:16:58,728
that are not going fast.

1054
01:17:00,320 --> 01:17:05,641
But I do think, again, once we get coding,
once we get AI models going fast,

1055
01:17:05,724 --> 01:17:07,277
then that will speed up

1056
01:17:07,360 --> 01:17:12,720
the ability of AI models to do everything
else. So while coding is going fast now,

1057
01:17:12,803 --> 01:17:13,517
I think once

1058
01:17:13,600 --> 01:17:17,437
the AI models are building the next
AI models and building everything else,

1059
01:17:17,520 --> 01:17:21,657
the whole economy will
kind of go at the same pace.

1060
01:17:21,741 --> 01:17:24,637
I am worried geographically, though.

1061
01:17:24,720 --> 01:17:31,695
I'm a little worried that just
proximity to AI, having heard about AI,

1062
01:17:31,778 --> 01:17:34,317
may be one differentiator.

1063
01:17:34,400 --> 01:17:41,169
So when I said the 10-20% growth rate, a
worry I have is that the growth rate could

1064
01:17:41,252 --> 01:17:42,077
be like 50%

1065
01:17:42,160 --> 01:17:48,461
in Silicon Valley and parts of the world
that are socially connected to Silicon

1066
01:17:48,544 --> 01:17:52,100
Valley, and not that much faster
than its current pace elsewhere.

1067
01:17:52,183 --> 01:17:54,077
I think that'd be
a pretty messed up world.

1068
01:17:54,160 --> 01:17:57,197
So one of the things I think
about a lot is how to prevent that.

1069
01:17:57,280 --> 01:18:01,837
Do you think that once we have this
country of geniuses in a data center, that

1070
01:18:01,920 --> 01:18:03,928
robotics is sort of quickly
solved afterwards?

1071
01:18:04,012 --> 01:18:06,157
Because it seems like
a big problem with robotics

1072
01:18:06,240 --> 01:18:10,676
is that a human can learn how
to teleoperate current hardware,

1073
01:18:10,759 --> 01:18:14,802
but current AI models can't, at least
not in a way that's super productive.

1074
01:18:14,885 --> 01:18:16,797
And so if we have
this ability to learn like

1075
01:18:16,880 --> 01:18:19,037
a human, shouldn't it solve
robotics immediately as well?

1076
01:18:19,120 --> 01:18:21,437
I don't think it's dependent
on learning like a human.

1077
01:18:21,520 --> 01:18:25,357
It could happen in different ways.
Again, we could have trained the model on

1078
01:18:25,440 --> 01:18:28,357
many different video games,
which are like robotic controls,

1079
01:18:28,440 --> 01:18:30,157
or many different simulated robotics

1080
01:18:30,240 --> 01:18:33,247
environments, or just train
them to control computer screens,

1081
01:18:33,331 --> 01:18:34,717
and they learn to generalize.

1082
01:18:34,800 --> 01:18:41,597
So it will happen... it's not necessarily
dependent on human-like learning.

1083
01:18:41,680 --> 01:18:44,228
Human-like learning is one way it
could happen. If the model's like, "Oh,

1084
01:18:44,311 --> 01:18:44,877
I pick up a robot,

1085
01:18:44,960 --> 01:18:49,497
I don't know how to use it, I learn,"
that could happen because we discovered

1086
01:18:49,580 --> 01:18:50,637
continual learning.

1087
01:18:50,720 --> 01:18:53,917
That could also happen because we trained
the model on a bunch of environments and

1088
01:18:54,000 --> 01:18:57,367
then generalized, or it could happen
because the model learns that in

1089
01:18:57,450 --> 01:18:58,400
the context length.

1090
01:18:58,960 --> 01:19:03,117
It doesn't actually matter which way.
If we go back to the discussion we had

1091
01:19:03,200 --> 01:19:08,800
an hour ago, that type of thing can
happen in several different ways.

1092
01:19:10,240 --> 01:19:15,186
But I do think when for whatever
reason the models have those skills,

1093
01:19:15,269 --> 01:19:16,717
then robotics will be

1094
01:19:16,800 --> 01:19:20,461
revolutionized—both the design of robots,
because the models will be much better

1095
01:19:20,545 --> 01:19:24,959
than humans at that, and also
the ability to control robots.

1096
01:19:25,043 --> 01:19:28,717
So we'll get better
at building the physical

1097
01:19:28,800 --> 01:19:32,109
hardware, building the physical robots,
and we'll also get better at controlling

1098
01:19:32,193 --> 01:19:36,717
it. Now, does that mean the robotics
industry will also be generating

1099
01:19:36,800 --> 01:19:40,397
trillions of dollars of revenue?
My answer there is yes, but there will be

1100
01:19:40,480 --> 01:19:44,213
the same extremely fast,
but not infinitely fast diffusion.

1101
01:19:44,296 --> 01:19:46,477
So will robotics be revolutionized?

1102
01:19:46,560 --> 01:19:51,840
Yeah, maybe tack on another year or two.
That's the way I think about these things.

1103
01:19:52,400 --> 01:19:57,127
Makes sense. There's a general
skepticism about extremely fast progress.

1104
01:19:57,210 --> 01:19:58,797
Here's my view. It sounds

1105
01:19:58,880 --> 01:20:01,933
like you are going to solve continual
learning one way or another within

1106
01:20:02,016 --> 01:20:02,717
a matter of years.

1107
01:20:02,800 --> 01:20:06,717
But just as people weren't talking about
continual learning a couple of years ago,

1108
01:20:06,800 --> 01:20:09,478
and then we realized, "Oh, why aren't
these models as useful as they could be

1109
01:20:09,561 --> 01:20:09,837
right now,

1110
01:20:09,920 --> 01:20:13,225
even though they are clearly passing
the Turing test and are experts in so many

1111
01:20:13,308 --> 01:20:13,997
different domains?

1112
01:20:14,080 --> 01:20:18,483
Maybe it's this thing." Then we solve
this thing and we realize, actually,

1113
01:20:18,567 --> 01:20:19,757
there's another thing

1114
01:20:19,840 --> 01:20:23,558
that human intelligence can do that's
a basis of human labor that these models

1115
01:20:23,641 --> 01:20:27,600
can't do. So why not think there
will be more things like this, where

1116
01:20:28,560 --> 01:20:31,980
we've found more pieces of human
intelligence? Well, to be clear,

1117
01:20:32,063 --> 01:20:36,982
I think continual learning, as I've said
before, might not be a barrier at all.

1118
01:20:37,066 --> 01:20:40,157
I think we may just get
there by pre-training

1119
01:20:40,240 --> 01:20:45,280
generalization and RL generalization.
I think there just

1120
01:20:48,240 --> 01:20:51,837
might not be such a thing at all.
In fact, I would point to the history

1121
01:20:51,920 --> 01:20:56,637
in ML of people coming up with things
that are barriers that end up kind of

1122
01:20:56,720 --> 01:21:03,720
dissolving within the big blob of compute.
People talked about, "How do your models

1123
01:21:06,480 --> 01:21:11,837
keep track of nouns and verbs?"
"They can understand syntactically,

1124
01:21:11,920 --> 01:21:16,637
but they can't understand semantically?
It's only statistical correlations."

1125
01:21:16,720 --> 01:21:19,437
"You can understand a paragraph,
you can’t understand a word.

1126
01:21:19,520 --> 01:21:21,348
There's reasoning,
you can't do reasoning."

1127
01:21:21,431 --> 01:21:23,037
But then suddenly it turns out you can

1128
01:21:23,120 --> 01:21:27,757
do code and math very well.
So I think there's actually a

1129
01:21:27,840 --> 01:21:33,868
stronger history of some of these things
seeming like a big deal and then kind

1130
01:21:33,951 --> 01:21:39,042
of dissolving. Some of them are real.
The need for data is real,

1131
01:21:39,126 --> 01:21:42,477
maybe continual learning is a real thing.

1132
01:21:42,560 --> 01:21:45,920
But again, I would ground
us in something like code.

1133
01:21:46,480 --> 01:21:50,077
I think we may get to the point
in a year or two where the models can

1134
01:21:50,160 --> 01:21:53,117
just do SWE end-to-end.
That's a whole task.

1135
01:21:53,200 --> 01:21:56,157
That's a whole sphere
of human activity that

1136
01:21:56,240 --> 01:22:00,800
we're just saying models can do now.
When you say end-to-end, do you mean

1137
01:22:02,000 --> 01:22:06,557
setting technical direction, understanding
the context of the problem, et cetera?

1138
01:22:06,640 --> 01:22:12,560
Yes. I mean all of that. Interesting.
I feel like that is AGI-complete,

1139
01:22:13,520 --> 01:22:17,197
which maybe is internally consistent.
But it's not like saying 90%

1140
01:22:17,280 --> 01:22:22,637
of code or 100% of code.
No, I gave this spectrum:

1141
01:22:22,720 --> 01:22:28,557
90% of code, 100% of code, 90%
of end-to-end SWE, 100% of end-to-end SWE.

1142
01:22:28,640 --> 01:22:31,837
New tasks are created for SWEs.
Eventually those get done as well.

1143
01:22:31,920 --> 01:22:35,677
It's a long spectrum there, but we're
traversing the spectrum very quickly.

1144
01:22:35,760 --> 01:22:39,510
I do think it's funny that I've seen
a couple of podcasts you've done where

1145
01:22:40,080 --> 01:22:42,957
the hosts will be like, "But Dwarkesh
wrote the essay about the continuous

1146
01:22:43,040 --> 01:22:43,597
learning thing."

1147
01:22:43,680 --> 01:22:47,840
It always makes me crack up because
you've been an AI researcher for 10 years.

1148
01:22:48,880 --> 01:22:53,517
I'm sure there's some feeling of,
"Okay, so a podcaster wrote an essay,

1149
01:22:53,600 --> 01:22:59,037
and every interview I get asked about it."
The truth of the matter is that we're all

1150
01:22:59,120 --> 01:23:04,717
trying to figure this out together.
There are some ways in which I'm

1151
01:23:04,800 --> 01:23:08,877
able to see things that others aren't.
These days that probably has more to do

1152
01:23:08,960 --> 01:23:12,477
with seeing a bunch of stuff within
Anthropic and having to make a bunch

1153
01:23:12,560 --> 01:23:13,677
of decisions than I have

1154
01:23:13,760 --> 01:23:20,077
any great research insight that others
don't. I'm running a 2,500 person company.

1155
01:23:20,160 --> 01:23:25,160
It's actually pretty hard for me
to have concrete research insight,

1156
01:23:25,244 --> 01:23:32,003
much harder than it would have been 10
years ago or even two or three years ago.

1157
01:23:32,086 --> 01:23:35,917
As we go towards a world of a full drop-in

1158
01:23:36,000 --> 01:23:42,077
remote worker replacement, does an API
pricing model still make the most sense?

1159
01:23:42,160 --> 01:23:45,837
If not, what is the correct
way to price AGI, or serve AGI?

1160
01:23:45,920 --> 01:23:49,251
I think there's going to be a bunch
of different business models here,

1161
01:23:49,335 --> 01:23:49,837
all at once,

1162
01:23:49,920 --> 01:23:56,920
that are going to be experimented with.
I actually do think that the API

1163
01:23:59,680 --> 01:24:03,237
model is more durable
than many people think.

1164
01:24:03,320 --> 01:24:06,877
One way I think about
it is if the technology

1165
01:24:06,960 --> 01:24:11,997
is advancing quickly, if it's advancing
exponentially, what that means is there's

1166
01:24:12,080 --> 01:24:19,080
always a surface area of new use cases
that have been developed in the last three

1167
01:24:19,444 --> 01:24:25,838
months. Any kind of product surface you
put in place is always at risk of sort

1168
01:24:25,921 --> 01:24:27,757
of becoming irrelevant.

1169
01:24:27,840 --> 01:24:32,217
Any given product surface probably makes
sense for a range of capabilities of

1170
01:24:32,301 --> 01:24:37,812
the model. The chatbot is already running
into limitations where making it smarter

1171
01:24:37,895 --> 01:24:41,837
doesn't really help the average consumer
that much. But I don't think that's

1172
01:24:41,920 --> 01:24:45,357
a limitation of AI models.
I don't think that's evidence

1173
01:24:45,440 --> 01:24:50,510
that the models are good enough
and them getting better doesn't matter to

1174
01:24:50,593 --> 01:24:54,918
the economy. It doesn't matter
to that particular product.

1175
01:24:55,001 --> 01:24:58,637
So I think the value
of the API is that the API

1176
01:24:58,720 --> 01:25:02,718
always offers an opportunity,
very close to the bare metal,

1177
01:25:02,801 --> 01:25:05,360
to build on what the latest thing is.

1178
01:25:06,720 --> 01:25:13,720
There's always going to be this front
of new startups and new ideas that

1179
01:25:14,400 --> 01:25:18,500
weren't possible a few months ago and are
possible because the model is advancing.

1180
01:25:19,520 --> 01:25:26,302
I actually predict that it's going
to exist alongside other models,

1181
01:25:26,386 --> 01:25:28,557
but we're always going

1182
01:25:28,640 --> 01:25:33,370
to have the API business model because
there's always going to be a need for

1183
01:25:33,453 --> 01:25:34,637
a thousand different

1184
01:25:34,720 --> 01:25:38,263
people to try experimenting
with the model in a different way.

1185
01:25:38,347 --> 01:25:42,605
100 of them become startups and ten
of them become big successful startups.

1186
01:25:42,688 --> 01:25:44,957
Two or three really end up being the way

1187
01:25:45,040 --> 01:25:47,917
that people use the model
of a given generation.

1188
01:25:48,000 --> 01:25:50,877
So I basically think it's
always going to exist.

1189
01:25:50,960 --> 01:25:55,677
At the same time, I'm sure there's
going to be other models as well.

1190
01:25:55,760 --> 01:26:00,317
Not every token that's output
by the model is worth the same amount.

1191
01:26:00,400 --> 01:26:07,400
Think about what is the value of the
tokens that the model outputs when someone

1192
01:26:10,181 --> 01:26:13,840
calls them up and says, "My Mac isn't
working," or something, the model's like,

1193
01:26:13,924 --> 01:26:14,757
"restart it."

1194
01:26:16,160 --> 01:26:22,240
Someone hasn't heard that before,
but the model said that 10 million times.

1195
01:26:23,280 --> 01:26:26,330
Maybe that's worth like a dollar
or a few cents or something.

1196
01:26:26,880 --> 01:26:33,880
Whereas if the model goes to one of the
pharmaceutical companies and it says, "Oh,

1197
01:26:34,800 --> 01:26:36,712
you know, this molecule you're developing,

1198
01:26:36,795 --> 01:26:39,277
you should take the aromatic
ring from that end of the

1199
01:26:39,360 --> 01:26:43,198
molecule and put it on that end
of the molecule. If you do that,

1200
01:26:43,282 --> 01:26:45,120
wonderful things will happen."

1201
01:26:46,800 --> 01:26:50,240
Those tokens could be worth
tens of millions of dollars.

1202
01:26:52,400 --> 01:26:56,797
So I think we're definitely going to see
business models that recognize that.

1203
01:26:56,880 --> 01:27:03,158
At some point we're going to see
"pay for results" in some form,

1204
01:27:03,241 --> 01:27:06,637
or we may see forms of compensation

1205
01:27:06,720 --> 01:27:13,720
that are like labor, that kind
of work by the hour. I don't know.

1206
01:27:13,977 --> 01:27:16,797
I think because it's a new

1207
01:27:16,880 --> 01:27:19,837
industry, a lot of things are going to be
tried. I don't know what will turn out to

1208
01:27:19,920 --> 01:27:24,397
be the right thing. I take your point that

1209
01:27:24,480 --> 01:27:28,145
people will have to try things to figure
out what is the best way to use this blob

1210
01:27:28,229 --> 01:27:28,877
of intelligence.

1211
01:27:28,960 --> 01:27:34,717
But what I find striking is Claude Code.
I don't think in the history of startups

1212
01:27:34,800 --> 01:27:40,307
there has been a single application that
has been as hotly competed in as coding

1213
01:27:40,391 --> 01:27:41,224
agents.

1214
01:27:42,960 --> 01:27:48,563
Claude Code is a category leader here.
That seems surprising to me.

1215
01:27:48,647 --> 01:27:52,211
It doesn't seem intrinsically
that Anthropic had to build this.

1216
01:27:52,295 --> 01:27:54,637
I wonder if you have
an accounting of why it had

1217
01:27:54,720 --> 01:27:58,077
to be Anthropic or how Anthropic ended
up building an application in addition

1218
01:27:58,160 --> 01:27:59,997
to the model underlying
it that was successful.

1219
01:28:00,080 --> 01:28:01,917
So it actually happened
in a pretty simple way,

1220
01:28:02,000 --> 01:28:08,640
which is that we had our own coding
models, which were good at coding.

1221
01:28:09,440 --> 01:28:14,317
Around the beginning of 2025, I said, "I
think the time has come where you can have

1222
01:28:14,400 --> 01:28:20,409
nontrivial acceleration of your own
research if you're an AI company by using

1223
01:28:20,492 --> 01:28:25,437
these models." Of course, you need an
interface, you need a harness to use them.

1224
01:28:25,520 --> 01:28:30,948
So I encouraged people internally. I
didn't say this is one thing that you have

1225
01:28:31,032 --> 01:28:34,928
to use. I just said people
should experiment with this.

1226
01:28:35,011 --> 01:28:37,677
I think it might have been originally

1227
01:28:37,760 --> 01:28:41,997
called Claude CLI, and then the name
eventually got changed to Claude Code.

1228
01:28:42,080 --> 01:28:47,777
Internally, it was the thing that everyone
was using and it was seeing fast internal

1229
01:28:47,861 --> 01:28:48,397
adoption.

1230
01:28:48,480 --> 01:28:52,380
I looked at it and I said, "Probably we
should launch this externally, right?"

1231
01:28:53,200 --> 01:28:59,040
It's seen such fast adoption within
Anthropic. Coding is a lot of what we do.

1232
01:28:59,920 --> 01:29:04,557
We have an audience of many, many hundreds
of people that's in some ways at least

1233
01:29:04,640 --> 01:29:08,317
representative of the external audience.
So it looks like we already have product

1234
01:29:08,400 --> 01:29:13,034
market fit. Let's launch this thing.
And then we launched it.

1235
01:29:13,117 --> 01:29:15,277
I think just the fact that we

1236
01:29:15,360 --> 01:29:20,180
ourselves are kind of developing the model
and we ourselves know what we most need

1237
01:29:20,263 --> 01:29:24,382
to use the model, I think it's kind
of creating this feedback loop. I see.

1238
01:29:24,466 --> 01:29:26,317
In the sense that you, let's say a

1239
01:29:26,400 --> 01:29:31,181
developer at Anthropic is like, "Ah, it
would be better if it was better at this X

1240
01:29:31,265 --> 01:29:35,037
thing." Then you bake that into
the next model that you build.

1241
01:29:35,120 --> 01:29:39,200
That's one version of it, but then there's
just the ordinary product iteration.

1242
01:29:41,520 --> 01:29:47,197
We have a bunch of coders within
Anthropic, they use Claude Code

1243
01:29:47,280 --> 01:29:50,237
every day and so we get fast feedback.
That was more important in the early days.

1244
01:29:50,320 --> 01:29:53,517
Now, of course, there are millions
of people using it, and so we get

1245
01:29:53,600 --> 01:29:58,077
a bunch of external feedback as well.
But it's just great to be able to get

1246
01:29:58,160 --> 01:30:03,837
kind of fast internal feedback.
I think this is the reason why we

1247
01:30:03,920 --> 01:30:07,840
launched a coding model and didn't
launch a pharmaceutical company.

1248
01:30:10,560 --> 01:30:14,557
My background's in biology, but we
don't have any of the resources that

1249
01:30:14,640 --> 01:30:21,640
are needed to launch
a pharmaceutical company.

1250
01:30:50,200 --> 01:30:57,200
Let me now ask you
about making AI go well.

1251
01:31:24,320 --> 01:31:28,451
It seems like whatever vision we
have about how AI goes well has to be

1252
01:31:28,534 --> 01:31:30,077
compatible with two things:

1253
01:31:30,160 --> 01:31:37,160
1) the ability to build and run AIs
is diffusing extremely rapidly and 2) the

1254
01:31:37,440 --> 01:31:41,135
population of AIs, the amount
we have and their intelligence,

1255
01:31:41,218 --> 01:31:43,200
will also increase very rapidly.

1256
01:31:44,160 --> 01:31:48,201
That means that lots of people will
be able to build huge populations

1257
01:31:48,284 --> 01:31:49,277
of misaligned AIs,

1258
01:31:49,360 --> 01:31:53,437
or AIs which are just companies
which are trying to increase their

1259
01:31:53,520 --> 01:31:57,837
footprint or have weird psyches like
Sydney Bing, but now they're superhuman.

1260
01:31:57,920 --> 01:32:02,797
What is a vision for a world in which
we have an equilibrium that is compatible

1261
01:32:02,880 --> 01:32:06,530
with lots of different AIs, some
of which are misaligned, running around?

1262
01:32:06,720 --> 01:32:10,242
I think in "The
Adolescence of Technology",

1263
01:32:10,326 --> 01:32:13,597
I was skeptical of the balance of power.

1264
01:32:13,680 --> 01:32:20,680
But the thing I was specifically skeptical
of is you have three or four of these

1265
01:32:21,302 --> 01:32:22,160
companies

1266
01:32:23,520 --> 01:32:30,520
all building models that are
derived from the same thing,

1267
01:32:31,880 --> 01:32:36,637
that they would check each other.

1268
01:32:36,720 --> 01:32:40,797
Or even that any number
of them would check each other.

1269
01:32:40,880 --> 01:32:46,747
We might live in an offense-dominant world
where one person or one AI model is smart

1270
01:32:46,830 --> 01:32:52,544
enough to do something that causes damage
for everything else. In the short run,

1271
01:32:52,628 --> 01:32:56,077
we have a limited number
of players now. So we can start

1272
01:32:56,160 --> 01:33:03,160
within the limited number of players.
We need to put in place the safeguards.

1273
01:33:03,680 --> 01:33:05,357
We need to make sure everyone
does the right alignment work.

1274
01:33:05,440 --> 01:33:08,693
We need to make sure
everyone has bioclassifiers.

1275
01:33:08,776 --> 01:33:11,757
Those are the immediate
things we need to do.

1276
01:33:11,840 --> 01:33:15,117
I agree that that doesn't solve
the problem in the long run,

1277
01:33:15,200 --> 01:33:21,348
particularly if the ability of AI models
to make other AI models proliferates,

1278
01:33:21,432 --> 01:33:26,080
then the whole thing can
become harder to solve.

1279
01:33:26,720 --> 01:33:30,717
I think in the long run we need
some architecture of governance.

1280
01:33:30,800 --> 01:33:34,957
We need some architecture of governance
that preserves human freedom,

1281
01:33:35,040 --> 01:33:42,040
but also allows us to govern a very
large number of human systems,

1282
01:33:45,160 --> 01:33:47,920
AI systems, hybrid

1283
01:33:52,560 --> 01:33:59,560
human-AI companies or economic units.
So we're gonna need to think about:

1284
01:34:01,520 --> 01:34:05,640
how do we protect the world
against bioterrorism?

1285
01:34:05,723 --> 01:34:09,840
How do we protect the world
against mirror life?

1286
01:34:11,200 --> 01:34:15,597
Probably we're gonna need some
kind of AI monitoring system

1287
01:34:15,680 --> 01:34:20,557
that monitors for all of these things.
But then we need to build this in a way

1288
01:34:20,640 --> 01:34:24,717
that preserves civil liberties
and our constitutional rights.

1289
01:34:24,800 --> 01:34:31,800
So I think just as anything else, it's
a new security landscape with a new set of

1290
01:34:34,640 --> 01:34:40,717
tools and a new set of vulnerabilities.
My worry is, if we had 100 years for this

1291
01:34:40,800 --> 01:34:44,837
to happen all very slowly,
we'd get used to it.

1292
01:34:44,920 --> 01:34:48,957
We've gotten used
to the presence of explosives

1293
01:34:49,040 --> 01:34:56,040
in society or the presence of various new
weapons or the presence of video cameras.

1294
01:34:58,320 --> 01:35:03,037
We would get used to it over 100 years and
we’d develop governance mechanisms. We'd

1295
01:35:03,120 --> 01:35:07,517
make our mistakes. My worry is just
that this is happening all so fast.

1296
01:35:07,600 --> 01:35:11,623
So maybe we need to do our thinking
faster about how to make these governance

1297
01:35:11,707 --> 01:35:12,560
mechanisms work.

1298
01:35:13,280 --> 01:35:15,900
It seems like in an
offense-dominant world,

1299
01:35:15,983 --> 01:35:19,357
over the course of the next
century—the idea is that AI

1300
01:35:19,440 --> 01:35:22,122
is making the progress that would happen
over the next century happen in some

1301
01:35:22,206 --> 01:35:25,874
period of five to ten years—we
would still need the same mechanisms,

1302
01:35:25,957 --> 01:35:29,197
or balance of power would
be similarly intractable,

1303
01:35:29,280 --> 01:35:36,237
even if humans were the only game in town.
I guess we have the advice of AI.

1304
01:35:36,320 --> 01:35:41,357
But it fundamentally doesn't seem like
a totally different ball game here.

1305
01:35:41,440 --> 01:35:44,157
If checks and balances were going to work,
they would work with humans as well.

1306
01:35:44,240 --> 01:35:47,437
If they aren't going to work,
they wouldn't work with AIs as well.

1307
01:35:47,520 --> 01:35:50,480
So maybe this just dooms human
checks and balances as well.

1308
01:35:51,440 --> 01:35:55,520
Again, I think there's some
way to make this happen.

1309
01:35:58,240 --> 01:36:01,890
The governments of the world may have
to work together to make it happen.

1310
01:36:02,400 --> 01:36:09,234
We may have to talk to AIs about
building societal structures in such a way

1311
01:36:09,318 --> 01:36:10,157
that these

1312
01:36:10,240 --> 01:36:15,302
defenses are possible. I don't know.
I don’t want to say this is so far ahead

1313
01:36:15,385 --> 01:36:15,837
in time,

1314
01:36:15,920 --> 01:36:20,437
but it’s so far ahead in technological
ability that may happen over a short

1315
01:36:20,520 --> 01:36:23,642
period of time, that it's hard
for us to anticipate it in advance.

1316
01:36:23,726 --> 01:36:25,517
Speaking of governments getting involved,

1317
01:36:25,600 --> 01:36:31,597
on December 26, the Tennessee legislature
introduced a bill which said, "It would

1318
01:36:31,680 --> 01:36:34,797
be an offense for a person to knowingly
train artificial intelligence to provide

1319
01:36:34,880 --> 01:36:39,357
emotional support, including through
open-ended conversations with a user."

1320
01:36:39,440 --> 01:36:46,440
Of course, one of the things that Claude
attempts to do is be a thoughtful,

1321
01:36:46,815 --> 01:36:48,797
knowledgeable friend.

1322
01:36:48,880 --> 01:36:52,077
In general, it seems like we're going
to have this patchwork of state laws.

1323
01:36:52,160 --> 01:36:55,588
A lot of the benefits that normal people
could experience as a result of AI are

1324
01:36:55,671 --> 01:36:56,077
going to be

1325
01:36:56,160 --> 01:36:59,277
curtailed, especially when we get into the
kinds of things you discuss in "Machines

1326
01:36:59,360 --> 01:37:02,797
of Loving Grace": biological freedom,
mental health improvements, et cetera.

1327
01:37:02,880 --> 01:37:08,932
It seems easy to imagine worlds in which
these get Whac-A-Moled away by different

1328
01:37:09,015 --> 01:37:09,917
laws, whereas

1329
01:37:10,000 --> 01:37:14,850
bills like this don't seem to address
the actual existential threats that you're

1330
01:37:14,933 --> 01:37:15,837
concerned about.

1331
01:37:15,920 --> 01:37:20,237
I'm curious to understand, in the context
of things like this, Anthropic's position

1332
01:37:20,320 --> 01:37:23,602
against the federal
moratorium on state AI laws.

1333
01:37:23,685 --> 01:37:27,120
There are many different
things going on at once.

1334
01:37:28,960 --> 01:37:34,557
I think that particular law is dumb.
It was clearly made by legislators

1335
01:37:34,640 --> 01:37:38,317
who just probably had little idea
what AI models could do and not do.

1336
01:37:38,400 --> 01:37:41,400
They're like, "AI models serving us,
that just sounds scary.

1337
01:37:41,840 --> 01:37:46,720
I don't want that to happen."
So we're not in favor of that.

1338
01:37:47,280 --> 01:37:50,185
But that wasn't the thing
that was being voted on.

1339
01:37:50,269 --> 01:37:52,397
The thing that was being voted on is:

1340
01:37:52,480 --> 01:37:59,156
we're going to ban all state regulation
of AI for 10 years with no apparent plan

1341
01:37:59,240 --> 01:38:03,418
to do any federal regulation of AI,
which would take Congress to pass,

1342
01:38:03,501 --> 01:38:04,960
which is a very high bar.

1343
01:38:05,680 --> 01:38:09,710
So the idea that we'd ban states
from doing anything for 10 years…

1344
01:38:09,793 --> 01:38:10,957
People said they had

1345
01:38:11,040 --> 01:38:14,917
a plan for the federal government, but
there was no actual proposal on the table.

1346
01:38:15,000 --> 01:38:15,357
There was

1347
01:38:15,440 --> 01:38:21,222
no actual attempt. Given the serious
dangers that I lay out in "Adolescence

1348
01:38:21,305 --> 01:38:22,317
of Technology"

1349
01:38:22,400 --> 01:38:29,117
around things like biological weapons
and bioterrorism autonomy risk, and the

1350
01:38:29,200 --> 01:38:34,957
timelines we've been talking about—10
years is an eternity—I think that's

1351
01:38:35,040 --> 01:38:36,557
a crazy thing to do.

1352
01:38:36,640 --> 01:38:42,077
So if that's the choice, if that's what
you force us to choose, then we're going

1353
01:38:42,160 --> 01:38:47,677
to choose not to have that moratorium.
I think the benefits of that position

1354
01:38:47,760 --> 01:38:51,357
exceed the costs, but it's not
a perfect position if that's the choice.

1355
01:38:51,440 --> 01:38:55,451
Now, I think the thing that we should do,
the thing that I would support,

1356
01:38:55,535 --> 01:39:00,101
is the federal government should step in,
not saying "states you can't regulate",

1357
01:39:00,185 --> 01:39:05,438
but "Here's what we're going to do,
and states you can't differ from this."

1358
01:39:05,521 --> 01:39:08,877
I think preemption is fine in the sense of

1359
01:39:08,960 --> 01:39:11,740
saying that the federal government
says, "Here is our standard.

1360
01:39:11,824 --> 01:39:12,877
This applies to everyone.

1361
01:39:12,960 --> 01:39:16,077
States can't do something different."
That would be something I would support

1362
01:39:16,160 --> 01:39:21,917
if it would be done in the right way.
But this idea of states, "You can't do

1363
01:39:22,000 --> 01:39:25,799
anything and we're not
doing anything either,"

1364
01:39:25,882 --> 01:39:29,597
that struck us as very
much not making sense.

1365
01:39:29,680 --> 01:39:33,437
I think it will not age well, it
is already starting to not age well with

1366
01:39:33,520 --> 01:39:38,240
all the backlash that you've seen.
Now, in terms of what we would want,

1367
01:39:39,200 --> 01:39:44,832
the things we've talked about are starting
with transparency standards in order

1368
01:39:44,915 --> 01:39:49,439
to monitor some of these autonomy
risks and bioterrorism risks.

1369
01:39:49,523 --> 01:39:53,037
As the risks become more
serious, as we get more

1370
01:39:53,120 --> 01:39:57,811
evidence for them, then I think we could
be more aggressive in some targeted ways

1371
01:39:57,895 --> 01:40:04,317
and say, "Hey, AI bioterrorism is really
a threat. Let's pass a law that forces

1372
01:40:04,400 --> 01:40:07,837
people to have classifiers."
I could even imagine… It depends.

1373
01:40:07,920 --> 01:40:11,358
It depends how serious the threat it
ends up being. We don't know for sure.

1374
01:40:11,442 --> 01:40:12,157
We need to pursue

1375
01:40:12,240 --> 01:40:15,142
this in an intellectually honest
way where we say that ahead of time,

1376
01:40:15,225 --> 01:40:16,397
the risk has not emerged yet.

1377
01:40:16,480 --> 01:40:20,400
But I could certainly imagine,
with the pace that things are going at,

1378
01:40:21,440 --> 01:40:24,093
a world where later
this year we say, "Hey,

1379
01:40:24,176 --> 01:40:26,957
this AI bioterrorism
stuff is really serious.

1380
01:40:27,040 --> 01:40:31,517
We should do something about it.
We should put it in a federal standard.

1381
01:40:31,600 --> 01:40:34,817
If the federal government won't act,
we should put it in a state standard."

1382
01:40:34,900 --> 01:40:35,917
I could totally see that.

1383
01:40:36,000 --> 01:40:42,120
I'm concerned about a world where if you
just consider the pace of progress you're

1384
01:40:42,203 --> 01:40:42,877
expecting,

1385
01:40:42,960 --> 01:40:48,397
the life cycle of legislation...
The benefits are, as you say because

1386
01:40:48,480 --> 01:40:55,480
of diffusion lag, slow enough that I
really do think this patchwork of state

1387
01:40:55,920 --> 01:40:57,817
laws, on the current
trajectory, would prohibit.

1388
01:40:57,901 --> 01:40:59,757
I mean if having
an emotional chatbot friend is

1389
01:40:59,840 --> 01:41:03,288
something that freaks people out, then
just imagine the kinds of actual benefits

1390
01:41:03,371 --> 01:41:05,942
from AI we want normal people
to be able to experience.

1391
01:41:06,025 --> 01:41:08,237
From improvements
in health and healthspan and

1392
01:41:08,320 --> 01:41:12,280
improvements in mental health and so
forth. Whereas at the same time,

1393
01:41:12,364 --> 01:41:13,277
it seems like you

1394
01:41:13,360 --> 01:41:18,116
think the dangers are already on the
horizon and I just don't see that much…

1395
01:41:18,199 --> 01:41:19,517
It seems like it would

1396
01:41:19,600 --> 01:41:23,997
be especially injurious to the benefits
of AI as compared to the dangers of AI.

1397
01:41:24,080 --> 01:41:27,597
So that's maybe where the cost
benefit makes less sense to me.

1398
01:41:27,680 --> 01:41:31,197
So there's a few things here.
People talk about there being

1399
01:41:31,280 --> 01:41:34,317
thousands of these state laws.
First of all, the vast,

1400
01:41:34,400 --> 01:41:40,957
vast majority of them do not pass.
The world works a certain way in theory,

1401
01:41:41,040 --> 01:41:44,557
but just because a law has been passed
doesn't mean it's really enforced.

1402
01:41:44,640 --> 01:41:48,797
The people implementing it may be
like, "Oh my God, this is stupid.

1403
01:41:48,880 --> 01:41:54,800
It would mean shutting off everything
that's ever been built in Tennessee."

1404
01:41:55,440 --> 01:42:01,625
Very often, laws are interpreted
in a way that makes them not as dangerous

1405
01:42:01,708 --> 01:42:02,557
or harmful.

1406
01:42:02,640 --> 01:42:06,023
On the same side, of course, you have
to worry if you're passing a law to stop

1407
01:42:06,107 --> 01:42:06,557
a bad thing;

1408
01:42:06,640 --> 01:42:13,640
you have this problem as well.
My basic view is that if we could

1409
01:42:16,400 --> 01:42:21,128
decide what laws were passed and how
things were done—and we’re only one small

1410
01:42:21,212 --> 01:42:21,437
input

1411
01:42:21,520 --> 01:42:28,160
into that—I would deregulate a lot of the
stuff around the health benefits of AI.

1412
01:42:29,840 --> 01:42:33,477
I don't worry as much
about the chatbot laws.

1413
01:42:33,560 --> 01:42:37,197
I actually worry more
about the drug approval

1414
01:42:37,280 --> 01:42:44,280
process, where I think AI models are going
to greatly accelerate the rate at which we

1415
01:42:44,372 --> 01:42:47,843
discover drugs,
and the pipeline will get jammed up.

1416
01:42:47,926 --> 01:42:50,797
The pipeline will not
be prepared to process

1417
01:42:50,880 --> 01:42:57,880
all the stuff that's going through it.
I think reform of the regulatory process

1418
01:42:58,720 --> 01:43:02,007
should bias more towards the fact
that we have a lot of things coming where

1419
01:43:02,091 --> 01:43:02,637
the safety and

1420
01:43:02,720 --> 01:43:09,262
efficacy is actually going to be really
crisp and clear, a beautiful thing,

1421
01:43:09,345 --> 01:43:11,200
and really effective.

1422
01:43:12,400 --> 01:43:18,808
Maybe we don't need all this
superstructure around it that was designed

1423
01:43:18,891 --> 01:43:21,277
around an era of drugs that

1424
01:43:21,360 --> 01:43:25,267
barely work and often have serious
side effects. At the same time,

1425
01:43:25,351 --> 01:43:26,477
I think we should be

1426
01:43:26,560 --> 01:43:33,560
ramping up quite significantly
the safety and security legislation.

1427
01:43:35,520 --> 01:43:41,939
Like I've said, starting with transparency
is my view of trying not to hamper

1428
01:43:42,022 --> 01:43:43,037
the industry,

1429
01:43:43,120 --> 01:43:46,717
trying to find the right balance. I'm
worried about it. Some people criticize

1430
01:43:46,800 --> 01:43:50,877
my essay for saying, "That's too slow.
The dangers of AI will come too soon

1431
01:43:50,960 --> 01:43:52,717
if we do that." Well, basically,

1432
01:43:52,800 --> 01:43:57,663
I think the last six months and maybe
the next few months are going to be about

1433
01:43:57,746 --> 01:44:02,797
transparency. Then, if these risks emerge
when we're more certain of them—which

1434
01:44:02,880 --> 01:44:07,837
I think we might be as soon as later this
year—then I think we need to act very fast

1435
01:44:07,920 --> 01:44:10,437
in the areas where we've
actually seen the risk.

1436
01:44:10,520 --> 01:44:13,037
I think the only way
to do this is to be nimble.

1437
01:44:13,120 --> 01:44:20,120
Now, the legislative process is normally
not nimble, but we need to emphasize the

1438
01:44:21,280 --> 01:44:23,769
urgency of this to everyone involved.
That's why I'm sending this message

1439
01:44:23,852 --> 01:44:27,552
of urgency. That's why I wrote
Adolescence of Technology.

1440
01:44:27,635 --> 01:44:30,797
I wanted policymakers,
economists, national

1441
01:44:30,880 --> 01:44:35,214
security professionals, and
decision-makers to read it so that they

1442
01:44:35,298 --> 01:44:36,797
have some hope of acting

1443
01:44:36,880 --> 01:44:42,877
faster than they would have otherwise.
Is there anything you can do or advocate

1444
01:44:42,960 --> 01:44:49,960
that would make it more certain that the
benefits of AI are better instantiated?

1445
01:44:51,040 --> 01:44:54,157
I feel like you have worked
with legislatures to say, "Okay,

1446
01:44:54,240 --> 01:44:56,865
we're going to prevent bioterrorism here.
We're going to increase transparency,

1447
01:44:56,949 --> 01:45:00,413
we're going to increase whistleblower
protection." But I think by default,

1448
01:45:00,497 --> 01:45:01,357
the actual benefits

1449
01:45:01,440 --> 01:45:08,397
we're looking forward to seem very fragile
to different kinds of moral panics or

1450
01:45:08,480 --> 01:45:12,877
political economy problems.
I don't actually agree that

1451
01:45:12,960 --> 01:45:16,957
much regarding the developed world.
I feel like in the developed world,

1452
01:45:17,040 --> 01:45:23,437
markets function pretty well.
When there's a lot of money to

1453
01:45:23,520 --> 01:45:26,591
be made on something and it's clearly
the best available alternative,

1454
01:45:26,674 --> 01:45:27,597
it's actually hard for

1455
01:45:27,680 --> 01:45:32,080
the regulatory system to stop it.
We're seeing that in AI itself.

1456
01:45:33,600 --> 01:45:37,440
A thing I've been trying to fight
for is export controls on chips to China.

1457
01:45:38,640 --> 01:45:41,840
That's in the national
security interest of the US.

1458
01:45:42,720 --> 01:45:49,618
That's squarely within the policy beliefs
of almost everyone in Congress of both

1459
01:45:49,702 --> 01:45:50,535
parties.

1460
01:45:52,320 --> 01:45:55,831
The case is very clear.
The counterarguments against it,

1461
01:45:55,914 --> 01:45:57,840
I'll politely call them fishy.

1462
01:45:59,280 --> 01:46:05,150
Yet it doesn't happen and we sell the
chips because there's so much money riding

1463
01:46:05,233 --> 01:46:06,067
on it.

1464
01:46:08,080 --> 01:46:12,160
That money wants to be made. In that case,
in my opinion, that's a bad thing.

1465
01:46:13,120 --> 01:46:17,776
But it also applies
when it's a good thing.

1466
01:46:17,860 --> 01:46:22,957
So if we're talking about
drugs and benefits of

1467
01:46:23,040 --> 01:46:28,814
the technology, I am not as worried
about those benefits being hampered in

1468
01:46:28,897 --> 01:46:34,914
the developed world. I am a little worried
about them going too slow. As I said,

1469
01:46:34,997 --> 01:46:37,517
I do think we should work to speed

1470
01:46:37,600 --> 01:46:41,117
the approval process in the FDA.
I do think we should fight against

1471
01:46:41,200 --> 01:46:44,919
these chatbot bills that you're
describing. Described individually,

1472
01:46:45,002 --> 01:46:45,997
I'm against them. I

1473
01:46:46,080 --> 01:46:51,256
think they're stupid. But I actually think
the bigger worry is the developing world,

1474
01:46:51,339 --> 01:46:51,757
where we

1475
01:46:51,840 --> 01:46:57,068
don't have functioning markets and where
we often can't build on the technology

1476
01:46:57,151 --> 01:47:01,277
that we've had. I worry more that
those folks will get left behind.

1477
01:47:01,360 --> 01:47:04,877
And I worry that even if the cures are
developed, maybe there's someone in rural

1478
01:47:04,960 --> 01:47:10,477
Mississippi who doesn't get it as well.
That's a smaller version of the concern

1479
01:47:10,560 --> 01:47:14,157
we have in the developing world.
So the things we've been doing

1480
01:47:14,240 --> 01:47:21,240
are working with philanthropists. We
work with folks who deliver medicine and

1481
01:47:26,560 --> 01:47:32,881
health interventions to the developing
world, to sub-Saharan Africa, India,

1482
01:47:32,964 --> 01:47:34,077
Latin America,

1483
01:47:34,160 --> 01:47:39,197
and other developing parts of the world.
That's the thing I think that

1484
01:47:39,280 --> 01:47:42,030
won't happen on its own.
You mentioned export controls.

1485
01:47:42,960 --> 01:47:47,440
Why shouldn't the US and China both have
a "country of geniuses in a data center"?

1486
01:47:48,640 --> 01:47:52,560
Why won’t it happen or why shouldn't
it happen? Why shouldn't it happen.

1487
01:47:54,720 --> 01:48:01,720
If this does happen,
we could have a few situations.

1488
01:48:02,400 --> 01:48:05,277
If we have an offense-dominant
situation, we could have a situation

1489
01:48:05,360 --> 01:48:12,241
like nuclear weapons, but more dangerous.
Either side could easily destroy

1490
01:48:12,325 --> 01:48:13,360
everything.

1491
01:48:14,000 --> 01:48:19,037
We could also have a world where it's
unstable. The nuclear equilibrium is

1492
01:48:19,120 --> 01:48:24,077
stable because it's deterrence.
But let's say there was uncertainty about,

1493
01:48:24,160 --> 01:48:29,178
if the two AIs fought, which AI would win?
That could create instability.

1494
01:48:29,262 --> 01:48:30,157
You often have

1495
01:48:30,240 --> 01:48:33,715
conflict when the two sides have
a different assessment of their likelihood

1496
01:48:33,798 --> 01:48:36,802
of winning. If one side is like,
"Oh yeah, there's a 90% chance I'll win,"

1497
01:48:36,886 --> 01:48:38,136
and the other side thinks

1498
01:48:40,160 --> 01:48:43,437
the same, then a fight is much more
likely. They can't both be right,

1499
01:48:43,520 --> 01:48:46,797
but they can both think that. But this
seems like a fully general argument

1500
01:48:46,880 --> 01:48:52,080
against the diffusion of AI technology.
That's the implication of this world.

1501
01:48:52,800 --> 01:48:55,837
Let me just go on, because I think
we will get diffusion eventually.

1502
01:48:55,920 --> 01:49:00,024
The other concern I have is that
governments will oppress their own people

1503
01:49:00,107 --> 01:49:00,941
with AI.

1504
01:49:04,160 --> 01:49:11,034
I'm worried about a world where you
have a country in which there’s already

1505
01:49:11,117 --> 01:49:12,880
a government that's

1506
01:49:16,240 --> 01:49:21,597
building a high-tech authoritarian state.
To be clear, this is about the government.

1507
01:49:21,680 --> 01:49:24,317
This is not about the people.
We need to find a way for

1508
01:49:24,400 --> 01:49:27,760
people everywhere to benefit.
My worry here is about governments.

1509
01:49:30,080 --> 01:49:33,837
My worry is if the world gets carved up
into two pieces, one of those two pieces

1510
01:49:33,920 --> 01:49:38,880
could be authoritarian or totalitarian in
a way that's very difficult to displace.

1511
01:49:39,440 --> 01:49:42,660
Now, will governments
eventually get powerful AI,

1512
01:49:42,743 --> 01:49:45,357
and is there a risk of authoritarianism?

1513
01:49:45,440 --> 01:49:49,040
Yes. Will governments eventually get
powerful AI, and is there a risk of

1514
01:49:52,080 --> 01:49:58,520
bad equilibria? Yes, I think both things.
But the initial conditions matter.

1515
01:49:58,603 --> 01:50:02,690
At some point, we're going to need
to set up the rules of the road.

1516
01:50:02,773 --> 01:50:05,357
I'm not saying that one
country, either the United

1517
01:50:05,440 --> 01:50:09,124
States or a coalition of democracies—which
I think would be a better setup,

1518
01:50:09,207 --> 01:50:09,677
although it

1519
01:50:09,760 --> 01:50:16,706
requires more international cooperation
than we currently seem to want

1520
01:50:16,790 --> 01:50:19,117
to make—should just say,

1521
01:50:19,200 --> 01:50:22,397
"These are the rules of the road."
There's going to be some negotiation.

1522
01:50:22,480 --> 01:50:26,589
The world is going to have
to grapple with this.

1523
01:50:26,673 --> 01:50:30,957
What I would like is for
the democratic nations of

1524
01:50:31,040 --> 01:50:38,040
the world—those whose governments
represent closer to pro-human values—are

1525
01:50:38,214 --> 01:50:39,197
holding the

1526
01:50:39,280 --> 01:50:44,077
stronger hand and have more leverage
when the rules of the road are set.

1527
01:50:44,160 --> 01:50:47,037
So I'm very concerned about
that initial condition.

1528
01:50:47,120 --> 01:50:51,677
I was re-listening to the interview from
three years ago, and one of the ways it

1529
01:50:51,760 --> 01:50:55,431
aged poorly is that I kept asking
questions assuming there was going to be

1530
01:50:55,514 --> 01:50:59,564
some key fulcrum moment two to three years
from now. In fact, being that far out,

1531
01:50:59,647 --> 01:51:03,537
it just seems like progress continues,
AI improves, AI is more diffused,

1532
01:51:03,621 --> 01:51:05,757
and people will use it for more things.

1533
01:51:05,840 --> 01:51:09,172
It seems like you're imagining a world
in the future where the countries get

1534
01:51:09,256 --> 01:51:12,142
together, and "Here's the rules
of the road, here's the leverage we have,

1535
01:51:12,225 --> 01:51:13,517
and here's the leverage you have."

1536
01:51:13,600 --> 01:51:17,440
But on the current trajectory,
everybody will have more AI.

1537
01:51:18,080 --> 01:51:19,917
Some of that AI will be used
by authoritarian countries.

1538
01:51:20,000 --> 01:51:22,317
Some of that within the authoritarian
countries will be used by private

1539
01:51:22,400 --> 01:51:26,317
actors versus state actors.
It's not clear who will benefit more.

1540
01:51:26,400 --> 01:51:28,468
It's always unpredictable
to tell in advance.

1541
01:51:28,551 --> 01:51:30,237
It seems like the internet privileged

1542
01:51:30,320 --> 01:51:33,120
authoritarian countries more
than you would've expected.

1543
01:51:33,280 --> 01:51:38,717
Maybe AI will be the opposite way around.
I want to better understand what

1544
01:51:38,800 --> 01:51:42,557
you're imagining here.
Just to be precise about it,

1545
01:51:42,640 --> 01:51:47,197
I think the exponential of the underlying
technology will continue as it has before.

1546
01:51:47,280 --> 01:51:51,442
The models get smarter and smarter, even
when they get to a "country of geniuses

1547
01:51:51,525 --> 01:51:52,480
in a data center."

1548
01:51:53,760 --> 01:51:55,997
I think you can continue
to make the model smarter.

1549
01:51:56,080 --> 01:52:01,757
There's a question of getting diminishing
returns on their value in the world.

1550
01:52:01,840 --> 01:52:05,840
How much does it matter after
you've already solved human biology?

1551
01:52:07,520 --> 01:52:10,884
At some point you can do harder,
more abstruse math problems,

1552
01:52:10,968 --> 01:52:12,637
but nothing after that matters.

1553
01:52:12,720 --> 01:52:16,836
Putting that aside, I do think
the exponential will continue,

1554
01:52:16,919 --> 01:52:18,557
but there will be certain

1555
01:52:18,640 --> 01:52:24,397
distinguished points on the exponential.
Companies, individuals, and countries

1556
01:52:24,480 --> 01:52:27,761
will reach those points
at different times.

1557
01:52:27,844 --> 01:52:31,517
In "The Adolescence
of Technology" I talk about:

1558
01:52:31,600 --> 01:52:38,157
Is a nuclear deterrent still
stable in the world of AI?

1559
01:52:38,240 --> 01:52:42,477
I don't know, but that's an example
of one thing we've taken for granted.

1560
01:52:42,560 --> 01:52:47,680
The technology could reach such a level
that we can no longer be certain of it.

1561
01:52:50,160 --> 01:52:55,537
Think of others. There are points
where if you reach a certain level,

1562
01:52:55,621 --> 01:52:57,437
maybe you have offensive

1563
01:52:57,520 --> 01:53:03,765
cyber dominance, and every computer system
is transparent to you after that unless

1564
01:53:03,848 --> 01:53:09,277
the other side has an equivalent defense.
I don't know what the critical moment

1565
01:53:09,360 --> 01:53:11,648
is or if there's a single critical moment.

1566
01:53:11,732 --> 01:53:14,077
But I think there will
be either a critical

1567
01:53:14,160 --> 01:53:18,636
moment, a small number
of critical moments,

1568
01:53:18,719 --> 01:53:22,877
or some critical window where AI confers

1569
01:53:22,960 --> 01:53:29,960
some large advantage from the perspective
of national security, and one country or

1570
01:53:30,320 --> 01:53:36,557
coalition has reached it before others.
I'm not advocating that they just say,

1571
01:53:36,640 --> 01:53:40,880
"Okay, we're in charge now."
That's not how I think about it.

1572
01:53:42,560 --> 01:53:45,357
The other side is always catching up.
There are extreme actions you're not

1573
01:53:45,440 --> 01:53:52,440
willing to take, and it's not right
to take complete control anyway.

1574
01:53:52,560 --> 01:53:57,495
But at the point that happens, people
are going to understand that the world has

1575
01:53:57,578 --> 01:53:57,997
changed.

1576
01:53:58,080 --> 01:54:05,080
There's going to be some negotiation,
implicit or explicit, about what the

1577
01:54:05,520 --> 01:54:12,520
post-AI world order looks like.
My interest is in making that

1578
01:54:14,080 --> 01:54:21,080
negotiation be one in which classical
liberal democracy has a strong hand.

1579
01:54:24,080 --> 01:54:27,357
I want to understand what that better
means, because you say in the essay,

1580
01:54:27,440 --> 01:54:31,866
"Autocracy is simply not a form
of government that people can accept in

1581
01:54:31,949 --> 01:54:33,517
the post-powerful AI age."

1582
01:54:33,600 --> 01:54:39,543
That sounds like you're saying the CCP as
an institution cannot exist after we get

1583
01:54:39,626 --> 01:54:40,459
AGI.

1584
01:54:41,040 --> 01:54:46,664
That seems like a very strong demand,
and it seems to imply a world where

1585
01:54:46,747 --> 01:54:47,837
the leading lab

1586
01:54:47,920 --> 01:54:54,157
or the leading country will be able to—and
by that language, should get to—determine

1587
01:54:54,240 --> 01:55:00,000
how the world is governed or what kinds
of governments are, and are not, allowed.

1588
01:55:02,400 --> 01:55:05,958
I believe that paragraph
said something like,

1589
01:55:06,041 --> 01:55:09,520
"You could take it
even further and say X."

1590
01:55:13,360 --> 01:55:16,110
I wasn't necessarily endorsing
that view. I was saying,

1591
01:55:18,080 --> 01:55:23,939
"Here's a weaker thing that I believe. We
have to worry a lot about authoritarians

1592
01:55:24,023 --> 01:55:27,300
and we should try to check
them and limit their power.

1593
01:55:27,383 --> 01:55:30,317
You could take this much
further and have a more

1594
01:55:30,400 --> 01:55:36,805
interventionist view that says
authoritarian countries with AI are these

1595
01:55:36,888 --> 01:55:38,157
self-fulfilling

1596
01:55:38,240 --> 01:55:42,930
cycles that are very hard to displace,
so you just need to get rid of them from

1597
01:55:43,014 --> 01:55:43,837
the beginning."

1598
01:55:43,920 --> 01:55:49,037
That has exactly all the problems you say.
If you were to make a commitment to

1599
01:55:49,120 --> 01:55:53,357
overthrowing every authoritarian country,
they would take a bunch of actions now

1600
01:55:53,440 --> 01:56:00,440
that could lead to instability.
That just may not be possible.

1601
01:56:02,240 --> 01:56:08,435
But the point I was making that I do
endorse is that it is quite possible

1602
01:56:08,518 --> 01:56:14,971
that... Today, the view, my view, in most
of the Western world is that democracy is

1603
01:56:15,054 --> 01:56:16,237
a better form of

1604
01:56:16,320 --> 01:56:20,957
government than authoritarianism.
But if a country’s authoritarian,

1605
01:56:21,040 --> 01:56:25,280
we don’t react the way we’d react if
they committed a genocide or something.

1606
01:56:27,360 --> 01:56:31,290
I guess what I'm saying is I'm
a little worried that in the age of AGI,

1607
01:56:31,373 --> 01:56:35,073
authoritarianism will have a different
meaning. It will be a graver thing.

1608
01:56:35,920 --> 01:56:39,837
We have to decide one way
or another how to deal with that.

1609
01:56:39,920 --> 01:56:45,757
The interventionist view is one possible
view. I was exploring such views.

1610
01:56:45,840 --> 01:56:47,597
It may end up being the

1611
01:56:47,680 --> 01:56:53,014
right view, or it may end up being
too extreme. But I do have hope.

1612
01:56:53,097 --> 01:56:55,197
One piece of hope I have is

1613
01:56:55,280 --> 01:57:00,797
that we have seen that as new
technologies are invented,

1614
01:57:00,880 --> 01:57:04,397
forms of government become obsolete.

1615
01:57:04,480 --> 01:57:10,557
I mentioned this in "Adolescence
of Technology", where I said feudalism

1616
01:57:10,640 --> 01:57:17,282
was basically a form of government,
and when we invented industrialization,

1617
01:57:17,365 --> 01:57:21,831
feudalism was no longer sustainable.
It no longer made sense. Why is that hope?

1618
01:57:21,914 --> 01:57:26,240
Couldn't that imply that democracy is no
longer going to be a competitive system?

1619
01:57:26,880 --> 01:57:33,880
Right, it could go either way.
But these problems with

1620
01:57:38,160 --> 01:57:44,237
authoritarianism get deeper.
I wonder if that's an indicator of

1621
01:57:44,320 --> 01:57:49,648
other problems that authoritarianism
will have. In other words,

1622
01:57:49,731 --> 01:57:52,397
because authoritarianism becomes

1623
01:57:52,480 --> 01:57:58,160
worse, people are more afraid of it.
They work harder to stop it.

1624
01:57:59,440 --> 01:58:03,538
You have to think in terms
of total equilibrium.

1625
01:58:03,621 --> 01:58:07,197
I just wonder if it will motivate new ways

1626
01:58:07,280 --> 01:58:13,597
of thinking about how to preserve and
protect freedom with the new technology.

1627
01:58:13,680 --> 01:58:20,680
Even more optimistically, will it lead to
a collective reckoning and a more emphatic

1628
01:58:22,240 --> 01:58:26,720
realization of how important some of the
things we take as individual rights are?

1629
01:58:27,360 --> 01:58:32,637
A more emphatic realization that
we really can't give these away.

1630
01:58:32,720 --> 01:58:36,160
We've seen there's no other
way to live that actually works.

1631
01:58:39,520 --> 01:58:46,477
I am actually hopeful that—it sounds too
idealistic, but I believe it could be the

1632
01:58:46,560 --> 01:58:49,833
case—dictatorships
become morally obsolete.

1633
01:58:49,916 --> 01:58:52,877
They become morally unworkable forms of

1634
01:58:52,960 --> 01:58:59,960
government and the crisis that that
creates is sufficient to force us to find

1635
01:59:00,850 --> 01:59:02,080
another way.

1636
01:59:03,680 --> 01:59:06,705
I think there is genuinely a tough
question here which I'm not sure how you

1637
01:59:06,788 --> 01:59:07,622
resolve.

1638
01:59:07,920 --> 01:59:11,070
We've had to come out one way
or another on it through history.

1639
01:59:11,760 --> 01:59:15,277
With China in the '70s and '80s,
we decided that even though it's an

1640
01:59:15,360 --> 01:59:17,057
authoritarian system,
we will engage with it.

1641
01:59:17,140 --> 01:59:18,877
I think in retrospect
that was the right call,

1642
01:59:18,960 --> 01:59:22,566
because it’s a state authoritarian
system but a billion-plus people are much

1643
01:59:22,649 --> 01:59:25,197
wealthier and better off than
they would've otherwise been.

1644
01:59:25,280 --> 01:59:27,197
It's not clear that it
would've stopped being

1645
01:59:27,280 --> 01:59:30,397
an authoritarian country otherwise.
You can just look at North Korea

1646
01:59:30,480 --> 01:59:33,760
as an example of that.
I don't know if it takes

1647
01:59:34,400 --> 01:59:39,019
that much intelligence to remain
an authoritarian country that continues

1648
01:59:39,102 --> 01:59:40,717
to coalesce its own power.

1649
01:59:40,800 --> 01:59:44,477
You can imagine a North Korea with an AI
that's much worse than everybody else's,

1650
01:59:44,560 --> 01:59:49,917
but still enough to keep power.
In general, it seems like we should just

1651
01:59:50,000 --> 01:59:54,637
have this attitude that the benefits of
AI—in the form of all these empowerments

1652
01:59:54,720 --> 02:00:00,237
of humanity and health—will be big.
Historically, we have decided it's good

1653
02:00:00,320 --> 02:00:02,571
to spread the benefits
of technology widely,

1654
02:00:02,654 --> 02:00:05,360
even to people whose
governments are authoritarian.

1655
02:00:06,000 --> 02:00:08,557
It is a tough question,
how to think about it with AI,

1656
02:00:08,640 --> 02:00:10,317
but historically we have said, "yes,

1657
02:00:10,400 --> 02:00:14,480
this is a positive-sum world, and it's
still worth diffusing the technology."

1658
02:00:15,280 --> 02:00:20,797
There are a number of choices we have.
Framing this as a government-to-government

1659
02:00:20,880 --> 02:00:27,757
decision in national security terms is one
lens, but there are a lot of other lenses.

1660
02:00:27,840 --> 02:00:31,520
You could imagine a world where we
produce all these cures to diseases.

1661
02:00:32,400 --> 02:00:35,892
The cures are fine to sell
to authoritarian countries,

1662
02:00:35,975 --> 02:00:38,077
but the data centers just aren't.

1663
02:00:38,160 --> 02:00:44,317
The chips and the data centers aren't,
and the AI industry itself isn't.

1664
02:00:44,400 --> 02:00:48,320
Another possibility I think
folks should think about is this.

1665
02:00:49,360 --> 02:00:54,164
Could there be developments we can
make—either that naturally happen as

1666
02:00:54,247 --> 02:00:59,757
a result of AI, or that we could make
happen by building technology on AI—that

1667
02:00:59,840 --> 02:01:05,277
create an equilibrium where it becomes
infeasible for authoritarian countries

1668
02:01:05,360 --> 02:01:09,600
to deny their people private use
of the benefits of the technology?

1669
02:01:12,400 --> 02:01:16,664
Are there equilibria where we can give
everyone in an authoritarian country

1670
02:01:16,747 --> 02:01:18,080
their own AI model that

1671
02:01:19,600 --> 02:01:24,327
defends them from surveillance and there
isn't a way for the authoritarian country

1672
02:01:24,411 --> 02:01:27,328
to crack down on this while
retaining power? I don't know.

1673
02:01:27,412 --> 02:01:29,757
That sounds to me like
if that went far enough,

1674
02:01:29,840 --> 02:01:34,268
it would be a reason why authoritarian
countries would disintegrate from

1675
02:01:34,351 --> 02:01:38,125
the inside. But maybe there's a middle
world where there's an equilibrium where,

1676
02:01:38,208 --> 02:01:39,197
if they want to hold on

1677
02:01:39,280 --> 02:01:45,357
to power, the authoritarians can't deny
individualized access to the technology.

1678
02:01:45,440 --> 02:01:49,680
But I actually do have a hope
for the more radical version.

1679
02:01:50,880 --> 02:01:54,797
Is it possible that the technology
might inherently have properties—or

1680
02:01:54,880 --> 02:02:01,117
that by building on it in certain ways
we could create properties—that have this

1681
02:02:01,200 --> 02:02:04,640
dissolving effect
on authoritarian structures? Now,

1682
02:02:04,724 --> 02:02:07,197
we hoped originally—think back to the

1683
02:02:07,280 --> 02:02:13,117
beginning of the Obama administration—that
social media and the internet would have

1684
02:02:13,200 --> 02:02:19,997
that property, and it turns out not to.
But what if we could try again with the

1685
02:02:20,080 --> 02:02:21,882
knowledge of how many
things could go wrong,

1686
02:02:21,966 --> 02:02:23,597
and that this is a different technology?

1687
02:02:23,680 --> 02:02:26,280
I don't know if it would work,
but it's worth a try.

1688
02:02:26,480 --> 02:02:30,077
It's just very unpredictable.
There are first principles reasons why

1689
02:02:30,160 --> 02:02:35,197
authoritarianism might be privileged.
It's all very unpredictable. We just

1690
02:02:35,280 --> 02:02:40,797
have to recognize the problem and come
up with 10 things we can try, try those,

1691
02:02:40,880 --> 02:02:43,484
and then assess which
ones are working, if any.

1692
02:02:43,568 --> 02:02:46,287
Then try new ones if
the old ones aren't working.

1693
02:02:46,370 --> 02:02:51,357
But I guess that nets out to today, as you
say, that we will not sell data centers,

1694
02:02:51,440 --> 02:02:56,006
or chips, and the ability to make
chips to China. So in some sense,

1695
02:02:56,089 --> 02:02:58,157
you are denying… There would be

1696
02:02:58,240 --> 02:03:01,206
some benefits to the Chinese economy,
Chinese people, et cetera,

1697
02:03:01,289 --> 02:03:02,397
because we're doing that.

1698
02:03:02,480 --> 02:03:05,128
Then there'd also be benefits
to the American economy because it's

1699
02:03:05,211 --> 02:03:05,997
a positive-sum world.

1700
02:03:06,080 --> 02:03:08,637
We could trade. They could have their
country's data centers doing one thing.

1701
02:03:08,720 --> 02:03:14,717
We could have ours doing another.
Already, you're saying it's not worth that

1702
02:03:14,800 --> 02:03:18,894
positive-sum stipend
to empower those countries?

1703
02:03:18,978 --> 02:03:22,637
What I would say
is that we are about to be

1704
02:03:22,720 --> 02:03:27,757
in a world where growth and economic
value will come very easily if we're

1705
02:03:27,840 --> 02:03:34,840
able to build these powerful AI models.
What will not come easily is distribution

1706
02:03:35,040 --> 02:03:39,680
of benefits, distribution
of wealth, political freedom.

1707
02:03:40,720 --> 02:03:43,117
These are the things that are
going to be hard to achieve.

1708
02:03:43,200 --> 02:03:49,695
So when I think about policy, I think that
the technology and the market will deliver

1709
02:03:49,778 --> 02:03:52,648
all the fundamental benefits,
this is my fundamental belief,

1710
02:03:52,731 --> 02:03:54,531
almost faster than we can take them.

1711
02:03:55,840 --> 02:04:01,438
These questions about distribution and
political freedom and rights are the ones

1712
02:04:01,522 --> 02:04:02,717
that will actually

1713
02:04:02,800 --> 02:04:06,317
matter and that policy should focus on.
Speaking of distribution, as you were

1714
02:04:06,400 --> 02:04:12,797
mentioning, we have developing countries.
In many cases, catch-up growth has been

1715
02:04:12,880 --> 02:04:15,357
weaker than we would have hoped for.
But when catch-up growth does happen,

1716
02:04:15,440 --> 02:04:18,397
it's fundamentally because
they have underutilized labor.

1717
02:04:18,480 --> 02:04:21,197
We can bring the capital and know-how from
developed countries to these countries,

1718
02:04:21,280 --> 02:04:26,557
and then they can grow quite rapidly.
Obviously, in a world where labor is no

1719
02:04:26,640 --> 02:04:30,077
longer the constraining factor,
this mechanism no longer works.

1720
02:04:30,160 --> 02:04:33,517
So is the hope basically
to rely on philanthropy from

1721
02:04:33,600 --> 02:04:37,840
the people or countries who immediately
get wealthy from AI? What is the hope?

1722
02:04:38,880 --> 02:04:44,317
Philanthropy should obviously play
some role, as it has in the past.

1723
02:04:44,400 --> 02:04:49,920
But I think growth is always better
and stronger if we can make it endogenous.

1724
02:04:50,640 --> 02:04:54,720
What are the relevant
industries in an AI-driven world?

1725
02:04:58,080 --> 02:05:00,797
I said we shouldn't build data centers in
China, but there's no reason we shouldn't

1726
02:05:00,880 --> 02:05:03,997
build data centers in Africa.
In fact, I think it'd be

1727
02:05:04,080 --> 02:05:07,930
great to build data centers in Africa.
As long as they're not owned by China,

1728
02:05:08,960 --> 02:05:13,600
we should build data centers in Africa.
I think that's a great thing to do.

1729
02:05:16,320 --> 02:05:21,360
There's no reason we can't build a
pharmaceutical industry that's AI-driven.

1730
02:05:22,960 --> 02:05:28,797
If AI is accelerating drug discovery, then
there will be a bunch of biotech startups.

1731
02:05:28,880 --> 02:05:31,197
Let's make sure some of those
happen in the developing world.

1732
02:05:31,280 --> 02:05:34,877
Certainly, during the transition—we can
talk about the point where humans have no

1733
02:05:34,960 --> 02:05:39,151
role—humans will still have some
role in starting up these companies

1734
02:05:39,234 --> 02:05:41,037
and supervising the AI models.

1735
02:05:41,120 --> 02:05:44,077
So let's make sure some of those
humans are in the developing world

1736
02:05:44,160 --> 02:05:46,051
so that fast growth can
happen there as well.

1737
02:05:46,134 --> 02:05:48,157
You guys recently announced
that Claude is going

1738
02:05:48,240 --> 02:05:50,995
to have a constitution that's
aligned to a set of values,

1739
02:05:51,078 --> 02:05:53,037
and not necessarily just to the end user.

1740
02:05:53,120 --> 02:05:55,997
There's a world I can imagine where
if it is aligned to the end user,

1741
02:05:56,080 --> 02:05:59,040
it preserves the balance of power we have
in the world today because everybody gets

1742
02:05:59,123 --> 02:06:03,597
to have their own AI that's advocating
for them. The ratio of bad actors to

1743
02:06:03,680 --> 02:06:07,130
good actors stays constant.
It seems to work out for our world today.

1744
02:06:07,360 --> 02:06:12,797
Why is it better not to do that, but to
have a specific set of values that the

1745
02:06:12,880 --> 02:06:16,637
AI should carry forward?
I'm not sure I'd quite

1746
02:06:16,720 --> 02:06:21,600
draw the distinction in that way. There
may be two relevant distinctions here.

1747
02:06:22,400 --> 02:06:25,495
I think you're talking about
a mix of the two. One is,

1748
02:06:25,578 --> 02:06:27,437
should we give the model a set of

1749
02:06:27,520 --> 02:06:31,437
instructions about "do
this" versus "don't do this"?

1750
02:06:31,520 --> 02:06:37,440
The other is, should we give the model
a set of principles for how to act?

1751
02:06:44,080 --> 02:06:48,797
It's kind of purely a practical
and empirical thing that we've observed.

1752
02:06:48,880 --> 02:06:52,877
By teaching the model principles,
getting it to learn from principles,

1753
02:06:52,960 --> 02:06:56,807
its behavior is more consistent,
it's easier to cover edge cases,

1754
02:06:56,890 --> 02:06:58,077
and the model is more

1755
02:06:58,160 --> 02:07:05,160
likely to do what people want it to do.
In other words, if you give it a list of

1756
02:07:09,600 --> 02:07:10,717
rules—"don't tell people how to hot-wire
a car, don't speak in Korean"—it doesn't

1757
02:07:10,800 --> 02:07:14,320
really understand the rules,
and it's hard to generalize from them.

1758
02:07:15,440 --> 02:07:20,720
It’s just a list of do’s and don’t’s.
Whereas if you give it principles—it

1759
02:07:21,440 --> 02:07:25,277
has some hard guardrails like "Don't make
biological weapons" but—overall you're

1760
02:07:25,360 --> 02:07:28,719
trying to understand what
it should be aiming to do,

1761
02:07:28,803 --> 02:07:31,037
how it should be aiming to operate.

1762
02:07:31,120 --> 02:07:34,906
So just from a practical perspective,
that turns out to be a more effective way

1763
02:07:34,989 --> 02:07:38,895
to train the model. That's the rules
versus principles trade-off.

1764
02:07:38,978 --> 02:07:42,157
Then there's another
thing you're talking about,

1765
02:07:42,240 --> 02:07:49,240
which is the corrigibility versus
intrinsic motivation trade-off.

1766
02:07:51,680 --> 02:07:58,680
How much should the model be a kind
of "skin suit" where it just directly

1767
02:08:02,000 --> 02:08:05,850
follows the instructions given to it
by whoever is giving those instructions,

1768
02:08:06,560 --> 02:08:11,124
versus how much should the model have an
inherent set of values and go off and do

1769
02:08:11,207 --> 02:08:12,240
things on its own?

1770
02:08:14,320 --> 02:08:20,103
There I would actually say everything
about the model is closer to the direction

1771
02:08:20,187 --> 02:08:21,020
that

1772
02:08:21,760 --> 02:08:24,477
it should mostly do what people want.
It should mostly follow instructions.

1773
02:08:24,560 --> 02:08:29,277
We're not trying to build something that
goes off and runs the world on its own.

1774
02:08:29,360 --> 02:08:32,238
We're actually pretty far
on the corrigible side. Now,

1775
02:08:32,321 --> 02:08:34,157
what we do say is there are certain

1776
02:08:34,240 --> 02:08:40,477
things that the model won't do.
I think we say it in various ways in the

1777
02:08:40,560 --> 02:08:42,692
constitution, that under
normal circumstances,

1778
02:08:42,776 --> 02:08:45,197
if someone asks the model
to do a task, it should do

1779
02:08:45,280 --> 02:08:50,330
that task. That should be the default.
But if you've asked it to do something

1780
02:08:50,413 --> 02:08:51,280
dangerous, or

1781
02:08:54,320 --> 02:09:01,197
to harm someone else,
then the model is unwilling to do that.

1782
02:09:01,280 --> 02:09:07,597
So I actually think of it as a mostly
corrigible model that has some limits,

1783
02:09:07,680 --> 02:09:11,997
but those limits are based on principles.
Then the fundamental question is,

1784
02:09:12,080 --> 02:09:15,517
how are those principles determined? This
is not a special question for Anthropic.

1785
02:09:15,600 --> 02:09:18,724
This would be a question
for any AI company.

1786
02:09:18,807 --> 02:09:22,077
But because you have
been the ones to actually

1787
02:09:22,160 --> 02:09:25,277
write down the principles,
I get to ask you this question.

1788
02:09:25,360 --> 02:09:29,437
Normally, a constitution is written down,
set in stone, and there's a process of

1789
02:09:29,520 --> 02:09:34,477
updating it and changing it and so forth.
In this case, it seems like a document

1790
02:09:34,560 --> 02:09:37,597
that people at Anthropic write,
that can be changed at any time,

1791
02:09:37,680 --> 02:09:42,909
that guides the behavior of systems
that are going to be the basis of a lot

1792
02:09:42,992 --> 02:09:44,480
of economic activity.

1793
02:09:45,680 --> 02:09:49,600
How do you think about how
those principles should be set?

1794
02:09:50,880 --> 02:09:57,880
I think there are maybe three sizes
of loop here, three ways to iterate.

1795
02:09:58,160 --> 02:10:01,837
One is we iterate within Anthropic. We
train the model, we're not happy with it,

1796
02:10:01,920 --> 02:10:05,600
and we change the constitution.
I think that's good to do.

1797
02:10:06,480 --> 02:10:10,527
Putting out public updates to the
constitution every once in a while

1798
02:10:10,610 --> 02:10:15,940
is good because people can comment on it.
The second level of loop is different

1799
02:10:16,023 --> 02:10:19,501
companies having different constitutions.
I think it’s useful.

1800
02:10:19,585 --> 02:10:21,427
Anthropic puts out a constitution,

1801
02:10:21,510 --> 02:10:28,510
Gemini puts out a constitution, and other
companies put out a constitution.

1802
02:10:28,800 --> 02:10:34,160
People can look at them and compare.
Outside observers can critique and say,

1803
02:10:34,800 --> 02:10:39,200
"I like this thing from this constitution
and this thing from that constitution."

1804
02:10:40,160 --> 02:10:45,677
That creates a soft incentive
and feedback for all the companies to

1805
02:10:45,760 --> 02:10:49,880
take the best of each element and improve.
Then I think there's a third loop,

1806
02:10:49,963 --> 02:10:50,317
which is

1807
02:10:50,400 --> 02:10:57,400
society beyond the AI companies and beyond
just those who comment without hard power.

1808
02:10:59,280 --> 02:11:02,322
There we've done some experiments.
A couple years ago,

1809
02:11:02,405 --> 02:11:04,637
we did an experiment with the Collective

1810
02:11:04,720 --> 02:11:10,653
Intelligence Project to basically poll
people and ask them what should be in

1811
02:11:10,737 --> 02:11:12,320
our AI constitution.

1812
02:11:15,200 --> 02:11:17,277
At the time, we incorporated
some of those changes.

1813
02:11:17,360 --> 02:11:19,517
So you could imagine doing something
like that with the new approach we've

1814
02:11:19,600 --> 02:11:23,757
taken to the constitution.
It's a little harder because

1815
02:11:23,840 --> 02:11:28,560
it was an easier approach to take when the
constitution was a list of dos and don'ts.

1816
02:11:29,120 --> 02:11:32,797
At the level of principles, it has
to have a certain amount of coherence.

1817
02:11:32,880 --> 02:11:37,837
But you could still imagine getting
views from a wide variety of people.

1818
02:11:37,920 --> 02:11:41,997
You could also imagine—and this
is a crazy idea, but this whole

1819
02:11:42,080 --> 02:11:49,080
interview is about crazy ideas—systems
of representative government having input.

1820
02:11:52,320 --> 02:11:55,437
I wouldn't do this today because
the legislative process is so slow.

1821
02:11:55,520 --> 02:11:59,142
This is exactly why I think we should
be careful about the legislative process

1822
02:11:59,225 --> 02:12:03,867
and AI regulation. But there's no
reason you couldn't, in principle, say,

1823
02:12:03,950 --> 02:12:06,877
"All AI models have to have a constitution

1824
02:12:06,960 --> 02:12:12,176
that starts with these things, and then
you can append other things after it,

1825
02:12:12,259 --> 02:12:13,277
but there has to

1826
02:12:13,360 --> 02:12:17,268
be this special section that takes
precedence." I wouldn't do that.

1827
02:12:17,351 --> 02:12:18,960
That's too rigid and sounds

1828
02:12:22,160 --> 02:12:26,237
overly prescriptive in a way that I
think overly aggressive legislation is.

1829
02:12:26,320 --> 02:12:32,637
But that is a thing you could try to do.
Is there some much less heavy-handed

1830
02:12:32,720 --> 02:12:36,240
version of that? Maybe.
I really like control loop two.

1831
02:12:37,840 --> 02:12:41,890
Obviously, this is not how constitutions
of actual governments do or should work.

1832
02:12:42,480 --> 02:12:46,877
There's not this vague sense in which
the Supreme Court will feel out how people

1833
02:12:46,960 --> 02:12:50,510
are feeling—what are the vibes—and
update the constitution accordingly.

1834
02:12:50,800 --> 02:12:54,480
With actual governments, there's
a more formal, procedural process.

1835
02:12:55,040 --> 02:12:59,017
But you have a vision
of competition between constitutions,

1836
02:12:59,100 --> 02:13:01,357
which is actually very reminiscent

1837
02:13:01,440 --> 02:13:05,006
of how some libertarian charter
cities people used to talk,

1838
02:13:05,089 --> 02:13:07,357
about what an archipelago of different

1839
02:13:07,440 --> 02:13:10,397
kinds of governments would look like.
There would be selection among them of

1840
02:13:10,480 --> 02:13:14,480
who could operate the most effectively
and where people would be the happiest.

1841
02:13:15,200 --> 02:13:21,840
In a sense, you're recreating that
vision of a utopia of archipelagos.

1842
02:13:23,360 --> 02:13:29,284
I think that vision has things to
recommend it and things that will go wrong

1843
02:13:29,368 --> 02:13:30,201
with it.

1844
02:13:31,760 --> 02:13:34,637
It's an interesting, in some ways
compelling, vision, but things will

1845
02:13:34,720 --> 02:13:40,477
go wrong that you hadn't imagined.
So I like loop two as well,

1846
02:13:40,560 --> 02:13:46,317
but I feel like the whole thing has got to
be some mix of loops one, two, and three,

1847
02:13:46,400 --> 02:13:51,600
and it's a matter of the proportions.
I think that's gotta be the answer.

1848
02:13:53,040 --> 02:13:57,718
When somebody eventually writes
the equivalent of The Making of the Atomic

1849
02:13:57,802 --> 02:13:58,877
Bomb for this era,

1850
02:13:58,960 --> 02:14:02,477
what is the thing that will be hardest
to glean from the historical record that

1851
02:14:02,560 --> 02:14:06,877
they're most likely to miss?
I think a few things. One is,

1852
02:14:06,960 --> 02:14:11,277
at every moment of this exponential,
the extent to which the world outside it

1853
02:14:11,360 --> 02:14:15,162
didn't understand it. This is a bias
that's often present in history.

1854
02:14:15,246 --> 02:14:17,277
Anything that actually happened looks

1855
02:14:17,360 --> 02:14:24,157
inevitable in retrospect.
When people look back, it will

1856
02:14:24,240 --> 02:14:30,924
be hard for them to put themselves in the
place of people who were actually making

1857
02:14:31,007 --> 02:14:31,997
a bet on this

1858
02:14:32,080 --> 02:14:36,677
thing to happen that wasn't inevitable,
that we had these arguments like

1859
02:14:36,760 --> 02:14:38,237
the arguments I make for

1860
02:14:38,320 --> 02:14:43,811
scaling or that continual
learning will be solved.

1861
02:14:43,894 --> 02:14:48,717
Some of us internally
put a high probability

1862
02:14:48,800 --> 02:14:55,800
on this happening, but there's a world
outside us that's not acting on that

1863
02:14:55,898 --> 02:14:56,731
at all.

1864
02:14:58,960 --> 02:15:04,960
I think the weirdness of it,
unfortunately the insularity of it...

1865
02:15:07,520 --> 02:15:10,157
If we're one year or two
years away from it happening,

1866
02:15:10,240 --> 02:15:12,014
the average person
on the street has no idea.

1867
02:15:12,097 --> 02:15:14,077
That's one of the things
I'm trying to change with

1868
02:15:14,160 --> 02:15:18,957
the memos, with talking to policymakers.
I don’t know but I think

1869
02:15:19,040 --> 02:15:26,040
that's just a crazy thing.
Finally, I would say—and this

1870
02:15:27,280 --> 02:15:32,868
probably applies to almost all historical
moments of crisis—how absolutely fast it

1871
02:15:32,952 --> 02:15:33,837
was happening,

1872
02:15:33,920 --> 02:15:39,517
how everything was happening all at once.
Decisions that you might think were

1873
02:15:39,600 --> 02:15:42,637
carefully calculated, well actually
you have to make that decision,

1874
02:15:42,720 --> 02:15:46,787
and then you have to make 30 other
decisions on the same day because it's all

1875
02:15:46,870 --> 02:15:47,757
happening so fast.

1876
02:15:47,840 --> 02:15:51,740
You don't even know which decisions are
going to turn out to be consequential.

1877
02:15:52,320 --> 02:15:59,320
One of my worries—although it's also an
insight into what's happening—is that some

1878
02:16:00,880 --> 02:16:05,084
very critical decision will be some
decision where someone just comes into

1879
02:16:05,167 --> 02:16:05,837
my office and

1880
02:16:05,920 --> 02:16:12,920
is like, "Dario, you have two minutes.
Should we do thing A or thing B on this?"

1881
02:16:14,640 --> 02:16:19,808
Someone gives me this random half-page
memo and asks, "Should we do A or B?"

1882
02:16:19,891 --> 02:16:23,098
I'm like, "I don't know.
I have to eat lunch. Let's do B."

1883
02:16:23,181 --> 02:16:25,997
That ends up being the most
consequential thing ever.

1884
02:16:26,080 --> 02:16:33,080
So final question. There aren't tech
CEOs who are usually writing 50-page memos

1885
02:16:33,848 --> 02:16:35,437
every few months.

1886
02:16:35,520 --> 02:16:39,997
It seems like you have managed to build
a role for yourself and a company around

1887
02:16:40,080 --> 02:16:47,080
you which is compatible with this
more intellectual-type role of CEO.

1888
02:16:47,280 --> 02:16:51,964
I want to understand how you
construct that. How does that work?

1889
02:16:52,047 --> 02:16:53,677
Do you just go away for

1890
02:16:53,760 --> 02:16:56,157
a couple of weeks and then you tell your
company, "This is the memo. Here's what

1891
02:16:56,240 --> 02:16:59,597
we're doing"? It's also reported that
you write a bunch of these internally.

1892
02:16:59,680 --> 02:17:02,380
For this particular one,
I wrote it over winter break.

1893
02:17:04,320 --> 02:17:08,557
I was having a hard time finding
the time to actually write it.

1894
02:17:08,640 --> 02:17:12,986
But I think about this in a broader way.
I think it relates to the culture of

1895
02:17:13,070 --> 02:17:13,677
the company.

1896
02:17:13,760 --> 02:17:18,498
I probably spend a third, maybe 40%, of my
time making sure the culture of Anthropic

1897
02:17:18,581 --> 02:17:18,957
is good.

1898
02:17:19,040 --> 02:17:25,557
As Anthropic has gotten larger, it's
gotten harder to get directly involved

1899
02:17:25,640 --> 02:17:26,877
in the training

1900
02:17:26,960 --> 02:17:30,797
of the models, the launch of the models,
the building of the products. It's 2,500

1901
02:17:30,880 --> 02:17:37,880
people. I have certain instincts, but it's
very difficult to get involved in every

1902
02:17:38,397 --> 02:17:39,680
single detail.

1903
02:17:41,120 --> 02:17:45,721
I try as much as possible, but one thing
that's very leveraged is making sure

1904
02:17:45,804 --> 02:17:46,877
Anthropic is a good

1905
02:17:46,960 --> 02:17:50,795
place to work, people like working there,
everyone thinks of themselves as team

1906
02:17:50,878 --> 02:17:54,547
members, and everyone works together
instead of against each other.

1907
02:17:54,630 --> 02:17:57,197
We've seen as some
of the other AI companies

1908
02:17:57,280 --> 02:18:01,933
have grown—without naming any names—we're
starting to see decoherence and people

1909
02:18:02,016 --> 02:18:03,117
fighting each other.

1910
02:18:03,200 --> 02:18:07,450
I would argue there was even a lot of that
from the beginning, but it's gotten worse.

1911
02:18:08,160 --> 02:18:14,637
I think we've done an extraordinarily good
job, even if not perfect, of holding the

1912
02:18:14,720 --> 02:18:17,650
company together, making
everyone feel the mission,

1913
02:18:17,734 --> 02:18:19,837
that we're sincere about the mission,

1914
02:18:19,920 --> 02:18:23,503
and that everyone has faith that everyone
else there is working for the right

1915
02:18:23,587 --> 02:18:27,495
reason. That we're a team, that people
aren't trying to get ahead at each other's

1916
02:18:27,579 --> 02:18:27,997
expense or

1917
02:18:28,080 --> 02:18:32,280
backstab each other, which again, I think
happens a lot at some of the other places.

1918
02:18:33,200 --> 02:18:36,477
How do you make that the case?
It's a lot of things. It's me,

1919
02:18:36,560 --> 02:18:41,037
it's Daniela, who runs the company
day to day, it's the co-founders,

1920
02:18:41,120 --> 02:18:44,317
it's the other people we hire,
it's the environment we try to create.

1921
02:18:44,400 --> 02:18:51,400
But I think an important thing in the
culture is that the other leaders as well,

1922
02:18:52,106 --> 02:18:53,757
but especially me,

1923
02:18:53,840 --> 02:18:58,797
have to articulate what the company
is about, why it's doing what it's doing,

1924
02:18:58,880 --> 02:19:02,665
what its strategy is, what its
values are, what its mission is,

1925
02:19:02,748 --> 02:19:04,160
and what it stands for.

1926
02:19:06,320 --> 02:19:09,837
When you get to 2,500 people,
you can't do that person by person.

1927
02:19:09,920 --> 02:19:12,477
You have to write, or you have
to speak to the whole company.

1928
02:19:12,560 --> 02:19:16,695
This is why I get up in front of the whole
company every two weeks and speak for

1929
02:19:16,778 --> 02:19:17,612
an hour.

1930
02:19:18,320 --> 02:19:22,317
I wouldn't say I write essays internally.
I do two things. One, I write this thing

1931
02:19:22,400 --> 02:19:27,277
called a DVQ, Dario Vision Quest.
I wasn't the one who named it that.

1932
02:19:27,360 --> 02:19:31,112
That's the name it received, and it's
one of these names that I tried to fight

1933
02:19:31,195 --> 02:19:35,723
because it made it sound like I was going
off and smoking peyote or something.

1934
02:19:35,806 --> 02:19:37,997
But the name just stuck. So I get up

1935
02:19:38,080 --> 02:19:42,957
in front of the company every two weeks.
I have a three or four-page document,

1936
02:19:43,040 --> 02:19:47,454
and I just talk through three or four
different topics about what's going

1937
02:19:47,537 --> 02:19:48,400
on internally,

1938
02:19:49,040 --> 02:19:53,997
the models we're producing, the products,
the outside industry, the world as a whole

1939
02:19:54,080 --> 02:19:59,757
as it relates to AI and geopolitically
in general. Just some mix of that. I go

1940
02:19:59,840 --> 02:20:04,111
through very honestly and I say,
"This is what I'm thinking,

1941
02:20:04,195 --> 02:20:10,004
and this is what Anthropic leadership
is thinking," and then I answer questions.

1942
02:20:10,087 --> 02:20:13,517
That direct connection
has a lot of value that

1943
02:20:13,600 --> 02:20:18,240
is hard to achieve when you're passing
things down the chain six levels deep.

1944
02:20:19,040 --> 02:20:26,040
A large fraction of the company comes
to attend, either in person or virtually.

1945
02:20:27,360 --> 02:20:30,159
It really means that you
can communicate a lot.

1946
02:20:30,243 --> 02:20:32,797
The other thing I do
is I have a channel in

1947
02:20:32,880 --> 02:20:35,930
Slack where I just write a bunch
of things and comment a lot.

1948
02:20:36,720 --> 02:20:42,122
Often that's in response to things I'm
seeing at the company or questions people

1949
02:20:42,206 --> 02:20:43,039
ask.

1950
02:20:44,320 --> 02:20:47,750
We do internal surveys and there are
things people are concerned about,

1951
02:20:47,833 --> 02:20:49,133
and so I'll write them up.

1952
02:20:50,640 --> 02:20:56,077
I'm just very honest about these things.
I just say them very directly.

1953
02:20:56,160 --> 02:21:00,432
The point is to get a reputation of
telling the company the truth about what's

1954
02:21:00,515 --> 02:21:04,551
happening, to call things what
they are, to acknowledge problems,

1955
02:21:04,634 --> 02:21:07,597
to avoid the sort
of corpo speak, the kind of

1956
02:21:07,680 --> 02:21:12,697
defensive communication that often
is necessary in public because the world

1957
02:21:12,780 --> 02:21:18,002
is very large and full of people who
are interpreting things in bad faith.

1958
02:21:18,085 --> 02:21:21,757
But if you have a company
of people who you trust,

1959
02:21:21,840 --> 02:21:28,617
and we try to hire people that we trust,
then you can really just be entirely

1960
02:21:28,700 --> 02:21:29,680
unfiltered.

1961
02:21:31,600 --> 02:21:33,517
I think that's an enormous
strength of the company.

1962
02:21:33,600 --> 02:21:37,357
It makes it a better place to work,
it makes people more than the sum of

1963
02:21:37,440 --> 02:21:37,997
their parts,

1964
02:21:38,080 --> 02:21:40,749
and increases the likelihood that we
accomplish the mission because everyone

1965
02:21:40,832 --> 02:21:41,437
is on the same page

1966
02:21:41,520 --> 02:21:44,779
about the mission, and everyone
is debating and discussing how best

1967
02:21:44,863 --> 02:21:46,077
to accomplish the mission.

1968
02:21:46,160 --> 02:21:49,917
Well, in lieu of an external Dario
Vision Quest, we have this interview.

1969
02:21:50,000 --> 02:21:54,557
This interview is a little like that. This
has been fun, Dario. Thanks for doing it.

1970
02:21:54,640 --> 02:22:00,960
Thank you, Dwarkesh.