WEBVTT

00:00:00.640 --> 00:00:04.212
We talked three years ago. In your view,
what has been the biggest update over

00:00:04.296 --> 00:00:05.197
the last three years?

00:00:05.280 --> 00:00:09.230
What has been the biggest difference
between what it felt like then versus now?

00:00:10.880 --> 00:00:17.880
Broadly speaking, the exponential of the
underlying technology has gone about as

00:00:18.800 --> 00:00:23.437
I expected it to go. There's plus or minus

00:00:23.520 --> 00:00:27.357
a year or two here and there.
I don't know that I would've

00:00:27.440 --> 00:00:34.237
predicted the specific direction of code.
But when I look at the exponential,

00:00:34.320 --> 00:00:39.517
it is roughly what I expected in terms
of the march of the models from smart high

00:00:39.600 --> 00:00:44.201
school student to smart college student
to beginning to do PhD and professional

00:00:44.284 --> 00:00:47.037
stuff, and in the case
of code reaching beyond that.

00:00:47.120 --> 00:00:49.517
The frontier is a little
bit uneven, but it's

00:00:49.600 --> 00:00:55.117
roughly what I expected.
What has been the most surprising

00:00:55.200 --> 00:01:00.694
thing is the lack of public recognition
of how close we are to the end of

00:01:00.778 --> 00:01:01.917
the exponential.

00:01:02.000 --> 00:01:08.957
To me, it is absolutely wild that you have
people — within the bubble and outside the

00:01:09.040 --> 00:01:16.040
bubble — talking about the same tired, old
hot-button political issues, when we are

00:01:19.920 --> 00:01:23.917
near the end of the exponential.
I want to understand what that

00:01:24.000 --> 00:01:26.957
exponential looks like right now.
The first question I asked you when

00:01:27.040 --> 00:01:30.990
we recorded three years ago was, "what’s
up with scaling and why does it work?"

00:01:31.440 --> 00:01:34.640
I have a similar question now,
but it feels more complicated.

00:01:35.280 --> 00:01:40.830
At least from the public's point of view,
three years ago there were well-known

00:01:40.913 --> 00:01:41.757
public trends

00:01:41.840 --> 00:01:45.037
across many orders of magnitude of compute
where you could see how the loss improves.

00:01:45.120 --> 00:01:49.597
Now we have RL scaling and there's
no publicly known scaling law for it.

00:01:49.680 --> 00:01:53.643
It's not even clear what the story is.
Is this supposed to be teaching the model

00:01:53.726 --> 00:01:56.449
skills? Is it supposed
to be teaching meta-learning?

00:01:56.532 --> 00:01:59.040
What is the scaling
hypothesis at this point?

00:01:59.840 --> 00:02:06.397
I actually have the same hypothesis
I had even all the way back in 2017.

00:02:06.480 --> 00:02:10.437
I think I talked about it last time,
but I wrote a doc called "The Big Blob

00:02:10.521 --> 00:02:11.760
of Compute Hypothesis".

00:02:12.880 --> 00:02:15.517
It wasn't about the scaling
of language models in particular.

00:02:15.600 --> 00:02:21.680
When I wrote it GPT-1 had just come out.
That was one among many things.

00:02:22.720 --> 00:02:26.077
Back in those days there was robotics.
People tried to work on reasoning as

00:02:26.160 --> 00:02:30.157
a separate thing from language models,
and there was scaling of the kind of RL

00:02:30.240 --> 00:02:35.992
that happened in AlphaGo
and in Dota at OpenAI.

00:02:36.076 --> 00:02:42.160
People remember StarCraft
at DeepMind, AlphaStar.

00:02:43.520 --> 00:02:50.520
It was written as a more general document.
Rich Sutton put out "The Bitter

00:02:52.160 --> 00:02:57.437
Lesson" a couple years later.
The hypothesis is basically the same.

00:02:57.520 --> 00:03:02.035
What it says is that all
the cleverness, all the techniques,

00:03:02.118 --> 00:03:04.717
all the "we need a new method to do

00:03:04.800 --> 00:03:08.477
something", that doesn't matter very much.
There are only a few things that matter.

00:03:08.560 --> 00:03:13.597
I think I listed seven of them.
One is how much raw compute you have.

00:03:13.680 --> 00:03:19.612
The second is the quantity of data.
The third is the quality and distribution

00:03:19.695 --> 00:03:20.237
of data.

00:03:20.320 --> 00:03:26.560
It needs to be a broad distribution.
The fourth is how long you train for.

00:03:27.200 --> 00:03:31.997
The fifth is that you need an objective
function that can scale to the moon.

00:03:32.080 --> 00:03:35.840
The pre-training objective function
is one such objective function.

00:03:36.720 --> 00:03:42.557
Another is the RL objective
function that says you have a goal,

00:03:42.640 --> 00:03:46.128
you're going to go out
and reach the goal. Within that,

00:03:46.212 --> 00:03:48.077
there's objective rewards like

00:03:48.160 --> 00:03:52.077
you see in math and coding, and there's
more subjective rewards like you see in

00:03:52.160 --> 00:03:59.117
RLHF or higher-order versions of that.
Then the sixth and seventh were things

00:03:59.200 --> 00:04:04.637
around normalization or conditioning,
just getting the numerical stability

00:04:04.720 --> 00:04:10.181
so that the big blob of compute flows in
this laminar way instead of running into

00:04:10.264 --> 00:04:15.597
problems. That was the hypothesis,
and it's a hypothesis I still hold.

00:04:15.680 --> 00:04:19.600
I don't think I've seen very
much that is not in line with it.

00:04:21.040 --> 00:04:26.740
The pre-training scaling laws were
one example of what we see there.

00:04:26.824 --> 00:04:29.120
Those have continued going.

00:04:31.120 --> 00:04:35.837
Now it's been widely reported,
we feel good about pre-training.

00:04:35.920 --> 00:04:41.837
It’s continuing to give us gains.
What has changed is that now we're

00:04:41.920 --> 00:04:46.077
also seeing the same thing for RL.
We're seeing a pre-training phase

00:04:46.160 --> 00:04:53.160
and then an RL phase on top of that.
With RL, it’s actually just the same.

00:04:55.280 --> 00:05:02.280
Even other companies have published
things in some of their releases that say,

00:05:03.851 --> 00:05:10.031
"We train the model on math
contests — AIME or other things —

00:05:10.114 --> 00:05:14.557
and how well the model
does is log-linear in

00:05:14.640 --> 00:05:17.437
how long we've trained it."
We see that as well,

00:05:17.520 --> 00:05:21.757
and it's not just math contests.
It's a wide variety of RL tasks.

00:05:21.840 --> 00:05:27.197
We're seeing the same scaling
in RL that we saw for pre-training.

00:05:27.280 --> 00:05:31.757
You mentioned Rich Sutton and "The Bitter
Lesson". I interviewed him last year,

00:05:31.840 --> 00:05:38.477
and he's actually very non-LLM-pilled.
I don’t know if this is his perspective,

00:05:38.560 --> 00:05:41.425
but one way to paraphrase
his objection is:

00:05:41.509 --> 00:05:44.717
Something which possesses
the true core of human

00:05:44.800 --> 00:05:49.816
learning would not require all these
billions of dollars of data and compute

00:05:49.899 --> 00:05:55.617
and these bespoke environments, to learn
how to use Excel, how to use PowerPoint,

00:05:55.701 --> 00:05:57.837
how to navigate a web browser.

00:05:57.920 --> 00:06:03.533
The fact that we have to build in these
skills using these RL environments hints

00:06:03.617 --> 00:06:08.637
that we are actually lacking
a core human learning algorithm.

00:06:08.720 --> 00:06:12.957
So we're scaling the wrong thing.
That does raise

00:06:13.040 --> 00:06:15.895
the question. Why are we doing all this RL
scaling if we think there's something

00:06:15.979 --> 00:06:20.037
that's going to be human-like
in its ability to learn on the fly?

00:06:20.120 --> 00:06:23.517
I think this puts together
several things that

00:06:23.600 --> 00:06:29.037
should be thought of differently.
There is a genuine puzzle here,

00:06:29.120 --> 00:06:33.040
but it may not matter.
In fact, I would guess it probably

00:06:33.760 --> 00:06:35.404
doesn't matter.
There is an interesting thing.

00:06:35.487 --> 00:06:37.357
Let me take the RL out
of it for a second, because I

00:06:37.440 --> 00:06:42.032
actually think it's a red herring to say
that RL is any different from pre-training

00:06:42.115 --> 00:06:42.960
in this matter.

00:06:43.760 --> 00:06:47.677
If we look at pre-training scaling,
it was very interesting

00:06:47.760 --> 00:06:53.297
back in 2017 when Alec
Radford was doing GPT-1.

00:06:53.380 --> 00:06:59.037
The models before GPT-1
were trained on datasets

00:06:59.120 --> 00:07:06.120
that didn't represent a wide distribution
of text. You had very standard language

00:07:08.080 --> 00:07:11.277
modeling benchmarks.
GPT-1 itself was trained on

00:07:11.360 --> 00:07:17.757
a bunch of fanfiction, I think actually.
It was literary text, which is a very

00:07:17.840 --> 00:07:23.197
small fraction of the text you can get.
In those days it was like a billion words

00:07:23.280 --> 00:07:30.280
or something, so small datasets
representing a pretty narrow distribution

00:07:31.244 --> 00:07:37.997
of what you can see in the world.
It didn't generalize well.

00:07:38.080 --> 00:07:43.280
If you did better
on some fanfiction corpus,

00:07:43.920 --> 00:07:47.277
it wouldn't generalize that well to other
tasks. We had all these measures. We had

00:07:47.360 --> 00:07:53.840
all these measures of how well it did at
predicting all these other kinds of texts.

00:07:55.840 --> 00:07:59.602
It was only when you trained over
all the tasks on the internet —

00:07:59.686 --> 00:08:01.437
when you did a general internet

00:08:01.520 --> 00:08:05.458
scrape from something like Common
Crawl or scraping links in Reddit,

00:08:05.541 --> 00:08:11.566
which is what we did for GPT-2 —
that you started to get generalization.

00:08:11.649 --> 00:08:15.757
I think we're seeing the same thing on RL.

00:08:15.840 --> 00:08:21.917
We're starting first with simple RL
tasks like training on math competitions,

00:08:22.000 --> 00:08:23.120
then moving to

00:08:24.640 --> 00:08:28.577
broader training that
involves things like code.

00:08:28.661 --> 00:08:31.677
Now we're moving to many other tasks.

00:08:31.760 --> 00:08:35.277
I think then we're going
to increasingly get generalization.

00:08:35.360 --> 00:08:39.837
So that kind of takes out the RL vs.
pre-training side of it.

00:08:39.920 --> 00:08:46.920
But there is a puzzle either way, which
is that in pre-training we use trillions

00:08:48.524 --> 00:08:49.600
of tokens.

00:08:50.640 --> 00:08:54.477
Humans don't see trillions of words.
So there is an actual sample

00:08:54.560 --> 00:08:59.837
efficiency difference here. There
is actually something different here.

00:08:59.920 --> 00:09:06.317
The models start from scratch
and they need much more training.

00:09:06.400 --> 00:09:13.400
But we also see that once they're trained,
if we give them a long context length of

00:09:15.600 --> 00:09:16.792
a million — the only thing
blocking long context is inference —

00:09:16.875 --> 00:09:17.197
they're very good at

00:09:17.280 --> 00:09:23.360
learning and adapting within that context.
So I don’t know the full answer to this.

00:09:24.080 --> 00:09:28.237
I think there's something going
on where pre-training is not like

00:09:28.320 --> 00:09:32.397
the process of humans learning, but it's
somewhere between the process of humans

00:09:32.480 --> 00:09:35.751
learning and the process
of human evolution.

00:09:35.834 --> 00:09:38.877
We get many of our priors from evolution.

00:09:38.960 --> 00:09:43.117
Our brain isn't just a blank slate.
Whole books have been written about this.

00:09:43.200 --> 00:09:45.357
The language models are
much more like blank slates.

00:09:45.440 --> 00:09:49.616
They literally start as random weights,
whereas the human brain starts with all

00:09:49.699 --> 00:09:53.321
these regions connected to all
these inputs and outputs.

00:09:53.404 --> 00:09:56.397
Maybe we should think
of pre-training — and

00:09:56.480 --> 00:10:02.477
for that matter, RL as well — as something
that exists in the middle space between

00:10:02.560 --> 00:10:06.641
human evolution and human
on-the-spot learning.

00:10:06.724 --> 00:10:10.717
And we should think
of the in-context learning

00:10:10.800 --> 00:10:16.297
that the models do as something between
long-term human learning and short-term

00:10:16.380 --> 00:10:19.928
human learning. So there's
this hierarchy. There’s evolution,

00:10:20.011 --> 00:10:22.557
there's long-term learning,
there's short-term

00:10:22.640 --> 00:10:28.637
learning, and there's just human reaction.
The LLM phases exist along this spectrum,

00:10:28.720 --> 00:10:31.860
but not necessarily
at exactly the same points.

00:10:31.943 --> 00:10:34.877
There’s no analog
to some of the human modes

00:10:34.960 --> 00:10:39.917
of learning the LLMs are falling in
between the points. Does that make sense?

00:10:40.000 --> 00:10:42.317
Yes, although some things
are still a bit confusing.

00:10:42.400 --> 00:10:45.757
For example, if the analogy is that this
is like evolution so it's fine that it's

00:10:45.840 --> 00:10:51.437
not sample efficient, then if we're
going to get super sample-efficient

00:10:51.520 --> 00:10:56.064
agent from in-context learning,
why are we bothering to build all these RL

00:10:56.147 --> 00:10:56.877
environments?

00:10:56.960 --> 00:11:00.317
There are companies whose work seems
to be teaching models how to use this API,

00:11:00.400 --> 00:11:04.094
how to use Slack, how to use whatever.
It's confusing to me why there's so much

00:11:04.177 --> 00:11:04.477
emphasis

00:11:04.560 --> 00:11:09.522
on that if the kind of agent that can
just learn on the fly is emerging or has

00:11:09.605 --> 00:11:10.640
already emerged.

00:11:11.600 --> 00:11:14.338
I can't speak for the
emphasis of anyone else.

00:11:14.421 --> 00:11:17.120
I can only talk about
how we think about it.

00:11:20.720 --> 00:11:25.037
The goal is not to teach the model
every possible skill within RL,

00:11:25.120 --> 00:11:28.251
just as we don't do that within
pre-training. Within pre-training,

00:11:28.334 --> 00:11:29.517
we're not trying to expose

00:11:29.600 --> 00:11:35.600
the model to every possible way
that words could be put together.

00:11:37.280 --> 00:11:41.997
Rather, the model trains on a lot
of things and then reaches generalization

00:11:42.080 --> 00:11:43.360
across pre-training.

00:11:43.920 --> 00:11:50.183
That was the transition from GPT-1
to GPT-2 that I saw up close.

00:11:50.267 --> 00:11:53.357
The model reaches a point. I had

00:11:53.440 --> 00:12:00.440
these moments where I was like, "Oh yeah,
you just give the model a list of numbers

00:12:00.657 --> 00:12:04.203
— this is the cost of the house,
this is the square feet of the house —

00:12:04.287 --> 00:12:05.837
and the model completes the pattern

00:12:05.920 --> 00:12:08.397
and does linear regression."
Not great, but it does it,

00:12:08.480 --> 00:12:12.200
and it's never seen
that exact thing before.

00:12:12.283 --> 00:12:15.917
So to the extent that
we are building these

00:12:16.000 --> 00:12:23.000
RL environments, the goal is very similar
to what was done five or ten years ago

00:12:23.118 --> 00:12:24.720
with pre-training.

00:12:26.560 --> 00:12:28.988
We're trying to get a whole bunch of data,

00:12:29.071 --> 00:12:32.157
not because we want to cover
a specific document or a

00:12:32.240 --> 00:12:35.917
specific skill, but because
we want to generalize.

00:12:36.000 --> 00:12:39.677
I think the framework
you're laying down obviously

00:12:39.760 --> 00:12:42.831
makes sense.
We're making progress toward AGI.

00:12:42.914 --> 00:12:45.917
Nobody at this point
disagrees we're going to

00:12:46.000 --> 00:12:49.437
achieve AGI this century.
The crux is you say we're

00:12:49.520 --> 00:12:54.320
hitting the end of the exponential.
Somebody else looks at this and says,

00:12:55.920 --> 00:12:59.642
"We've been making progress since 2012,
and by 2035 we'll have a human-like

00:12:59.725 --> 00:13:00.558
agent."

00:13:04.800 --> 00:13:07.277
Obviously we’re seeing in these models
the kinds of things that evolution did,

00:13:07.360 --> 00:13:09.528
or that learning within
a human lifetime does.

00:13:09.611 --> 00:13:11.437
I want to understand what you’re seeing

00:13:11.520 --> 00:13:14.670
that makes you think it's one
year away and not ten years away.

00:13:17.680 --> 00:13:24.680
There are two claims you could make
here, one stronger and one weaker.

00:13:26.560 --> 00:13:33.560
Starting with the weaker claim,
when I first saw the scaling back in 2019,

00:13:35.120 --> 00:13:41.133
I wasn’t sure. This was a 50/50 thing.
I thought I saw something.

00:13:41.217 --> 00:13:43.197
My claim was that this

00:13:43.280 --> 00:13:50.280
was much more likely than anyone thinks.
Maybe there's a 50% chance this happens.

00:13:51.600 --> 00:13:55.362
On the basic hypothesis of, as you put it,

00:13:55.445 --> 00:14:00.397
within ten years we'll get
to what I call a "country of

00:14:00.480 --> 00:14:06.877
geniuses in a data center", I'm at 90% on
that. It's hard to go much higher than 90%

00:14:06.960 --> 00:14:12.835
because the world is so unpredictable.
Maybe the irreducible uncertainty puts us

00:14:12.919 --> 00:14:13.357
at 95%,

00:14:13.440 --> 00:14:20.440
where you get to things like multiple
companies having internal turmoil,

00:14:20.953 --> 00:14:23.040
Taiwan gets invaded,

00:14:24.160 --> 00:14:29.997
all the fabs get blown up by missiles.
Now you've jinxed us, Dario.

00:14:30.080 --> 00:14:37.080
You could construct a 5% world where
things get delayed for ten years.

00:14:43.920 --> 00:14:49.200
There's another 5% which is that I'm very
confident on tasks that can be verified.

00:14:50.160 --> 00:14:53.200
With coding, except
for that irreducible uncertainty,

00:14:54.400 --> 00:14:56.093
I think we'll be there
in one or two years.

00:14:56.177 --> 00:14:58.077
There's no way we will
not be there in ten years

00:14:58.160 --> 00:15:02.041
in terms of being able
to do end-to-end coding.

00:15:02.124 --> 00:15:05.837
My one little bit
of fundamental uncertainty,

00:15:05.920 --> 00:15:10.205
even on long timescales,
is about tasks that aren't verifiable:

00:15:10.288 --> 00:15:12.160
planning a mission to Mars;

00:15:14.320 --> 00:15:21.320
doing some fundamental scientific
discovery like CRISPR; writing a novel.

00:15:21.440 --> 00:15:27.517
It’s hard to verify those tasks.
I am almost certain we have a

00:15:27.600 --> 00:15:33.120
reliable path to get there, but if there's
a little bit of uncertainty it's there.

00:15:34.320 --> 00:15:39.600
On the ten-year timeline I'm at 90%,
which is about as certain as you can be.

00:15:40.400 --> 00:15:46.157
I think it's crazy to say
that this won't happen by 2035.

00:15:46.240 --> 00:15:48.717
In some sane world, it would
be outside the mainstream.

00:15:48.800 --> 00:15:55.800
But the emphasis on verification hints to
me a lack of belief that these models are

00:15:57.467 --> 00:15:58.637
generalized.

00:15:58.720 --> 00:16:03.251
If you think about humans, we're both
good at things for which we get verifiable

00:16:03.334 --> 00:16:07.837
reward and things for which we don't.
No, this is why I’m almost sure.

00:16:07.920 --> 00:16:12.557
We already see substantial generalization
from things that verify to things that

00:16:12.640 --> 00:16:15.357
don't. We're already seeing that.
But it seems like you were emphasizing

00:16:15.440 --> 00:16:20.221
this as a spectrum which will split
apart which domains in which we see more

00:16:20.304 --> 00:16:21.137
progress.

00:16:21.600 --> 00:16:24.359
That doesn't seem like
how humans get better.

00:16:24.442 --> 00:16:27.517
The world in which we
don't get there is the world

00:16:27.600 --> 00:16:34.600
in which we do all the verifiable
things. Many of them generalize,

00:16:34.800 --> 00:16:39.997
but we don't fully get there.
We don’t fully color in the other side

00:16:40.080 --> 00:16:47.080
of the box. It's not a binary thing. Even
if generalization is weak and you can only

00:16:47.347 --> 00:16:49.237
do verifiable domains, it's not clear to
me you could automate software engineering

00:16:49.320 --> 00:16:53.548
in such a world. You are "a
software engineer" in some sense,

00:16:53.631 --> 00:16:58.397
but part of being a software
engineer for you involves

00:16:58.480 --> 00:17:00.888
writing long memos
about your grand vision.

00:17:00.971 --> 00:17:03.437
I don’t think that’s
part of the job of SWE.

00:17:03.520 --> 00:17:04.877
That's part of the job of the
company, not SWE specifically.

00:17:04.960 --> 00:17:09.280
But SWE does involve design
documents and other things like that.

00:17:10.400 --> 00:17:13.150
The models are already pretty
good at writing comments.

00:17:14.800 --> 00:17:20.780
Again, I’m making much weaker
claims here than I believe,

00:17:20.863 --> 00:17:24.397
to distinguish between two things.

00:17:24.480 --> 00:17:27.080
We're already almost there
for software engineering.

00:17:28.000 --> 00:17:31.334
By what metric? There's one metric which
is how many lines of code are written

00:17:31.417 --> 00:17:32.250
by AI.

00:17:32.800 --> 00:17:36.227
If you consider other productivity
improvements in the history of software

00:17:36.311 --> 00:17:38.813
engineering, compilers write
all the lines of software.

00:17:38.896 --> 00:17:40.877
There's a difference
between how many lines

00:17:40.960 --> 00:17:44.566
are written and how big
the productivity improvement is.

00:17:44.649 --> 00:17:46.560
"We’re almost there" meaning…

00:17:47.120 --> 00:17:51.270
How big is the productivity improvement,
not just how many lines are written by AI?

00:17:52.000 --> 00:17:57.037
I actually agree with you on this.
I've made a series of predictions on

00:17:57.120 --> 00:18:02.000
code and software engineering. I think
people have repeatedly misunderstood them.

00:18:03.920 --> 00:18:09.437
Let me lay out the spectrum.
About eight or nine months ago,

00:18:09.520 --> 00:18:16.520
I said the AI model will be writing 90% of
the lines of code in three to six months.

00:18:16.800 --> 00:18:21.597
That happened, at least at some places.
It happened at Anthropic, happened with

00:18:21.680 --> 00:18:27.437
many people downstream using our models.
But that's actually a very weak criterion.

00:18:27.520 --> 00:18:31.564
People thought I was saying that we
won't need 90% of the software engineers.

00:18:31.647 --> 00:18:32.797
Those things are worlds

00:18:32.880 --> 00:18:37.943
apart. The spectrum is:
90% of code is written by the model,

00:18:38.026 --> 00:18:41.117
100% of code is written by the model.

00:18:41.200 --> 00:18:46.823
That's a big difference in productivity.
90% of the end-to-end SWE tasks —

00:18:46.906 --> 00:18:47.517
including

00:18:47.600 --> 00:18:54.077
things like compiling, setting up clusters
and environments, testing features,

00:18:54.160 --> 00:19:01.160
writing memos — are done by the models.
100% of today's SWE tasks are done by

00:19:01.440 --> 00:19:02.397
the models.

00:19:02.480 --> 00:19:05.997
Even when that happens, it doesn't mean
software engineers are out of a job.

00:19:06.080 --> 00:19:10.157
There are new higher-level things
they can do, where they can manage.

00:19:10.240 --> 00:19:15.677
Then further down the spectrum, there's
90% less demand for SWEs, which I think

00:19:15.760 --> 00:19:21.357
will happen but this is a spectrum.
I wrote about it in "The Adolescence

00:19:21.440 --> 00:19:25.040
of Technology" where I went through
this kind of spectrum with farming.

00:19:26.720 --> 00:19:29.120
I actually totally agree with you on that.

00:19:29.920 --> 00:19:32.077
These are very different
benchmarks from each other,

00:19:32.160 --> 00:19:34.786
but we're proceeding
through them super fast.

00:19:34.870 --> 00:19:37.760
Part of your vision
is that going from 90 to 100

00:19:38.670 --> 00:19:45.357
is going to happen fast, and that it
leads to huge productivity improvements.

00:19:45.440 --> 00:19:48.214
But what I notice is that even
in greenfield projects people start

00:19:48.297 --> 00:19:49.037
with Claude Code or

00:19:49.120 --> 00:19:54.557
something, people report starting a lot of
projects… Do we see in the world out there

00:19:54.640 --> 00:19:58.157
a renaissance of software, all these new
features that wouldn't exist otherwise?

00:19:58.240 --> 00:20:01.997
At least so far, it doesn't seem like
we see that. So that does make me wonder.

00:20:02.080 --> 00:20:09.080
Even if I never had to intervene with
Claude Code, the world is complicated.

00:20:09.200 --> 00:20:14.557
Jobs are complicated. Closing the loop on
self-contained systems, whether it’s just

00:20:14.640 --> 00:20:19.760
writing software or something, how much
broader gains would we see just from that?

00:20:20.400 --> 00:20:24.557
Maybe that should dilute our estimation
of the "country of geniuses".

00:20:24.640 --> 00:20:31.640
I simultaneously agree with you that it's
a reason why these things don't happen

00:20:34.311 --> 00:20:40.000
instantly, but at the same time,
I think the effect is gonna be very fast.

00:20:41.360 --> 00:20:46.957
You could have these two poles.
One is that AI is not going to make

00:20:47.040 --> 00:20:52.397
progress. It's slow. It's going to take
forever to diffuse within the economy.

00:20:52.480 --> 00:20:56.477
Economic diffusion has become one
of these buzzwords that's a reason why

00:20:56.560 --> 00:21:00.110
we're not going to make AI progress,
or why AI progress doesn't matter.

00:21:00.560 --> 00:21:04.560
The other axis is that we'll get recursive
self-improvement, the whole thing.

00:21:05.120 --> 00:21:07.770
Can't you just draw
an exponential line on the curve?

00:21:08.400 --> 00:21:15.400
We're going to have Dyson spheres around
the sun so many nanoseconds after we get

00:21:16.803 --> 00:21:17.757
recursive.

00:21:17.840 --> 00:21:23.277
I'm completely caricaturing the view here,
but there are these two extremes.

00:21:23.360 --> 00:21:29.198
But what we've seen from the beginning,
at least if you look within Anthropic,

00:21:29.282 --> 00:21:35.627
there's this bizarre 10x per year growth
in revenue that we've seen. So in 2023,

00:21:35.710 --> 00:21:42.424
it was zero to $100 million. In 2024, it
was $100 million to $1 billion. In 2025,

00:21:42.507 --> 00:21:45.600
it was $1 billion to $ 9-10 billion.

00:21:46.320 --> 00:21:48.502
You guys should have just bought a billion
dollars of your own products so you could

00:21:48.585 --> 00:21:49.419
just…

00:21:50.640 --> 00:21:53.997
And the first month of this year,
that exponential is...

00:21:54.080 --> 00:22:01.080
You would think it would slow down, but
we added another few billion to revenue

00:22:02.858 --> 00:22:04.080
in January.

00:22:05.600 --> 00:22:10.157
Obviously that curve can't go on forever.
The GDP is only so large.

00:22:10.240 --> 00:22:15.703
I would even guess that it
bends somewhat this year,

00:22:15.787 --> 00:22:20.077
but that is a fast curve. That's a really

00:22:20.160 --> 00:22:25.173
fast curve. I would bet it stays pretty
fast even as the scale goes to the entire

00:22:25.257 --> 00:22:32.003
economy. So I think we should be thinking
about this middle world where things are

00:22:32.087 --> 00:22:33.997
extremely fast, but not

00:22:34.080 --> 00:22:37.523
instant, where they take time
because of economic diffusion,

00:22:37.607 --> 00:22:39.757
because of the need to close the loop.

00:22:39.840 --> 00:22:46.840
Because it's fiddly: "I have to do
change management within my enterprise…

00:22:48.518 --> 00:22:50.077
I set this up,

00:22:50.160 --> 00:22:54.037
but I have to change the security
permissions on this in order to make it

00:22:54.120 --> 00:22:54.954
actually work…

00:22:55.600 --> 00:23:01.437
I had this old piece of software that
checks the model before it's compiled

00:23:01.520 --> 00:23:05.357
and released and I have to rewrite it.
Yes, the model can do that, but I have

00:23:05.440 --> 00:23:09.200
to tell the model to do that.
It has to take time to do that."

00:23:10.080 --> 00:23:17.080
So I think everything we've seen so far is
compatible with the idea that there's one

00:23:17.258 --> 00:23:20.202
fast exponential that's
the capability of the model.

00:23:20.286 --> 00:23:22.317
Then there's another fast exponential

00:23:22.400 --> 00:23:26.157
that's downstream of that, which is the
diffusion of the model into the economy.

00:23:26.240 --> 00:23:32.051
Not instant, not slow, much faster
than any previous technology,

00:23:32.134 --> 00:23:34.160
but it has its limits.

00:23:37.120 --> 00:23:41.809
When I look inside Anthropic, when I
look at our customers: fast adoption,

00:23:41.892 --> 00:23:45.437
but not infinitely fast.
Can I try a hot take on you? Yeah.

00:23:45.520 --> 00:23:48.573
I feel like diffusion
is cope that people say.

00:23:48.656 --> 00:23:51.437
When the model isn't able to do something,

00:23:51.520 --> 00:23:54.100
they're like, "oh,
but it's a diffusion issue."

00:23:54.183 --> 00:23:56.877
But then you should use
the comparison to humans.

00:23:56.960 --> 00:24:00.477
You would think that the inherent
advantages that AIs have would make

00:24:00.560 --> 00:24:01.677
diffusion a much easier

00:24:01.760 --> 00:24:05.997
problem for new AIs getting onboarded
than new humans getting onboarded.

00:24:06.080 --> 00:24:08.637
An AI can read your entire
Slack and your drive in minutes.

00:24:08.720 --> 00:24:12.077
They can share all the knowledge that the
other copies of the same instance have.

00:24:12.160 --> 00:24:14.397
You don't have this adverse selection
problem when you're hiring AI, so you

00:24:14.480 --> 00:24:17.203
can just hire copies of a vetted AI model.

00:24:17.287 --> 00:24:20.077
Hiring a human is so
much more of a hassle.

00:24:20.160 --> 00:24:23.357
People hire humans all the time.
We pay humans upwards of $50 trillion

00:24:23.440 --> 00:24:28.513
in wages because they're useful,
even though in principle it would be much

00:24:28.596 --> 00:24:32.612
easier to integrate AIs into
the economy than it is to hire humans.

00:24:32.695 --> 00:24:34.797
The diffusion doesn't really explain.

00:24:34.880 --> 00:24:41.757
I think diffusion is very real
and doesn't exclusively have

00:24:41.840 --> 00:24:48.840
to do with limitations on the AI models.
Again, there are people who use diffusion

00:24:49.760 --> 00:24:54.157
as kind of a buzzword to say this isn't a
big deal. I'm not talking about that. I'm

00:24:54.240 --> 00:24:57.997
not talking about how AI will diffuse
at the speed of previous technologies.

00:24:58.080 --> 00:25:02.491
I think AI will diffuse much faster
than previous technologies have,

00:25:02.574 --> 00:25:07.550
but not infinitely fast. I'll just give
an example of this. There's Claude Code.

00:25:07.634 --> 00:25:10.160
Claude Code is extremely easy to set up.

00:25:10.960 --> 00:25:14.077
If you're a developer, you can
just start using Claude Code.

00:25:14.160 --> 00:25:19.437
There is no reason why a developer at a
large enterprise should not be adopting

00:25:19.520 --> 00:25:25.197
Claude Code as quickly as an individual
developer or developer at a startup.

00:25:25.280 --> 00:25:31.037
We do everything we can to promote it.
We sell Claude Code to enterprises.

00:25:31.120 --> 00:25:36.478
Big enterprises, big financial companies,
big pharmaceutical companies,

00:25:36.561 --> 00:25:38.317
all of them are adopting

00:25:38.400 --> 00:25:45.400
Claude Code much faster than enterprises
typically adopt new technology. But again,

00:25:46.320 --> 00:25:51.421
it takes time. Any given
feature or any given product,

00:25:51.504 --> 00:25:54.877
like Claude Code or Cowork, will get

00:25:54.960 --> 00:26:00.632
adopted by the individual developers
who are on Twitter all the time,

00:26:00.715 --> 00:26:02.717
by the Series A startups,

00:26:02.800 --> 00:26:09.800
many months faster than they will get
adopted by a large enterprise that does

00:26:10.080 --> 00:26:11.120
food sales.

00:26:11.840 --> 00:26:14.557
There are just a number of factors.
You have to go through legal,

00:26:14.640 --> 00:26:20.877
you have to provision it for everyone.
It has to pass security and compliance.

00:26:20.960 --> 00:26:25.613
The leaders of the company who are
further away from the AI revolution are

00:26:25.696 --> 00:26:26.637
forward-looking,

00:26:26.720 --> 00:26:31.037
but they have to say, "Oh, it makes
sense for us to spend 50 million.

00:26:31.120 --> 00:26:35.037
This is what this Claude Code thing is.
This is why it helps our company.

00:26:35.120 --> 00:26:37.597
This is why it makes us more productive."
Then they have to explain

00:26:37.680 --> 00:26:42.717
to the people two levels below. They have
to say, "Okay, we have 3,000 developers.

00:26:42.800 --> 00:26:45.117
Here's how we're going to roll
it out to our developers."

00:26:45.200 --> 00:26:50.397
We have conversations like this every day.
We are doing everything we can to make

00:26:50.480 --> 00:26:55.760
Anthropic's revenue grow 20 or 30x
a year instead of 10x a year.

00:26:57.440 --> 00:27:01.440
Again, many enterprises are just
saying, "This is so productive.

00:27:02.160 --> 00:27:05.037
We're going to take shortcuts
in our usual procurement process."

00:27:05.120 --> 00:27:08.477
They're moving much faster than
when we tried to sell them just

00:27:08.560 --> 00:27:13.677
the ordinary API, which many of them use.
Claude Code is a more compelling product,

00:27:13.760 --> 00:27:16.544
but it's not an infinitely
compelling product.

00:27:16.627 --> 00:27:19.037
I don't think even AGI or powerful AI or

00:27:19.120 --> 00:27:22.877
"country of geniuses in a data center"
will be an infinitely compelling product.

00:27:22.960 --> 00:27:26.777
It will be a compelling product
enough maybe to get 3-5x, or 10x,

00:27:26.860 --> 00:27:30.158
a year of growth, even when you're
in the hundreds of billions of dollars,

00:27:30.242 --> 00:27:31.997
which is extremely hard
to do and has never been

00:27:32.080 --> 00:27:34.456
done in history before,
but not infinitely fast.

00:27:34.539 --> 00:27:36.557
I buy that it would be a slight slowdown.

00:27:36.640 --> 00:27:39.357
Maybe this is not your claim,
but sometimes people talk about this like,

00:27:39.440 --> 00:27:43.033
"Oh, the capabilities are there,
but because of diffusion...

00:27:43.117 --> 00:27:45.200
otherwise we're basically at AGI".

00:27:46.160 --> 00:27:49.277
I don't believe we're basically at AGI.
I think if you had the "country

00:27:49.360 --> 00:27:53.757
of geniuses in a data center"...
If we had the "country of geniuses

00:27:53.840 --> 00:27:57.277
in a data center", we would know it.
We would know it if you had the

00:27:57.360 --> 00:28:01.517
"country of geniuses in a data center".
Everyone in this room would know it.

00:28:01.600 --> 00:28:07.677
Everyone in Washington would know it.
People in rural parts might not know it,

00:28:07.760 --> 00:28:14.760
but we would know it. We don't
have that now. That is very clear.

00:29:42.400 --> 00:29:46.432
Coming back to concrete prediction…
Because there are so many different things

00:29:46.516 --> 00:29:47.277
to disambiguate,

00:29:47.360 --> 00:29:50.797
it can be easy to talk past each other
when we're talking about capabilities.

00:29:50.880 --> 00:29:53.092
For example, when I
interviewed you three years ago,

00:29:53.175 --> 00:29:54.637
I asked you a prediction about what

00:29:54.720 --> 00:29:58.370
we should expect three years from now.
You were right. You said,

00:29:58.453 --> 00:30:00.237
"We should expect systems which,

00:30:00.320 --> 00:30:04.077
if you talk to them for the course of an
hour, it's hard to tell them apart from

00:30:04.160 --> 00:30:07.037
a generally well-educated human."
I think you were right about that.

00:30:07.120 --> 00:30:11.869
I think spiritually I feel unsatisfied
because my internal expectation was

00:30:11.953 --> 00:30:15.507
that such a system could automate
large parts of white-collar work.

00:30:15.591 --> 00:30:17.677
So it might be more
productive to talk about

00:30:17.760 --> 00:30:20.560
the actual end capabilities
you want from such a system.

00:30:21.520 --> 00:30:24.988
I will basically tell
you where I think we are.

00:30:25.071 --> 00:30:28.237
Let me ask a very
specific question so that

00:30:28.320 --> 00:30:32.077
we can figure out exactly what kinds of
capabilities we should think about soon.

00:30:32.160 --> 00:30:35.268
Maybe I'll ask about it in the
context of a job I understand well,

00:30:35.351 --> 00:30:36.477
not because it's the most

00:30:36.560 --> 00:30:40.437
relevant job, but just because I
can evaluate the claims about it.

00:30:40.520 --> 00:30:41.997
Take video editors. I have

00:30:42.080 --> 00:30:47.837
video editors. Part of their job involves
learning about our audience's preferences,

00:30:47.920 --> 00:30:50.717
learning about my preferences and tastes,
and the different trade-offs we have.

00:30:50.800 --> 00:30:54.950
They’re, over the course of many months,
building up this understanding of context.

00:30:55.600 --> 00:30:58.877
The skill and ability they have six
months into the job, a model that can

00:30:58.960 --> 00:31:03.760
pick up that skill on the job on the fly,
when should we expect such an AI system?

00:31:04.400 --> 00:31:08.031
I guess what you're talking about is
that we're doing this interview for three

00:31:08.114 --> 00:31:08.948
hours.

00:31:09.920 --> 00:31:11.837
Someone's going to come in,
someone's going to edit it.

00:31:11.920 --> 00:31:14.885
They're going to be like,
"Oh, I don't know,

00:31:14.968 --> 00:31:18.640
Dario scratched his head
and we could edit that out."

00:31:19.200 --> 00:31:22.397
"Magnify that." "There was this long

00:31:22.480 --> 00:31:24.863
discussion that is less
interesting to people.

00:31:24.947 --> 00:31:27.277
There's another thing
that's more interesting

00:31:27.360 --> 00:31:33.517
to people, so let's make this edit."
I think the "country of geniuses in

00:31:33.600 --> 00:31:38.088
a data center" will be able to do that.
The way it will be able to do that is it

00:31:38.171 --> 00:31:43.517
will have general control of a computer
screen. You'll be able to feed this in.

00:31:43.600 --> 00:31:46.896
It'll be able to also use
the computer screen to go on the web,

00:31:46.979 --> 00:31:48.320
look at all your previous

00:31:49.360 --> 00:31:53.917
interviews, look at what people are saying
on Twitter in response to your interviews,

00:31:54.000 --> 00:31:59.037
talk to you, ask you questions, talk to
your staff, look at the history of edits

00:31:59.120 --> 00:32:04.560
that you did, and from that, do the job. I
think that's dependent on several things.

00:32:06.720 --> 00:32:10.157
I think this is one of the things
that's actually blocking deployment:

00:32:10.240 --> 00:32:14.431
getting to the point on computer use where
the models are really masters at using

00:32:14.514 --> 00:32:15.347
the computer.

00:32:16.320 --> 00:32:20.797
We've seen this climb in benchmarks, and
benchmarks are always imperfect measures.

00:32:20.880 --> 00:32:27.880
But I think when we first released
computer use a year and a quarter ago,

00:32:30.474 --> 00:32:33.677
OSWorld was at maybe 15%.

00:32:33.760 --> 00:32:38.400
I don't remember exactly,
but we've climbed from that to 65-70%.

00:32:40.240 --> 00:32:45.296
There may be harder measures as well, but
I think computer use has to pass a point

00:32:45.380 --> 00:32:49.480
of reliability. Can I just follow up on
that before you move on to the next point?

00:32:50.000 --> 00:32:54.077
For years, I've been trying to build
different internal LLM tools for myself.

00:32:54.160 --> 00:32:59.517
Often I have these text-in, text-out
tasks, which should be dead center

00:32:59.600 --> 00:33:03.120
in the repertoire of these models.
Yet I still hire humans to do them.

00:33:03.760 --> 00:33:07.597
If it's something like, "identify what the
best clips would be in this transcript",

00:33:07.680 --> 00:33:10.266
maybe the LLMs do
a seven-out-of-ten job on them.

00:33:10.349 --> 00:33:12.717
But there's not this
ongoing way I can engage

00:33:12.800 --> 00:33:16.317
with them to help them get better at the
job the way I could with a human employee.

00:33:16.400 --> 00:33:20.317
That missing ability, even if you
solve computer use, would still block

00:33:20.400 --> 00:33:24.117
my ability to offload
an actual job to them.

00:33:24.200 --> 00:33:27.917
This gets back to what
we were talking about

00:33:28.000 --> 00:33:31.498
before with learning on the job.
It's very interesting.

00:33:31.581 --> 00:33:33.600
I think with the coding agents,

00:33:34.160 --> 00:33:38.795
I don't think people would say that
learning on the job is what is preventing

00:33:38.878 --> 00:33:39.837
the coding agents

00:33:39.920 --> 00:33:46.077
from doing everything end to end. They
keep getting better. We have engineers

00:33:46.160 --> 00:33:51.437
at Anthropic who don't write any code.
When I look at the productivity, to your

00:33:51.520 --> 00:33:56.162
previous question, we have folks
who say, "This GPU kernel, this chip,

00:33:56.245 --> 00:33:57.917
I used to write it myself.

00:33:58.000 --> 00:34:03.040
I just have Claude do it." There's
this enormous improvement in productivity.

00:34:04.560 --> 00:34:11.560
When I see Claude Code, familiarity with
the codebase or a feeling that the model

00:34:13.120 --> 00:34:17.531
hasn't worked at the company for a year,
that's not high up on the list

00:34:17.614 --> 00:34:18.797
of complaints I see.

00:34:18.880 --> 00:34:22.557
I think what I'm saying is that we're
kind of taking a different path.

00:34:22.640 --> 00:34:23.917
Don't you think with coding
that's because there

00:34:24.000 --> 00:34:28.797
is an external scaffold of memory which
exists instantiated in the codebase?

00:34:28.880 --> 00:34:31.037
I don't know how many
other jobs have that.

00:34:31.120 --> 00:34:33.277
Coding made fast
progress precisely because

00:34:33.360 --> 00:34:37.517
it has this unique advantage
that other economic activity doesn't.

00:34:37.600 --> 00:34:43.386
But when you say that, what you're
implying is that by reading the codebase

00:34:43.470 --> 00:34:47.997
into the context, I have everything
that the human needed to learn on the job.

00:34:48.080 --> 00:34:52.436
So that would be an example
of—whether it's written or not,

00:34:52.519 --> 00:34:54.317
whether it's available or

00:34:54.400 --> 00:34:59.360
not—a case where everything you needed
to know you got from the context window.

00:35:00.640 --> 00:35:02.853
What we think of as learning—"I
started this job,

00:35:02.937 --> 00:35:05.197
it's going to take me six
months to understand the

00:35:05.280 --> 00:35:07.335
code base"—the model just
did it in the context.

00:35:07.418 --> 00:35:09.117
I honestly don't know how to think about

00:35:09.200 --> 00:35:13.920
this because there are people who
qualitatively report what you're saying.

00:35:16.080 --> 00:35:20.473
I'm sure you saw last year, there was
a major study where they had experienced

00:35:20.557 --> 00:35:26.547
developers try to close pull requests in
repositories that they were familiar with.

00:35:26.630 --> 00:35:28.717
Those developers reported an

00:35:28.800 --> 00:35:31.597
uplift. They reported that they felt more
productive with the use of these models.

00:35:31.680 --> 00:35:34.957
But in fact, if you look at their output
and how much was actually merged back in,

00:35:35.040 --> 00:35:37.117
there was a 20% downlift.
They were less productive

00:35:37.200 --> 00:35:40.317
as a result of using these models.
So I'm trying to square the qualitative

00:35:40.400 --> 00:35:44.557
feeling that people feel with these
models versus, 1) in a macro level,

00:35:44.640 --> 00:35:48.557
where is this renaissance of software? And
then 2) when people do these independent

00:35:48.640 --> 00:35:53.277
evaluations, why are we not seeing
the productivity benefits we would expect?

00:35:53.360 --> 00:35:56.235
Within Anthropic,
this is just really unambiguous.

00:35:56.318 --> 00:35:58.957
We're under an incredible
amount of commercial

00:35:59.040 --> 00:36:02.807
pressure and make it even harder for
ourselves because we have all this safety

00:36:02.890 --> 00:36:03.597
stuff we do that

00:36:03.680 --> 00:36:10.680
I think we do more than other companies.
The pressure to survive economically

00:36:11.280 --> 00:36:14.562
while also keeping
our values is just incredible.

00:36:14.646 --> 00:36:18.080
We're trying to keep
this 10x revenue curve going.

00:36:18.800 --> 00:36:23.277
There is zero time for bullshit.
There is zero time for feeling

00:36:23.360 --> 00:36:29.840
like we're productive when we're not.
These tools make us a lot more productive.

00:36:30.960 --> 00:36:33.917
Why do you think we're concerned
about competitors using the tools?

00:36:34.000 --> 00:36:38.677
Because we think we're
ahead of the competitors.

00:36:38.760 --> 00:36:43.437
We wouldn't be going
through all this trouble if

00:36:43.520 --> 00:36:46.773
this were secretly
reducing our productivity.

00:36:46.857 --> 00:36:49.517
We see the end productivity every few

00:36:49.600 --> 00:36:54.237
months in the form of model launches.
There's no kidding yourself about this.

00:36:54.320 --> 00:37:00.557
The models make you more productive. 1)
People feeling like they're productive is

00:37:00.640 --> 00:37:02.717
qualitatively predicted
by studies like this.

00:37:02.800 --> 00:37:04.637
But 2) if I just look at the end output,

00:37:04.720 --> 00:37:07.660
obviously you guys are
making fast progress.

00:37:07.743 --> 00:37:10.477
But the idea was supposed to be that with

00:37:10.560 --> 00:37:14.077
recursive self-improvement, you make
a better AI, the AI helps you build a

00:37:14.160 --> 00:37:18.797
better next AI, et cetera, et cetera.
What I see instead—if I look at you,

00:37:18.880 --> 00:37:22.537
OpenAI, DeepMind—is that people are
just shifting around the podium every few

00:37:22.620 --> 00:37:25.037
months. Maybe you think that stops
because you've won or whatever.

00:37:25.120 --> 00:37:31.517
But why are we not seeing the person with
the best coding model have this lasting

00:37:31.600 --> 00:37:35.631
advantage if in fact there are these
enormous productivity gains from the last

00:37:35.714 --> 00:37:36.548
coding model.

00:37:38.160 --> 00:37:44.324
I think my model of the situation is
that there's an advantage that's gradually

00:37:44.407 --> 00:37:50.957
growing. I would say right now the coding
models give maybe, I don't know,

00:37:51.040 --> 00:37:58.040
a 15-20% total factor speed up. That's
my view. Six months ago, it was maybe 5%.

00:38:01.280 --> 00:38:05.392
So it didn't matter. 5% doesn't register.
It's now just getting to the point where

00:38:05.475 --> 00:38:06.309
it's

00:38:06.400 --> 00:38:11.680
one of several factors that kind of
matters. That's going to keep speeding up.

00:38:12.400 --> 00:38:18.397
I think six months ago, there were several
companies that were at roughly the same

00:38:18.480 --> 00:38:21.764
point because this wasn't
a notable factor,

00:38:21.848 --> 00:38:25.837
but I think it's starting
to speed up more and more.

00:38:25.920 --> 00:38:31.188
I would also say there are multiple
companies that write models that are used

00:38:31.271 --> 00:38:32.717
for code and we're not

00:38:32.800 --> 00:38:39.398
perfectly good at preventing some of these
other companies from using our models

00:38:39.481 --> 00:38:40.400
internally.

00:38:41.120 --> 00:38:48.120
So I think everything we're seeing
is consistent with this kind of snowball

00:38:48.898 --> 00:38:49.731
model.

00:38:52.000 --> 00:38:58.778
Again, my theme in all of this is
all of this is soft takeoff, soft,

00:38:58.862 --> 00:39:03.290
smooth exponentials, although
the exponentials are relatively steep.

00:39:03.373 --> 00:39:05.677
So we're seeing this
snowball gather momentum

00:39:05.760 --> 00:39:12.760
where it's like 10%, 20%, 25%, 40%.
As you go, Amdahl's law, you have

00:39:13.120 --> 00:39:17.357
to get all the things that are preventing
you from closing the loop out of the way.

00:39:17.440 --> 00:39:20.960
But this is one of the biggest
priorities within Anthropic.

00:39:22.560 --> 00:39:27.565
Stepping back, before in the stack
we were talking about when do we get

00:39:27.648 --> 00:39:29.357
this on-the-job learning?

00:39:29.440 --> 00:39:32.797
It seems like the point you were making
on the coding thing is that we actually

00:39:32.880 --> 00:39:35.997
don't need on-the-job learning.
You can have tremendous productivity

00:39:36.080 --> 00:39:39.498
improvements, you can have potentially
trillions of dollars of revenue for AI

00:39:39.581 --> 00:39:40.317
companies, without

00:39:40.400 --> 00:39:45.727
this basic human ability to learn
on the job. Maybe that's not your claim,

00:39:45.811 --> 00:39:47.117
you should clarify.

00:39:47.200 --> 00:39:52.157
But in most domains of economic activity,
people say, "I hired somebody,

00:39:52.240 --> 00:39:53.837
they weren't that useful

00:39:53.920 --> 00:39:57.403
for the first few months, and then
over time they built up the context,

00:39:57.487 --> 00:40:00.077
understanding." It's actually hard
to define what we're talking about here.

00:40:00.160 --> 00:40:04.634
But they got something and then now
they're a powerhorse and they're so

00:40:04.717 --> 00:40:09.144
valuable to us. If AI doesn't develop
this ability to learn on the fly,

00:40:09.227 --> 00:40:11.917
I'm a bit skeptical
that we're going to see

00:40:12.000 --> 00:40:16.512
huge changes to the world without
that ability. I think two things here.

00:40:16.595 --> 00:40:21.917
There's the state of the technology right
now. Again, we have these two stages.

00:40:22.000 --> 00:40:27.399
We have the pre-training and RL stage
where you throw a bunch of data and tasks

00:40:27.482 --> 00:40:31.517
into the models and then they generalize.
So it's like learning, but it's like

00:40:31.600 --> 00:40:38.397
learning from more data and not learning
over one human or one model's lifetime.

00:40:38.480 --> 00:40:42.477
So again, this is situated between
evolution and human learning.

00:40:42.560 --> 00:40:45.597
But once you learn all
those skills, you have them.

00:40:45.680 --> 00:40:50.060
Just like with pre-training,
just how the models know more,

00:40:50.143 --> 00:40:52.557
if I look at a pre-trained model,

00:40:52.640 --> 00:40:55.837
it knows more about the history
of samurai in Japan than I do.

00:40:55.920 --> 00:41:02.920
It knows more about baseball than I do.
It knows more about low-pass filters

00:41:03.520 --> 00:41:08.877
and electronics, all of these things.
Its knowledge is way broader than mine.

00:41:08.960 --> 00:41:15.960
So I think even just that may get us
to the point where the models are better

00:41:16.745 --> 00:41:22.359
at everything. We also have, again, just
with scaling the kind of existing setup,

00:41:22.443 --> 00:41:23.917
the in-context learning.

00:41:24.000 --> 00:41:27.437
I would describe it as kind
of like human on-the-job learning,

00:41:27.520 --> 00:41:30.144
but a little weaker
and a little short term.

00:41:30.228 --> 00:41:33.037
You look at in-context
learning and if you give

00:41:33.120 --> 00:41:35.212
the model a bunch
of examples it does get it.

00:41:35.296 --> 00:41:37.596
There's real learning
that happens in context.

00:41:38.160 --> 00:41:42.160
A million tokens is a lot.
That can be days of human learning.

00:41:42.880 --> 00:41:49.880
If you think about the model reading
a million words, how long would it

00:41:50.400 --> 00:41:57.117
take me to read a million? Days or weeks
at least. So you have these two things.

00:41:57.200 --> 00:42:00.800
I think these two things within
the existing paradigm may just be enough

00:42:00.884 --> 00:42:01.517
to get you the

00:42:01.600 --> 00:42:04.717
"country of geniuses in a data center".
I don't know for sure, but I think

00:42:04.800 --> 00:42:08.819
they're going to get you a large
fraction of it. There may be gaps,

00:42:08.902 --> 00:42:10.717
but I certainly think that just

00:42:10.800 --> 00:42:16.308
as things are, this is enough to generate
trillions of dollars of revenue.

00:42:16.391 --> 00:42:17.517
That's one. Two,

00:42:17.600 --> 00:42:23.011
is this idea of continual learning,
this idea of a single model learning on

00:42:23.094 --> 00:42:23.927
the job.

00:42:24.320 --> 00:42:29.197
I think we're working on that too.
There's a good chance that in the next

00:42:29.280 --> 00:42:36.280
year or two, we also solve that.
Again, I think you get most

00:42:36.800 --> 00:42:43.800
of the way there without it.
The trillions of dollars a year market,

00:42:45.280 --> 00:42:48.447
maybe all of the national security
implications and the safety implications

00:42:48.530 --> 00:42:54.328
that I wrote about in "Adolescence of
Technology" can happen without it. But we,

00:42:54.411 --> 00:42:57.757
and I imagine others, are working on it.

00:42:57.840 --> 00:43:02.957
There's a good chance that we will
get there within the next year or two.

00:43:03.040 --> 00:43:06.540
There are a bunch of ideas.
I won't go into all of them in detail, but

00:43:07.040 --> 00:43:10.797
one is just to make the context longer.
There's nothing preventing

00:43:10.880 --> 00:43:14.317
longer contexts from working.
You just have to train at longer contexts

00:43:14.400 --> 00:43:16.451
and then learn to serve them at inference.

00:43:16.535 --> 00:43:18.637
Both of those are
engineering problems that

00:43:18.720 --> 00:43:22.320
we are working on and I would assume
others are working on them as well.

00:43:22.480 --> 00:43:25.997
This context length increase, it seemed
like there was a period from 2020 to 2023

00:43:26.080 --> 00:43:30.942
where from GPT-3 to GPT-4 Turbo, there
was an increase from 2000 context lengths

00:43:31.025 --> 00:43:31.437
to 128K.

00:43:31.520 --> 00:43:36.957
I feel like for the two-ish years since
then, we've been in the same-ish ballpark.

00:43:37.040 --> 00:43:41.357
When context lengths get much longer
than that, people report qualitative

00:43:41.440 --> 00:43:45.600
degradation in the ability of the
model to consider that full context.

00:43:47.120 --> 00:43:49.707
So I'm curious what you're internally
seeing that makes you think,

00:43:49.790 --> 00:43:50.557
"10 million contexts,

00:43:50.640 --> 00:43:54.397
100 million contexts to get six months
of human learning and building context".

00:43:54.480 --> 00:43:58.397
This isn't a research problem. This is
an engineering and inference problem.

00:43:58.480 --> 00:44:03.280
If you want to serve long context,
you have to store your entire KV cache.

00:44:06.560 --> 00:44:11.517
It's difficult to store all the memory
in the GPUs, to juggle the memory around.

00:44:11.600 --> 00:44:15.197
I don't even know the details. At this
point, this is at a level of detail

00:44:15.280 --> 00:44:20.031
that I'm no longer able to follow,
although I knew it in the GPT-3 era.

00:44:20.114 --> 00:44:24.318
"These are the weights, these are
the activations you have to store…"

00:44:24.401 --> 00:44:26.797
But these days the whole thing is flipped

00:44:26.880 --> 00:44:30.161
because we have MoE
models and all of that.

00:44:30.244 --> 00:44:34.000
Regarding this degradation
you're talking about,

00:44:34.640 --> 00:44:38.051
without getting too specific,
there's two things.

00:44:38.134 --> 00:44:41.117
There's the context
length you train at and

00:44:41.200 --> 00:44:45.357
there's a context length that you serve
at. If you train at a small context length

00:44:45.440 --> 00:44:49.437
and then try to serve at a long context
length, maybe you get these degradations.

00:44:49.520 --> 00:44:52.797
It's better than nothing, you might still
offer it, but you get these degradations.

00:44:52.880 --> 00:44:55.480
Maybe it's harder to train
at a long context length.

00:44:56.880 --> 00:45:01.040
I want to, at the same time,
ask about maybe some rabbit holes.

00:45:01.600 --> 00:45:04.477
Wouldn't you expect that if you had
to train on longer context length,

00:45:04.560 --> 00:45:09.291
that would mean that you're able to get
less samples in for the same amount

00:45:09.374 --> 00:45:10.207
of compute?

00:45:10.880 --> 00:45:14.397
Maybe it's not worth diving deep on that.
I want to get an answer to the

00:45:14.480 --> 00:45:20.557
bigger picture question.
I don't feel a preference

00:45:20.640 --> 00:45:25.037
for a human editor that's been working for
me for six months versus an AI that's been

00:45:25.120 --> 00:45:31.200
working with me for six months, what year
do you predict that that will be the case?

00:45:33.840 --> 00:45:37.251
My guess for that is there's a lot
of problems where basically we can do

00:45:37.335 --> 00:45:38.077
this when we have

00:45:38.160 --> 00:45:45.160
the "country of geniuses in a data
center". My picture for that,

00:45:45.375 --> 00:45:47.997
if you made me guess, is

00:45:48.080 --> 00:45:52.460
one to two years, maybe one to three
years. It's really hard to tell.

00:45:52.543 --> 00:45:54.077
I have a strong view—99%,

00:45:54.160 --> 00:45:59.920
95%—that all this will happen in 10 years.
I think that's just a super safe bet.

00:46:00.560 --> 00:46:04.637
I have a hunch—this is more like a 50/50
thing—that it's going to be more like

00:46:04.720 --> 00:46:07.867
one to two, maybe more like one
to three. So one to three years.

00:46:07.950 --> 00:46:08.960
Country of geniuses,

00:46:10.160 --> 00:46:14.237
and the slightly less economically
valuable task of editing videos.

00:46:14.320 --> 00:46:16.957
It seems pretty economically
valuable, let me tell you.

00:46:17.040 --> 00:46:19.134
It's just there are a lot
of use cases like that.

00:46:19.218 --> 00:46:20.557
There are a lot of similar ones.

00:46:20.640 --> 00:46:23.597
So you're predicting that
within one to three years.

00:46:23.680 --> 00:46:27.788
And then, generally, Anthropic has
predicted that by late '26 or early '27 we

00:46:27.871 --> 00:46:28.877
will have AI systems

00:46:28.960 --> 00:46:32.979
that "have the ability to navigate
interfaces available to humans doing

00:46:33.062 --> 00:46:34.077
digital work today,

00:46:34.160 --> 00:46:37.958
intellectual capabilities matching
or exceeding that of Nobel Prize winners,

00:46:38.041 --> 00:46:38.877
and the ability to

00:46:38.960 --> 00:46:42.317
interface with the physical world".
You gave an interview two months ago

00:46:42.400 --> 00:46:48.717
with DealBook where you were emphasizing
your company's more responsible compute

00:46:48.800 --> 00:46:52.700
scaling as compared to your competitors.
I'm trying to square these two views.

00:46:52.960 --> 00:46:57.517
If you really believe that we're going
to have a country of geniuses, you want as

00:46:57.600 --> 00:47:00.557
big a data center as you can get.
There's no reason to slow down.

00:47:00.640 --> 00:47:04.877
The TAM of a Nobel Prize winner, that can
actually do everything a Nobel Prize

00:47:04.960 --> 00:47:10.477
winner can do, is trillions of dollars.
So I'm trying to square this conservatism,

00:47:10.560 --> 00:47:13.933
which seems rational if you
have more moderate timelines,

00:47:14.016 --> 00:47:16.320
with your stated views about progress.

00:47:16.880 --> 00:47:20.170
It actually all fits together.
We go back to this fast,

00:47:20.253 --> 00:47:22.400
but not infinitely fast, diffusion.

00:47:23.120 --> 00:47:26.271
Let's say that we're making
progress at this rate.

00:47:26.354 --> 00:47:29.200
The technology is making
progress this fast.

00:47:29.920 --> 00:47:36.920
I have very high conviction that we're
going to get there within a few years.

00:47:39.040 --> 00:47:41.357
I have a hunch that we're going
to get there within a year or two.

00:47:41.440 --> 00:47:46.717
So there’s a little uncertainty
on the technical side, but pretty strong

00:47:46.800 --> 00:47:51.517
confidence that it won't be off by much.
What I'm less certain about is, again,

00:47:51.600 --> 00:47:56.557
the economic diffusion side.
I really do believe that we could

00:47:56.640 --> 00:48:03.640
have models that are a country of geniuses
in the data center in one to two years.

00:48:03.920 --> 00:48:10.920
One question is: How many years after that
do the trillions in revenue start rolling

00:48:13.034 --> 00:48:13.868
in?

00:48:14.160 --> 00:48:18.080
I don't think it's guaranteed
that it's going to be immediate.

00:48:19.440 --> 00:48:26.440
It could be one year, it could be two
years, I could even stretch it to five

00:48:27.120 --> 00:48:32.482
years although I'm skeptical of that.
So we have this uncertainty.

00:48:32.565 --> 00:48:34.957
Even if the technology goes as

00:48:35.040 --> 00:48:41.006
fast as I suspect that it will, we don't
know exactly how fast it's going to drive

00:48:41.090 --> 00:48:45.699
revenue. We know it's coming, but with
the way you buy these data centers,

00:48:45.782 --> 00:48:49.997
if you're off by a couple years,
that can be ruinous. It is just like how I

00:48:50.080 --> 00:48:54.850
wrote in "Machines of Loving Grace".
I said I think we might get this powerful

00:48:54.933 --> 00:48:57.837
AI, this "country of genius in the data
center". That description you gave comes

00:48:57.920 --> 00:49:02.717
from "Machines of Loving Grace". I said
we'll get that in 2026, maybe 2027. Again,

00:49:02.800 --> 00:49:06.840
that is my hunch. I wouldn't be
surprised if I'm off by a year or two,

00:49:06.923 --> 00:49:10.381
but that is my hunch. Let's say
that happens. That's the starting gun.

00:49:10.464 --> 00:49:12.814
How long does it take
to cure all the diseases?

00:49:13.360 --> 00:49:18.582
That's one of the ways that drives
a huge amount of economic value.

00:49:18.665 --> 00:49:21.037
You cure every disease. There's

00:49:21.120 --> 00:49:23.917
a question of how much of that goes to the
pharmaceutical company or the AI company,

00:49:24.000 --> 00:49:28.425
but there's an enormous consumer surplus
because —assuming we can get access

00:49:28.509 --> 00:49:29.197
for everyone,

00:49:29.280 --> 00:49:34.157
which I care about greatly—we cure all of
these diseases. How long does it take? You

00:49:34.240 --> 00:49:40.877
have to do the biological discovery,
you have to manufacture the new drug,

00:49:40.960 --> 00:49:43.255
you have to go through
the regulatory process.

00:49:43.339 --> 00:49:45.200
We saw this with vaccines and COVID.

00:49:47.680 --> 00:49:51.120
We got the vaccine out to everyone,
but it took a year and a half.

00:49:52.000 --> 00:49:57.461
My question is: How long does it take
to get the cure for everything—which AI

00:49:57.544 --> 00:49:58.397
is the genius

00:49:58.480 --> 00:50:02.957
that can in theory invent—out to everyone?
How long from when that AI first exists

00:50:03.040 --> 00:50:08.080
in the lab to when diseases have
actually been cured for everyone?

00:50:09.920 --> 00:50:14.477
We've had a polio vaccine for 50 years.
We're still trying to eradicate it in the

00:50:14.560 --> 00:50:18.077
most remote corners of Africa.
The Gates Foundation is trying

00:50:18.160 --> 00:50:19.997
as hard as they can.
Others are trying as hard

00:50:20.080 --> 00:50:25.011
as they can. But that's difficult.
Again, I don't expect most of the economic

00:50:25.094 --> 00:50:30.005
diffusion to be as difficult as that.
That's the most difficult case.

00:50:30.089 --> 00:50:32.397
But there's a real dilemma here.

00:50:32.480 --> 00:50:39.037
Where I've settled on it is that it will
be faster than anything we've seen in the

00:50:39.120 --> 00:50:45.920
world, but it still has its limits.
So when we go to buying data centers,

00:50:47.120 --> 00:50:53.997
again, the curve I'm looking at is: we've
had a 10x a year increase every year.

00:50:54.080 --> 00:51:00.125
At the beginning of this year, we're
looking at $10 billion in annualized

00:51:00.208 --> 00:51:01.042
revenue.

00:51:02.320 --> 00:51:06.378
We have to decide how much compute to buy.

00:51:06.461 --> 00:51:10.717
It takes a year or two
to actually build out

00:51:10.800 --> 00:51:16.877
the data centers, to reserve the data
center. Basically I'm saying, "In 2027,

00:51:16.960 --> 00:51:23.960
how much compute do I get?"
I could assume that the

00:51:24.720 --> 00:51:31.117
revenue will continue growing 10x a year,
so it'll be $100 billion at the end of

00:51:31.200 --> 00:51:38.200
2026 and $1 trillion at the end of 2027.
Actually it would be $5 trillion dollars

00:51:39.520 --> 00:51:43.517
of compute because it would be
$1 trillion a year for five years.

00:51:43.600 --> 00:51:48.480
I could buy $1 trillion of compute
that starts at the end of 2027.

00:51:49.200 --> 00:51:54.267
If my revenue is not $1 trillion
dollars, if it's even $800 billion,

00:51:54.351 --> 00:51:56.237
there's no force on earth,

00:51:56.320 --> 00:52:01.417
there's no hedge on earth that could stop
me from going bankrupt if I buy that much

00:52:01.501 --> 00:52:02.334
compute.

00:52:03.200 --> 00:52:07.197
Even though a part of my brain wonders
if it's going to keep growing 10x,

00:52:07.280 --> 00:52:12.207
I can't buy $1 trillion
a year of compute in 2027.

00:52:12.290 --> 00:52:17.117
If I'm just off by a year
in that rate of growth,

00:52:17.200 --> 00:52:23.760
or if the growth rate is 5x a year instead
of 10x a year, then you go bankrupt.

00:52:25.360 --> 00:52:31.010
So you end up in a world where you're
supporting hundreds of billions,

00:52:31.093 --> 00:52:32.240
not trillions.

00:52:33.120 --> 00:52:38.477
You accept some risk that there's so much
demand that you can't support the revenue,

00:52:38.560 --> 00:52:43.597
and you accept some risk that you
got it wrong and it's still slow.

00:52:43.680 --> 00:52:48.618
When I talked about behaving responsibly,
what I meant actually was not the absolute

00:52:48.702 --> 00:52:49.535
amount.

00:52:51.920 --> 00:52:55.677
I think it is true we're spending somewhat
less than some of the other players.

00:52:55.760 --> 00:53:00.578
It's actually the other things, like
have we been thoughtful about it or are we

00:53:00.661 --> 00:53:05.117
YOLOing and saying, "We're going to do
$100 billion here or $100 billion there"?

00:53:05.200 --> 00:53:09.597
I get the impression that some of the
other companies have not written down

00:53:09.680 --> 00:53:12.557
the spreadsheet, that they don't really
understand the risks they're taking.

00:53:12.640 --> 00:53:16.176
They're just doing stuff
because it sounds cool.

00:53:16.259 --> 00:53:19.117
We've thought carefully about it. We're

00:53:19.200 --> 00:53:23.835
an enterprise business. Therefore,
we can rely more on revenue.

00:53:23.918 --> 00:53:26.157
It's less fickle than consumer.

00:53:26.240 --> 00:53:30.784
We have better margins, which is the
buffer between buying too much and buying

00:53:30.867 --> 00:53:31.437
too little.

00:53:31.520 --> 00:53:37.117
I think we bought an amount that allows
us to capture pretty strong upside worlds.

00:53:37.200 --> 00:53:42.157
It won't capture the full 10x a year.
Things would have to go pretty badly for

00:53:42.240 --> 00:53:46.397
us to be in financial trouble.
So we've thought carefully and

00:53:46.480 --> 00:53:48.797
we've made that balance.
That's what I mean when

00:53:48.880 --> 00:53:53.997
I say that we're being responsible.
So it seems like it's possible that we

00:53:54.080 --> 00:53:56.609
actually just have different definitions
of the "country of a genius in a data

00:53:56.692 --> 00:53:59.479
center". Because when I think
of actual human geniuses,

00:53:59.562 --> 00:54:02.477
an actual country of human
geniuses in a data center,

00:54:02.560 --> 00:54:08.237
I would happily buy $5 trillion worth
of compute to run an actual country of

00:54:08.320 --> 00:54:11.277
human geniuses in a data center.
Let's say JPMorgan or Moderna or

00:54:11.360 --> 00:54:14.557
whatever doesn't want to use them.
I've got a country of geniuses.

00:54:14.640 --> 00:54:17.558
They'll start their own company. If they
can't start their own company and they're

00:54:17.641 --> 00:54:20.770
bottlenecked by clinical trials… It is
worth stating that with clinical trials,

00:54:20.853 --> 00:54:24.724
most clinical trials fail because the drug
doesn't work. There's not efficacy.

00:54:24.808 --> 00:54:27.117
I make exactly that point in "Machines of

00:54:27.200 --> 00:54:30.077
Loving Grace", I say the clinical
trials are going to go much faster

00:54:30.160 --> 00:54:33.332
than we're used to,
but not infinitely fast. Okay,

00:54:33.416 --> 00:54:35.677
and then suppose it takes a year for

00:54:35.760 --> 00:54:38.410
the clinical trials to work out so that
you're getting revenue from that and can

00:54:38.493 --> 00:54:42.643
make more drugs. Okay, well, you've got a
country of geniuses and you're an AI lab.

00:54:44.000 --> 00:54:48.800
You could use many more AI researchers.
You also think there are these

00:54:48.884 --> 00:54:49.917
self-reinforcing

00:54:50.000 --> 00:54:55.997
gains from smart people working on AI
tech. You can have the data center

00:54:56.080 --> 00:55:01.517
working on AI progress.
Are there substantially

00:55:01.600 --> 00:55:06.783
more gains from buying $1 trillion a year
of compute versus $300 billion a year

00:55:06.867 --> 00:55:09.677
of compute? If your competitor
is buying a trillion, yes there is.

00:55:09.760 --> 00:55:12.705
Well, no, there's some
gain, but then again,

00:55:12.788 --> 00:55:16.160
there's this chance that
they go bankrupt before.

00:55:17.040 --> 00:55:21.700
Again, if you're off by only
a year, you destroy yourselves.

00:55:21.784 --> 00:55:26.997
That's the balance. We're buying a lot.
We're buying a hell of a lot.

00:55:27.080 --> 00:55:30.317
We're buying an amount
that's comparable to

00:55:30.400 --> 00:55:36.648
what the biggest players in the game
are buying. But if you're asking me,

00:55:36.732 --> 00:55:38.640
"Why haven't we signed

00:55:39.520 --> 00:55:43.277
$10 trillion of compute starting
in mid-2027?"... First of all,

00:55:43.360 --> 00:55:44.557
it can't be produced.

00:55:44.640 --> 00:55:50.157
There isn't that much in the world.
But second, what if the country of

00:55:50.240 --> 00:55:55.997
geniuses comes, but it comes in mid-2028
instead of mid-2027? You go bankrupt.

00:55:56.080 --> 00:56:00.397
So if your projection is one to three
years, it seems like you should want

00:56:00.480 --> 00:56:06.080
$10 trillion of compute
by 2029 at the latest?

00:56:06.164 --> 00:56:11.517
Even in the longest
version of the timelines

00:56:11.600 --> 00:56:16.237
you state, the compute you are ramping
up to build doesn't seem in accordance.

00:56:16.320 --> 00:56:21.357
What makes you think that?
Human wages, let's say,

00:56:21.440 --> 00:56:26.246
are on the order of $50 trillion a year—
So I won't talk about Anthropic

00:56:26.329 --> 00:56:27.197
in particular,

00:56:27.280 --> 00:56:34.280
but if you talk about the industry, the
amount of compute the industry is building

00:56:37.050 --> 00:56:38.397
this year is

00:56:38.480 --> 00:56:45.480
probably, call it, 10-15 gigawatts.
It goes up by roughly 3x a year.

00:56:48.480 --> 00:56:54.797
So next year's 30-40 gigawatts.
2028 might be 100 gigawatts.

00:56:54.880 --> 00:56:58.400
2029 might be like 300 gigawatts.

00:57:03.760 --> 00:57:07.277
I'm doing the math in my head,
but each gigawatt costs maybe $10 billion,

00:57:07.360 --> 00:57:13.997
on the order of $10-15 billion a year.
You put that all together and you're

00:57:14.080 --> 00:57:15.514
getting about what you described.
You’re getting exactly that.

00:57:15.597 --> 00:57:16.077
You're getting multiple

00:57:16.160 --> 00:57:22.480
trillions a year by 2028 or 2029.
You're getting exactly what you predict.

00:57:23.040 --> 00:57:25.917
That's for the industry.
That's for the industry, that’s right.

00:57:26.000 --> 00:57:31.983
Suppose Anthropic's compute keeps
3x-ing a year, and then by 2027-28,

00:57:32.066 --> 00:57:34.000
you have 10 gigawatts.

00:57:34.960 --> 00:57:40.557
Multiply that by, as you say, $10 billion.
So then it's like $100 billion a year.

00:57:40.640 --> 00:57:43.267
But then you're saying the TAM
by 2028 is $200 billion.

00:57:43.350 --> 00:57:46.328
Again, I don't want to give
exact numbers for Anthropic,

00:57:46.411 --> 00:57:48.077
but these numbers are too small.

00:57:48.160 --> 00:57:55.160
Okay, interesting. You've told investors

00:58:49.920 --> 00:58:52.666
that you plan to be
profitable starting in 2028.

00:58:52.749 --> 00:58:55.437
This is the year when
we're potentially getting

00:58:55.520 --> 00:59:01.073
the country of geniuses as a data center.
This is now going to unlock all

00:59:01.156 --> 00:59:06.190
this progress in medicine
and health and new technologies.

00:59:06.273 --> 00:59:10.480
Wouldn't this be exactly
the time where you'd

00:59:11.040 --> 00:59:14.661
want to reinvest in the business and build
bigger "countries" so they can make more

00:59:14.744 --> 00:59:15.578
discoveries?

00:59:16.320 --> 00:59:19.280
Profitability is this kind
of weird thing in this field.

00:59:21.760 --> 00:59:28.760
I don't think in this field profitability
is actually a measure of spending down

00:59:32.000 --> 00:59:35.997
versus investing in the business.
Let's just take a model of this.

00:59:36.080 --> 00:59:39.729
I actually think profitability happens
when you underestimated the amount

00:59:39.813 --> 00:59:40.957
of demand you were going

00:59:41.040 --> 00:59:44.977
to get and loss happens when you
overestimated the amount of demand you

00:59:45.061 --> 00:59:49.984
were going to get, because you're
buying the data centers ahead of time.

00:59:50.068 --> 00:59:52.317
Think about it this way. Again,

00:59:52.400 --> 00:59:54.818
these are stylized facts.
These numbers are not exact.

00:59:54.901 --> 00:59:56.717
I'm just trying to make a toy model here.

00:59:56.800 --> 01:00:00.686
Let's say half of your compute is for
training and half of your compute is

01:00:00.769 --> 01:00:01.602
for inference.

01:00:02.640 --> 01:00:06.480
The inference has some gross
margin that's more than 50%.

01:00:07.200 --> 01:00:10.223
So what that means is that if
you were in steady-state,

01:00:10.306 --> 01:00:16.882
you build a data center and if you knew
exactly the demand you were getting,

01:00:16.966 --> 01:00:21.600
you would get a certain amount of revenue.

01:00:23.120 --> 01:00:28.637
Let’s say you pay $100 billion a year for
compute. On $50 billion a year you support

01:00:28.720 --> 01:00:35.120
$150 billion of revenue. The other
$50 billion is used for training.

01:00:36.240 --> 01:00:39.997
Basically you’re profitable
and you make $50 billion of profit.

01:00:40.080 --> 01:00:45.517
Those are the economics of the industry
today, or not today but where we’re

01:00:45.600 --> 01:00:49.037
projecting forward in a year or two.
The only thing that makes that not the

01:00:49.120 --> 01:00:53.443
case is if you get less
demand than $50 billion.

01:00:53.526 --> 01:00:57.757
Then you have more than
50% of your data center

01:00:57.840 --> 01:01:01.357
for research and you're not profitable.
So you train stronger models,

01:01:01.440 --> 01:01:07.037
but you're not profitable. If you
get more demand than you thought, then

01:01:07.120 --> 01:01:13.152
research gets squeezed, but you're kind of
able to support more inference and you're

01:01:13.235 --> 01:01:14.400
more profitable.

01:01:16.000 --> 01:01:18.957
Maybe I'm not explaining it well, but
the thing I'm trying to say is that you

01:01:19.040 --> 01:01:24.317
decide the amount of compute first.
Then you have some target desire of

01:01:24.400 --> 01:01:28.637
inference versus training,
but that gets determined by demand.

01:01:28.720 --> 01:01:30.637
It doesn't get determined by you.
What I'm hearing is the reason

01:01:30.720 --> 01:01:35.920
you're predicting profit is that you are
systematically underinvesting in compute?

01:01:37.040 --> 01:01:40.132
No, no, no. I'm saying
it's hard to predict.

01:01:40.216 --> 01:01:43.597
These things about 2028
and when it will happen,

01:01:43.680 --> 01:01:46.477
that's our attempt to do
the best we can with investors.

01:01:46.560 --> 01:01:50.397
All of this stuff is really uncertain
because of the cone of uncertainty.

01:01:50.480 --> 01:01:55.440
We could be profitable in 2026
if the revenue grows fast enough.

01:01:58.960 --> 01:02:03.040
If we overestimate or underestimate
the next year, that could swing wildly.

01:02:04.400 --> 01:02:08.834
What I'm trying to get at is that you
have a model in your head of a business

01:02:08.917 --> 01:02:13.917
that invests, invests, invests,
gets scale and then becomes profitable.

01:02:14.000 --> 01:02:16.157
There's a single point
at which things turn around.

01:02:16.240 --> 01:02:19.277
I don't think the economics
of this industry work that way.

01:02:19.360 --> 01:02:23.306
I see. So if I'm understanding correctly,
you're saying that because of

01:02:23.389 --> 01:02:24.157
the discrepancy

01:02:24.240 --> 01:02:27.128
between the amount of compute we should
have gotten and the amount of compute we

01:02:27.211 --> 01:02:30.637
got, we were sort of forced to make
profit. But that doesn't mean we're going

01:02:30.720 --> 01:02:33.437
to continue making profit.
We're going to reinvest the money

01:02:33.520 --> 01:02:37.757
because now AI has made so much progress
and we want a bigger country of geniuses.

01:02:37.840 --> 01:02:42.560
So back into revenue is high,
but losses are also high.

01:02:44.160 --> 01:02:48.119
If every year we predict exactly
what the demand is going to be,

01:02:48.202 --> 01:02:50.077
we'll be profitable every year.

01:02:50.160 --> 01:02:56.215
Because spending 50% of your
compute on research, roughly,

01:02:56.298 --> 01:03:00.237
plus a gross margin that's higher than

01:03:00.320 --> 01:03:03.711
50% and correct demand
prediction leads to profit.

01:03:03.795 --> 01:03:07.117
That's the profitable
business model that I think

01:03:07.200 --> 01:03:13.437
is kind of there, but obscured by these
building ahead and prediction errors.

01:03:13.520 --> 01:03:20.520
I guess you're treating the 50% as a sort
of given constant, whereas in fact,

01:03:21.040 --> 01:03:23.680
if AI progress is fast and you can
increase the progress by scaling up more,

01:03:23.763 --> 01:03:26.717
you should just have more than 50% and not
make profit. But here's what I'll say. You

01:03:26.800 --> 01:03:33.040
might want to scale it up more.
Remember the log returns to scale.

01:03:34.080 --> 01:03:40.480
If 70% would get you a very little
bit of a smaller model through a factor

01:03:40.564 --> 01:03:41.440
of 1.4x...

01:03:42.320 --> 01:03:49.296
That extra $20 billion, each dollar
there is worth much less to you because of

01:03:49.379 --> 01:03:51.197
the log-linear setup.

01:03:51.280 --> 01:03:58.280
So you might find that it's better
to invest that $20 billion in serving

01:03:58.960 --> 01:04:05.037
inference or in hiring engineers who
are kind of better at what they're doing.

01:04:05.120 --> 01:04:08.453
So the reason I said 50%...
That's not exactly our target.

01:04:08.536 --> 01:04:10.397
It's not exactly going to be 50%.

01:04:10.480 --> 01:04:16.520
It’ll probably vary over time.
What I'm saying is the log-linear return,

01:04:16.604 --> 01:04:18.477
what it leads to is you

01:04:18.560 --> 01:04:25.403
spend of order one fraction
of the business. Like not 5%, not 95%.

01:04:25.487 --> 01:04:28.867
Then you get diminishing returns.

01:04:28.950 --> 01:04:34.160
I feel strange that I'm convincing Dario
to believe in AI progress or something.

01:04:34.960 --> 01:04:37.837
Okay, you don't invest in research
because it has diminishing returns,

01:04:37.920 --> 01:04:38.762
but you invest in the other
things you mentioned.

01:04:38.845 --> 01:04:39.517
I think profit at a sort of macro level—

01:04:39.600 --> 01:04:42.357
Again, I'm talking about
diminishing returns,

01:04:42.440 --> 01:04:45.280
but after you're spending
$50 billion a year.

01:04:46.240 --> 01:04:51.597
This is a point I'm sure you would make,
but diminishing returns on a genius could

01:04:51.680 --> 01:04:53.997
be quite high. More generally,

01:04:54.080 --> 01:04:58.237
what is profit in a market economy?
Profit is basically saying other

01:04:58.320 --> 01:05:01.997
companies in the market can do more
things with this money than I can.

01:05:02.080 --> 01:05:06.477
Put aside Anthropic. I don't want
to give information about Anthropic.

01:05:06.560 --> 01:05:09.997
That’s why I'm giving these stylized
numbers. But let's just derive the

01:05:10.080 --> 01:05:17.080
equilibrium of the industry.
Why doesn't everyone spend 100% of their

01:05:21.040 --> 01:05:23.294
compute on training
and not serve any customers?

01:05:23.377 --> 01:05:25.437
It's because if they
didn't get any revenue,

01:05:25.520 --> 01:05:27.357
they couldn't raise money,
they couldn't do compute deals,

01:05:27.440 --> 01:05:29.463
they couldn't buy more
compute the next year.

01:05:29.546 --> 01:05:31.757
So there's going to be
an equilibrium where every

01:05:31.840 --> 01:05:38.717
company spends less than 100% on training
and certainly less than 100% on inference.

01:05:38.800 --> 01:05:43.804
It should be clear why you don't just
serve the current models and never train

01:05:43.887 --> 01:05:48.143
another model, because then you don't have
any demand because you'll fall behind.

01:05:48.226 --> 01:05:52.590
So there's some equilibrium. It's not
gonna be 10%, it's not gonna be 90%.

01:05:52.673 --> 01:05:55.517
Let's just say as a
stylized fact, it's 50%.

01:05:55.600 --> 01:05:59.460
That's what I'm getting at.
I think we're gonna be in a position where

01:05:59.543 --> 01:06:01.037
that equilibrium of how much

01:06:01.120 --> 01:06:07.466
you spend on training is less than
the gross margins that you're able to get

01:06:07.549 --> 01:06:11.517
on compute. So the underlying
economics are profitable.

01:06:11.600 --> 01:06:14.637
The problem is you
have this hellish demand

01:06:14.720 --> 01:06:20.359
prediction problem when you're buying the
next year of compute and you might guess

01:06:20.443 --> 01:06:26.408
under and be very profitable
but have no compute for research.

01:06:26.492 --> 01:06:30.557
Or you might guess over and you are not

01:06:30.640 --> 01:06:34.805
profitable and you have all
the compute for research in the world.

01:06:34.888 --> 01:06:36.157
Does that make sense?

01:06:36.240 --> 01:06:40.625
Just as a dynamic model of the industry?
Maybe stepping back,

01:06:40.708 --> 01:06:42.237
I'm not saying I think

01:06:42.320 --> 01:06:45.631
the "country of geniuses" is going to come
in two years and therefore you should buy

01:06:45.715 --> 01:06:46.548
this compute.

01:06:47.120 --> 01:06:51.517
To me, the end conclusion you're
arriving at makes a lot of sense.

01:06:51.600 --> 01:06:56.929
But that's because it seems like "country
of geniuses" is hard and there's a long

01:06:57.012 --> 01:06:57.597
way to go.

01:06:57.680 --> 01:07:02.451
So stepping back, the thing I'm trying
to get at is more that it seems like

01:07:02.534 --> 01:07:03.357
your worldview

01:07:03.440 --> 01:07:07.140
is compatible with somebody who says,
"We're like 10 years away from a world

01:07:07.223 --> 01:07:07.837
in which we're

01:07:07.920 --> 01:07:14.637
generating trillions of dollars of value."
That's just not my view. So I'll make

01:07:14.720 --> 01:07:19.917
another prediction. It is hard for me
to see that there won't be trillions

01:07:20.000 --> 01:07:26.077
of dollars in revenue before 2030.
I can construct a plausible world.

01:07:26.160 --> 01:07:31.197
It takes maybe three years. That would
be the end of what I think it's plausible.

01:07:31.280 --> 01:07:36.317
Like in 2028, we get the real "country
of geniuses in the data center".

01:07:36.400 --> 01:07:43.400
The revenue's going into the low hundreds
of billions by 2028, and then the country

01:07:46.400 --> 01:07:51.331
of geniuses accelerates it to trillions.
We’re basically on the slow end

01:07:51.415 --> 01:07:55.714
of diffusion. It takes two
years to get to the trillions.

01:07:55.797 --> 01:07:59.757
That would be the world
where it takes until 2030.

01:07:59.840 --> 01:08:05.357
I suspect even composing the technical
exponential and diffusion exponential,

01:08:05.440 --> 01:08:10.397
we’ll get there before 2030. So you
laid out a model where Anthropic makes

01:08:10.480 --> 01:08:14.717
profit because it seems like fundamentally
we're in a compute-constrained world.

01:08:14.800 --> 01:08:20.160
So eventually we keep growing compute— I
think the way the profit comes is… Again,

01:08:21.040 --> 01:08:23.765
let's just abstract
the whole industry here.

01:08:23.849 --> 01:08:26.957
Let's just imagine we're
in an economics textbook.

01:08:27.040 --> 01:08:32.320
We have a small number of firms.
Each can invest a limited amount.

01:08:33.680 --> 01:08:38.477
Each can invest some fraction in R&D.
They have some marginal cost to serve.

01:08:38.560 --> 01:08:45.560
The gross profit margins on that marginal
cost are very high because inference

01:08:45.829 --> 01:08:50.160
is efficient. There's some competition,
but the models are also differentiated.

01:08:52.720 --> 01:08:55.517
Companies will compete to push
their research budgets up.

01:08:55.600 --> 01:09:00.557
But because there's a small number of
players, we have the... What is it called?

01:09:00.640 --> 01:09:05.837
The Cournot equilibrium, I think, is what
the small number of firm equilibrium is.

01:09:05.920 --> 01:09:12.920
The point is it doesn't equilibrate
to perfect competition with zero margins.

01:09:15.600 --> 01:09:20.158
If there's three firms in the economy
and all are kind of independently behaving

01:09:20.242 --> 01:09:24.797
rationally, it doesn't equilibrate
to zero. Help me understand that, because

01:09:24.880 --> 01:09:30.165
right now we do have three leading
firms and they're not making profit.

01:09:30.248 --> 01:09:31.760
So what is changing?

01:09:33.840 --> 01:09:38.077
Again, the gross margins
right now are very positive.

01:09:38.160 --> 01:09:41.106
What's happening is a
combination of two things.

01:09:41.189 --> 01:09:43.757
One is that we're still in the exponential

01:09:43.840 --> 01:09:50.840
scale-up phase of compute. A model
gets trained. Let's say a model got

01:09:53.600 --> 01:10:00.600
trained that costs $1 billion last year.
Then this year it produced $4 billion of

01:10:02.800 --> 01:10:08.129
revenue and cost $1 billion
to inference from. Again,

01:10:08.213 --> 01:10:12.317
I'm using stylized numbers here, but that

01:10:12.400 --> 01:10:17.842
would be 75% gross
margins and this 25% tax.

01:10:17.926 --> 01:10:23.117
So that model as a whole makes $2 billion.

01:10:23.200 --> 01:10:27.437
But at the same time, we're spending $10
billion to train the next model because

01:10:27.520 --> 01:10:31.117
there's an exponential scale-up.
So the company loses money. Each model

01:10:31.200 --> 01:10:35.597
makes money, but the company loses money.
The equilibrium I'm talking about is an

01:10:35.680 --> 01:10:41.440
equilibrium where we have the "country
of geniuses in a data center", but that

01:10:43.440 --> 01:10:47.954
model training scale-up has equilibrated
more. Maybe it's still going up.

01:10:48.037 --> 01:10:52.507
We're still trying to predict the demand,
but it's more leveled out.

01:10:52.590 --> 01:10:55.680
I'm confused about
a couple of things there.

01:10:56.720 --> 01:11:00.637
Let's start with the current world.
In the current world, you're right that,

01:11:00.720 --> 01:11:04.328
as you said before, if you treat
each individual model as a company,

01:11:04.411 --> 01:11:05.197
it's profitable.

01:11:05.280 --> 01:11:10.361
But of course, a big part of the
production function of being a frontier

01:11:10.444 --> 01:11:12.957
lab is training the next model,
right? Yes, that's right.

01:11:13.040 --> 01:11:16.157
If you didn't do that, then you'd
make profit for two months and then

01:11:16.240 --> 01:11:19.117
you wouldn't have margins because
you wouldn't have the best model.

01:11:19.200 --> 01:11:23.597
But at some point that reaches
the biggest scale that it can reach.

01:11:23.680 --> 01:11:26.708
And then in equilibrium,
we have algorithmic improvements,

01:11:26.791 --> 01:11:28.317
but we're spending roughly the

01:11:28.400 --> 01:11:32.720
same amount to train the next model
as we spend to train the current model.

01:11:37.520 --> 01:11:40.131
At some point you run out
of money in the economy.

01:11:40.215 --> 01:11:42.557
A fixed lump of labor
fallacy… The economy is

01:11:42.640 --> 01:11:44.317
going to grow, right? That's one
of your predictions. We're going

01:11:44.400 --> 01:11:47.357
to have the data centers in space.
Yes, but this is another example

01:11:47.440 --> 01:11:53.037
of the theme I was talking about.
The economy will grow much faster

01:11:53.120 --> 01:11:59.190
with AI than I think it ever has before.
Right now the compute is growing 3x

01:11:59.273 --> 01:12:03.597
a year. I don't believe the economy
is gonna grow 300% a year.

01:12:03.680 --> 01:12:08.157
I said this in "Machines of Loving
Grace", I think we may get 10-20%

01:12:08.240 --> 01:12:13.197
per year growth in the economy, but we're
not gonna get 300% growth in the economy.

01:12:13.280 --> 01:12:18.877
So I think in the end, if compute becomes
the majority of what the economy produces,

01:12:18.960 --> 01:12:22.557
it's gonna be capped by that.
So let's assume a model

01:12:22.640 --> 01:12:26.877
where compute stays capped. The world
where frontier labs are making money

01:12:26.960 --> 01:12:30.715
is one where they continue
to make fast progress.

01:12:30.798 --> 01:12:34.397
Because fundamentally
your margin is limited by

01:12:34.480 --> 01:12:36.957
how good the alternative is.
So you are able to make money

01:12:37.040 --> 01:12:39.437
because you have a frontier model.
If you didn't have a frontier model

01:12:39.520 --> 01:12:45.037
you wouldn't be making money.
So this model requires there

01:12:45.120 --> 01:12:47.997
never to be a steady state.
Forever and ever you keep

01:12:48.080 --> 01:12:51.037
making more algorithmic progress.
I don't think that's true. I mean,

01:12:51.120 --> 01:12:58.120
I feel like we're in an economics class.
Do you know the Tyler Cowen quote?

01:12:59.920 --> 01:13:03.677
We never stop talking about economics.
We never stop talking about economics.

01:13:03.760 --> 01:13:10.760
So no, I don't think this field's
going to be a monopoly.

01:13:12.560 --> 01:13:15.117
All my lawyers never want me
to say the word "monopoly".

01:13:15.200 --> 01:13:17.837
But I don't think this field's
going to be a monopoly.

01:13:17.920 --> 01:13:21.437
You do get industries in which
there are a small number of players.

01:13:21.520 --> 01:13:27.517
Not one, but a small number of players.
Ordinarily, the way you get monopolies

01:13:27.600 --> 01:13:34.600
like Facebook or Meta—I always call
them Facebook—is these kinds of network

01:13:36.275 --> 01:13:40.957
effects. The way you get industries in
which there are a small number of players,

01:13:41.040 --> 01:13:45.538
is very high costs of entry.
Cloud is like this.

01:13:45.622 --> 01:13:49.357
I think cloud is a good example of this.

01:13:49.440 --> 01:13:52.197
There are three, maybe four,
players within cloud.

01:13:52.280 --> 01:13:55.120
I think that's the same
for AI, three, maybe four.

01:13:56.640 --> 01:14:02.157
The reason is that it's so expensive.
It requires so much expertise and so

01:14:02.240 --> 01:14:08.157
much capital to run a cloud company.
You have to put up all this capital.

01:14:08.240 --> 01:14:10.044
In addition to putting
up all this capital,

01:14:10.128 --> 01:14:11.757
you have to get all of this other stuff

01:14:11.840 --> 01:14:14.527
that requires a lot
of skill to make it happen.

01:14:14.611 --> 01:14:17.357
So if you go to someone
and you're like, "I want

01:14:17.440 --> 01:14:21.063
to disrupt this industry, here's
$100 billion." You're like, "okay,

01:14:21.146 --> 01:14:22.557
I'm putting in $100 billion

01:14:22.640 --> 01:14:25.941
and also betting that you can do all these
other things that these people have been

01:14:26.025 --> 01:14:29.277
doing." Only to decrease the profit.
The effect of your entering

01:14:29.360 --> 01:14:33.197
is that profit margins go down.
So, we have equilibria like this

01:14:33.280 --> 01:14:37.147
all the time in the economy
where we have a few players.

01:14:37.230 --> 01:14:39.757
Profits are not astronomical. Margins

01:14:39.840 --> 01:14:46.059
are not astronomical, but they're
not zero. That's what we see on cloud.

01:14:46.142 --> 01:14:50.800
Cloud is very undifferentiated.
Models are more differentiated than cloud.

01:14:51.680 --> 01:14:57.076
Everyone knows Claude is good
at different things than GPT is good at,

01:14:57.160 --> 01:14:58.877
than Gemini is good at.

01:14:58.960 --> 01:15:03.360
It's not just that Claude's good at
coding, GPT is good at math and reasoning.

01:15:05.120 --> 01:15:08.578
It's more subtle than that. Models
are good at different types of coding.

01:15:08.661 --> 01:15:09.597
Models have different

01:15:09.680 --> 01:15:14.270
styles. I think these things are
actually quite different from each other,

01:15:14.354 --> 01:15:15.597
and so I would expect

01:15:15.680 --> 01:15:21.688
more differentiation than
you see in cloud. Now,

01:15:21.771 --> 01:15:26.637
there actually is one counter-argument.

01:15:26.720 --> 01:15:30.320
That counter-argument is if
the process of producing models,

01:15:32.960 --> 01:15:37.517
if AI models can do that themselves, then
that could spread throughout the economy.

01:15:37.600 --> 01:15:41.197
But that is not an argument
for commoditizing AI models in general.

01:15:41.280 --> 01:15:45.357
That's kind of an argument for
commoditizing the whole economy at once.

01:15:45.440 --> 01:15:48.877
I don't know what quite happens
in that world where basically anyone

01:15:48.960 --> 01:15:51.277
can do anything,
anyone can build anything,

01:15:51.360 --> 01:15:53.677
and there's no moat
around anything at all.

01:15:53.760 --> 01:15:58.477
I don't know, maybe we want that world.
Maybe that's the end state here.

01:15:58.560 --> 01:16:05.560
Maybe when AI models can do everything, if
we've solved all the safety and security

01:16:08.231 --> 01:16:09.197
problems,

01:16:09.280 --> 01:16:16.280
that's one of the mechanisms for the
economy just flattening itself again.

01:16:17.920 --> 01:16:22.480
But that's kind of far post-"country
of geniuses in the data center."

01:16:23.440 --> 01:16:28.224
Maybe a finer way to put
that potential point is:

01:16:28.308 --> 01:16:32.397
1) it seems like AI research is especially

01:16:32.480 --> 01:16:37.252
loaded on raw intellectual power, which
will be especially abundant in the world

01:16:37.335 --> 01:16:39.664
of AGI. And 2) if you just
look at the world today,

01:16:39.748 --> 01:16:41.837
there are very few
technologies that seem to be

01:16:41.920 --> 01:16:46.208
diffusing as fast as AI
algorithmic progress.

01:16:46.291 --> 01:16:49.997
So that does hint that this industry is

01:16:50.080 --> 01:16:54.157
sort of structurally diffusive.
I think coding is going fast, but

01:16:54.240 --> 01:16:57.445
I think AI research is a superset
of coding and there are aspects of it

01:16:57.528 --> 01:16:58.728
that are not going fast.

01:17:00.320 --> 01:17:05.641
But I do think, again, once we get coding,
once we get AI models going fast,

01:17:05.724 --> 01:17:07.277
then that will speed up

01:17:07.360 --> 01:17:12.720
the ability of AI models to do everything
else. So while coding is going fast now,

01:17:12.803 --> 01:17:13.517
I think once

01:17:13.600 --> 01:17:17.437
the AI models are building the next
AI models and building everything else,

01:17:17.520 --> 01:17:21.657
the whole economy will
kind of go at the same pace.

01:17:21.741 --> 01:17:24.637
I am worried geographically, though.

01:17:24.720 --> 01:17:31.695
I'm a little worried that just
proximity to AI, having heard about AI,

01:17:31.778 --> 01:17:34.317
may be one differentiator.

01:17:34.400 --> 01:17:41.169
So when I said the 10-20% growth rate, a
worry I have is that the growth rate could

01:17:41.252 --> 01:17:42.077
be like 50%

01:17:42.160 --> 01:17:48.461
in Silicon Valley and parts of the world
that are socially connected to Silicon

01:17:48.544 --> 01:17:52.100
Valley, and not that much faster
than its current pace elsewhere.

01:17:52.183 --> 01:17:54.077
I think that'd be
a pretty messed up world.

01:17:54.160 --> 01:17:57.197
So one of the things I think
about a lot is how to prevent that.

01:17:57.280 --> 01:18:01.837
Do you think that once we have this
country of geniuses in a data center, that

01:18:01.920 --> 01:18:03.928
robotics is sort of quickly
solved afterwards?

01:18:04.012 --> 01:18:06.157
Because it seems like
a big problem with robotics

01:18:06.240 --> 01:18:10.676
is that a human can learn how
to teleoperate current hardware,

01:18:10.759 --> 01:18:14.802
but current AI models can't, at least
not in a way that's super productive.

01:18:14.885 --> 01:18:16.797
And so if we have
this ability to learn like

01:18:16.880 --> 01:18:19.037
a human, shouldn't it solve
robotics immediately as well?

01:18:19.120 --> 01:18:21.437
I don't think it's dependent
on learning like a human.

01:18:21.520 --> 01:18:25.357
It could happen in different ways.
Again, we could have trained the model on

01:18:25.440 --> 01:18:28.357
many different video games,
which are like robotic controls,

01:18:28.440 --> 01:18:30.157
or many different simulated robotics

01:18:30.240 --> 01:18:33.247
environments, or just train
them to control computer screens,

01:18:33.331 --> 01:18:34.717
and they learn to generalize.

01:18:34.800 --> 01:18:41.597
So it will happen... it's not necessarily
dependent on human-like learning.

01:18:41.680 --> 01:18:44.228
Human-like learning is one way it
could happen. If the model's like, "Oh,

01:18:44.311 --> 01:18:44.877
I pick up a robot,

01:18:44.960 --> 01:18:49.497
I don't know how to use it, I learn,"
that could happen because we discovered

01:18:49.580 --> 01:18:50.637
continual learning.

01:18:50.720 --> 01:18:53.917
That could also happen because we trained
the model on a bunch of environments and

01:18:54.000 --> 01:18:57.367
then generalized, or it could happen
because the model learns that in

01:18:57.450 --> 01:18:58.400
the context length.

01:18:58.960 --> 01:19:03.117
It doesn't actually matter which way.
If we go back to the discussion we had

01:19:03.200 --> 01:19:08.800
an hour ago, that type of thing can
happen in several different ways.

01:19:10.240 --> 01:19:15.186
But I do think when for whatever
reason the models have those skills,

01:19:15.269 --> 01:19:16.717
then robotics will be

01:19:16.800 --> 01:19:20.461
revolutionized—both the design of robots,
because the models will be much better

01:19:20.545 --> 01:19:24.959
than humans at that, and also
the ability to control robots.

01:19:25.043 --> 01:19:28.717
So we'll get better
at building the physical

01:19:28.800 --> 01:19:32.109
hardware, building the physical robots,
and we'll also get better at controlling

01:19:32.193 --> 01:19:36.717
it. Now, does that mean the robotics
industry will also be generating

01:19:36.800 --> 01:19:40.397
trillions of dollars of revenue?
My answer there is yes, but there will be

01:19:40.480 --> 01:19:44.213
the same extremely fast,
but not infinitely fast diffusion.

01:19:44.296 --> 01:19:46.477
So will robotics be revolutionized?

01:19:46.560 --> 01:19:51.840
Yeah, maybe tack on another year or two.
That's the way I think about these things.

01:19:52.400 --> 01:19:57.127
Makes sense. There's a general
skepticism about extremely fast progress.

01:19:57.210 --> 01:19:58.797
Here's my view. It sounds

01:19:58.880 --> 01:20:01.933
like you are going to solve continual
learning one way or another within

01:20:02.016 --> 01:20:02.717
a matter of years.

01:20:02.800 --> 01:20:06.717
But just as people weren't talking about
continual learning a couple of years ago,

01:20:06.800 --> 01:20:09.478
and then we realized, "Oh, why aren't
these models as useful as they could be

01:20:09.561 --> 01:20:09.837
right now,

01:20:09.920 --> 01:20:13.225
even though they are clearly passing
the Turing test and are experts in so many

01:20:13.308 --> 01:20:13.997
different domains?

01:20:14.080 --> 01:20:18.483
Maybe it's this thing." Then we solve
this thing and we realize, actually,

01:20:18.567 --> 01:20:19.757
there's another thing

01:20:19.840 --> 01:20:23.558
that human intelligence can do that's
a basis of human labor that these models

01:20:23.641 --> 01:20:27.600
can't do. So why not think there
will be more things like this, where

01:20:28.560 --> 01:20:31.980
we've found more pieces of human
intelligence? Well, to be clear,

01:20:32.063 --> 01:20:36.982
I think continual learning, as I've said
before, might not be a barrier at all.

01:20:37.066 --> 01:20:40.157
I think we may just get
there by pre-training

01:20:40.240 --> 01:20:45.280
generalization and RL generalization.
I think there just

01:20:48.240 --> 01:20:51.837
might not be such a thing at all.
In fact, I would point to the history

01:20:51.920 --> 01:20:56.637
in ML of people coming up with things
that are barriers that end up kind of

01:20:56.720 --> 01:21:03.720
dissolving within the big blob of compute.
People talked about, "How do your models

01:21:06.480 --> 01:21:11.837
keep track of nouns and verbs?"
"They can understand syntactically,

01:21:11.920 --> 01:21:16.637
but they can't understand semantically?
It's only statistical correlations."

01:21:16.720 --> 01:21:19.437
"You can understand a paragraph,
you can’t understand a word.

01:21:19.520 --> 01:21:21.348
There's reasoning,
you can't do reasoning."

01:21:21.431 --> 01:21:23.037
But then suddenly it turns out you can

01:21:23.120 --> 01:21:27.757
do code and math very well.
So I think there's actually a

01:21:27.840 --> 01:21:33.868
stronger history of some of these things
seeming like a big deal and then kind

01:21:33.951 --> 01:21:39.042
of dissolving. Some of them are real.
The need for data is real,

01:21:39.126 --> 01:21:42.477
maybe continual learning is a real thing.

01:21:42.560 --> 01:21:45.920
But again, I would ground
us in something like code.

01:21:46.480 --> 01:21:50.077
I think we may get to the point
in a year or two where the models can

01:21:50.160 --> 01:21:53.117
just do SWE end-to-end.
That's a whole task.

01:21:53.200 --> 01:21:56.157
That's a whole sphere
of human activity that

01:21:56.240 --> 01:22:00.800
we're just saying models can do now.
When you say end-to-end, do you mean

01:22:02.000 --> 01:22:06.557
setting technical direction, understanding
the context of the problem, et cetera?

01:22:06.640 --> 01:22:12.560
Yes. I mean all of that. Interesting.
I feel like that is AGI-complete,

01:22:13.520 --> 01:22:17.197
which maybe is internally consistent.
But it's not like saying 90%

01:22:17.280 --> 01:22:22.637
of code or 100% of code.
No, I gave this spectrum:

01:22:22.720 --> 01:22:28.557
90% of code, 100% of code, 90%
of end-to-end SWE, 100% of end-to-end SWE.

01:22:28.640 --> 01:22:31.837
New tasks are created for SWEs.
Eventually those get done as well.

01:22:31.920 --> 01:22:35.677
It's a long spectrum there, but we're
traversing the spectrum very quickly.

01:22:35.760 --> 01:22:39.510
I do think it's funny that I've seen
a couple of podcasts you've done where

01:22:40.080 --> 01:22:42.957
the hosts will be like, "But Dwarkesh
wrote the essay about the continuous

01:22:43.040 --> 01:22:43.597
learning thing."

01:22:43.680 --> 01:22:47.840
It always makes me crack up because
you've been an AI researcher for 10 years.

01:22:48.880 --> 01:22:53.517
I'm sure there's some feeling of,
"Okay, so a podcaster wrote an essay,

01:22:53.600 --> 01:22:59.037
and every interview I get asked about it."
The truth of the matter is that we're all

01:22:59.120 --> 01:23:04.717
trying to figure this out together.
There are some ways in which I'm

01:23:04.800 --> 01:23:08.877
able to see things that others aren't.
These days that probably has more to do

01:23:08.960 --> 01:23:12.477
with seeing a bunch of stuff within
Anthropic and having to make a bunch

01:23:12.560 --> 01:23:13.677
of decisions than I have

01:23:13.760 --> 01:23:20.077
any great research insight that others
don't. I'm running a 2,500 person company.

01:23:20.160 --> 01:23:25.160
It's actually pretty hard for me
to have concrete research insight,

01:23:25.244 --> 01:23:32.003
much harder than it would have been 10
years ago or even two or three years ago.

01:23:32.086 --> 01:23:35.917
As we go towards a world of a full drop-in

01:23:36.000 --> 01:23:42.077
remote worker replacement, does an API
pricing model still make the most sense?

01:23:42.160 --> 01:23:45.837
If not, what is the correct
way to price AGI, or serve AGI?

01:23:45.920 --> 01:23:49.251
I think there's going to be a bunch
of different business models here,

01:23:49.335 --> 01:23:49.837
all at once,

01:23:49.920 --> 01:23:56.920
that are going to be experimented with.
I actually do think that the API

01:23:59.680 --> 01:24:03.237
model is more durable
than many people think.

01:24:03.320 --> 01:24:06.877
One way I think about
it is if the technology

01:24:06.960 --> 01:24:11.997
is advancing quickly, if it's advancing
exponentially, what that means is there's

01:24:12.080 --> 01:24:19.080
always a surface area of new use cases
that have been developed in the last three

01:24:19.444 --> 01:24:25.838
months. Any kind of product surface you
put in place is always at risk of sort

01:24:25.921 --> 01:24:27.757
of becoming irrelevant.

01:24:27.840 --> 01:24:32.217
Any given product surface probably makes
sense for a range of capabilities of

01:24:32.301 --> 01:24:37.812
the model. The chatbot is already running
into limitations where making it smarter

01:24:37.895 --> 01:24:41.837
doesn't really help the average consumer
that much. But I don't think that's

01:24:41.920 --> 01:24:45.357
a limitation of AI models.
I don't think that's evidence

01:24:45.440 --> 01:24:50.510
that the models are good enough
and them getting better doesn't matter to

01:24:50.593 --> 01:24:54.918
the economy. It doesn't matter
to that particular product.

01:24:55.001 --> 01:24:58.637
So I think the value
of the API is that the API

01:24:58.720 --> 01:25:02.718
always offers an opportunity,
very close to the bare metal,

01:25:02.801 --> 01:25:05.360
to build on what the latest thing is.

01:25:06.720 --> 01:25:13.720
There's always going to be this front
of new startups and new ideas that

01:25:14.400 --> 01:25:18.500
weren't possible a few months ago and are
possible because the model is advancing.

01:25:19.520 --> 01:25:26.302
I actually predict that it's going
to exist alongside other models,

01:25:26.386 --> 01:25:28.557
but we're always going

01:25:28.640 --> 01:25:33.370
to have the API business model because
there's always going to be a need for

01:25:33.453 --> 01:25:34.637
a thousand different

01:25:34.720 --> 01:25:38.263
people to try experimenting
with the model in a different way.

01:25:38.347 --> 01:25:42.605
100 of them become startups and ten
of them become big successful startups.

01:25:42.688 --> 01:25:44.957
Two or three really end up being the way

01:25:45.040 --> 01:25:47.917
that people use the model
of a given generation.

01:25:48.000 --> 01:25:50.877
So I basically think it's
always going to exist.

01:25:50.960 --> 01:25:55.677
At the same time, I'm sure there's
going to be other models as well.

01:25:55.760 --> 01:26:00.317
Not every token that's output
by the model is worth the same amount.

01:26:00.400 --> 01:26:07.400
Think about what is the value of the
tokens that the model outputs when someone

01:26:10.181 --> 01:26:13.840
calls them up and says, "My Mac isn't
working," or something, the model's like,

01:26:13.924 --> 01:26:14.757
"restart it."

01:26:16.160 --> 01:26:22.240
Someone hasn't heard that before,
but the model said that 10 million times.

01:26:23.280 --> 01:26:26.330
Maybe that's worth like a dollar
or a few cents or something.

01:26:26.880 --> 01:26:33.880
Whereas if the model goes to one of the
pharmaceutical companies and it says, "Oh,

01:26:34.800 --> 01:26:36.712
you know, this molecule you're developing,

01:26:36.795 --> 01:26:39.277
you should take the aromatic
ring from that end of the

01:26:39.360 --> 01:26:43.198
molecule and put it on that end
of the molecule. If you do that,

01:26:43.282 --> 01:26:45.120
wonderful things will happen."

01:26:46.800 --> 01:26:50.240
Those tokens could be worth
tens of millions of dollars.

01:26:52.400 --> 01:26:56.797
So I think we're definitely going to see
business models that recognize that.

01:26:56.880 --> 01:27:03.158
At some point we're going to see
"pay for results" in some form,

01:27:03.241 --> 01:27:06.637
or we may see forms of compensation

01:27:06.720 --> 01:27:13.720
that are like labor, that kind
of work by the hour. I don't know.

01:27:13.977 --> 01:27:16.797
I think because it's a new

01:27:16.880 --> 01:27:19.837
industry, a lot of things are going to be
tried. I don't know what will turn out to

01:27:19.920 --> 01:27:24.397
be the right thing. I take your point that

01:27:24.480 --> 01:27:28.145
people will have to try things to figure
out what is the best way to use this blob

01:27:28.229 --> 01:27:28.877
of intelligence.

01:27:28.960 --> 01:27:34.717
But what I find striking is Claude Code.
I don't think in the history of startups

01:27:34.800 --> 01:27:40.307
there has been a single application that
has been as hotly competed in as coding

01:27:40.391 --> 01:27:41.224
agents.

01:27:42.960 --> 01:27:48.563
Claude Code is a category leader here.
That seems surprising to me.

01:27:48.647 --> 01:27:52.211
It doesn't seem intrinsically
that Anthropic had to build this.

01:27:52.295 --> 01:27:54.637
I wonder if you have
an accounting of why it had

01:27:54.720 --> 01:27:58.077
to be Anthropic or how Anthropic ended
up building an application in addition

01:27:58.160 --> 01:27:59.997
to the model underlying
it that was successful.

01:28:00.080 --> 01:28:01.917
So it actually happened
in a pretty simple way,

01:28:02.000 --> 01:28:08.640
which is that we had our own coding
models, which were good at coding.

01:28:09.440 --> 01:28:14.317
Around the beginning of 2025, I said, "I
think the time has come where you can have

01:28:14.400 --> 01:28:20.409
nontrivial acceleration of your own
research if you're an AI company by using

01:28:20.492 --> 01:28:25.437
these models." Of course, you need an
interface, you need a harness to use them.

01:28:25.520 --> 01:28:30.948
So I encouraged people internally. I
didn't say this is one thing that you have

01:28:31.032 --> 01:28:34.928
to use. I just said people
should experiment with this.

01:28:35.011 --> 01:28:37.677
I think it might have been originally

01:28:37.760 --> 01:28:41.997
called Claude CLI, and then the name
eventually got changed to Claude Code.

01:28:42.080 --> 01:28:47.777
Internally, it was the thing that everyone
was using and it was seeing fast internal

01:28:47.861 --> 01:28:48.397
adoption.

01:28:48.480 --> 01:28:52.380
I looked at it and I said, "Probably we
should launch this externally, right?"

01:28:53.200 --> 01:28:59.040
It's seen such fast adoption within
Anthropic. Coding is a lot of what we do.

01:28:59.920 --> 01:29:04.557
We have an audience of many, many hundreds
of people that's in some ways at least

01:29:04.640 --> 01:29:08.317
representative of the external audience.
So it looks like we already have product

01:29:08.400 --> 01:29:13.034
market fit. Let's launch this thing.
And then we launched it.

01:29:13.117 --> 01:29:15.277
I think just the fact that we

01:29:15.360 --> 01:29:20.180
ourselves are kind of developing the model
and we ourselves know what we most need

01:29:20.263 --> 01:29:24.382
to use the model, I think it's kind
of creating this feedback loop. I see.

01:29:24.466 --> 01:29:26.317
In the sense that you, let's say a

01:29:26.400 --> 01:29:31.181
developer at Anthropic is like, "Ah, it
would be better if it was better at this X

01:29:31.265 --> 01:29:35.037
thing." Then you bake that into
the next model that you build.

01:29:35.120 --> 01:29:39.200
That's one version of it, but then there's
just the ordinary product iteration.

01:29:41.520 --> 01:29:47.197
We have a bunch of coders within
Anthropic, they use Claude Code

01:29:47.280 --> 01:29:50.237
every day and so we get fast feedback.
That was more important in the early days.

01:29:50.320 --> 01:29:53.517
Now, of course, there are millions
of people using it, and so we get

01:29:53.600 --> 01:29:58.077
a bunch of external feedback as well.
But it's just great to be able to get

01:29:58.160 --> 01:30:03.837
kind of fast internal feedback.
I think this is the reason why we

01:30:03.920 --> 01:30:07.840
launched a coding model and didn't
launch a pharmaceutical company.

01:30:10.560 --> 01:30:14.557
My background's in biology, but we
don't have any of the resources that

01:30:14.640 --> 01:30:21.640
are needed to launch
a pharmaceutical company.

01:30:50.200 --> 01:30:57.200
Let me now ask you
about making AI go well.

01:31:24.320 --> 01:31:28.451
It seems like whatever vision we
have about how AI goes well has to be

01:31:28.534 --> 01:31:30.077
compatible with two things:

01:31:30.160 --> 01:31:37.160
1) the ability to build and run AIs
is diffusing extremely rapidly and 2) the

01:31:37.440 --> 01:31:41.135
population of AIs, the amount
we have and their intelligence,

01:31:41.218 --> 01:31:43.200
will also increase very rapidly.

01:31:44.160 --> 01:31:48.201
That means that lots of people will
be able to build huge populations

01:31:48.284 --> 01:31:49.277
of misaligned AIs,

01:31:49.360 --> 01:31:53.437
or AIs which are just companies
which are trying to increase their

01:31:53.520 --> 01:31:57.837
footprint or have weird psyches like
Sydney Bing, but now they're superhuman.

01:31:57.920 --> 01:32:02.797
What is a vision for a world in which
we have an equilibrium that is compatible

01:32:02.880 --> 01:32:06.530
with lots of different AIs, some
of which are misaligned, running around?

01:32:06.720 --> 01:32:10.242
I think in "The
Adolescence of Technology",

01:32:10.326 --> 01:32:13.597
I was skeptical of the balance of power.

01:32:13.680 --> 01:32:20.680
But the thing I was specifically skeptical
of is you have three or four of these

01:32:21.302 --> 01:32:22.160
companies

01:32:23.520 --> 01:32:30.520
all building models that are
derived from the same thing,

01:32:31.880 --> 01:32:36.637
that they would check each other.

01:32:36.720 --> 01:32:40.797
Or even that any number
of them would check each other.

01:32:40.880 --> 01:32:46.747
We might live in an offense-dominant world
where one person or one AI model is smart

01:32:46.830 --> 01:32:52.544
enough to do something that causes damage
for everything else. In the short run,

01:32:52.628 --> 01:32:56.077
we have a limited number
of players now. So we can start

01:32:56.160 --> 01:33:03.160
within the limited number of players.
We need to put in place the safeguards.

01:33:03.680 --> 01:33:05.357
We need to make sure everyone
does the right alignment work.

01:33:05.440 --> 01:33:08.693
We need to make sure
everyone has bioclassifiers.

01:33:08.776 --> 01:33:11.757
Those are the immediate
things we need to do.

01:33:11.840 --> 01:33:15.117
I agree that that doesn't solve
the problem in the long run,

01:33:15.200 --> 01:33:21.348
particularly if the ability of AI models
to make other AI models proliferates,

01:33:21.432 --> 01:33:26.080
then the whole thing can
become harder to solve.

01:33:26.720 --> 01:33:30.717
I think in the long run we need
some architecture of governance.

01:33:30.800 --> 01:33:34.957
We need some architecture of governance
that preserves human freedom,

01:33:35.040 --> 01:33:42.040
but also allows us to govern a very
large number of human systems,

01:33:45.160 --> 01:33:47.920
AI systems, hybrid

01:33:52.560 --> 01:33:59.560
human-AI companies or economic units.
So we're gonna need to think about:

01:34:01.520 --> 01:34:05.640
how do we protect the world
against bioterrorism?

01:34:05.723 --> 01:34:09.840
How do we protect the world
against mirror life?

01:34:11.200 --> 01:34:15.597
Probably we're gonna need some
kind of AI monitoring system

01:34:15.680 --> 01:34:20.557
that monitors for all of these things.
But then we need to build this in a way

01:34:20.640 --> 01:34:24.717
that preserves civil liberties
and our constitutional rights.

01:34:24.800 --> 01:34:31.800
So I think just as anything else, it's
a new security landscape with a new set of

01:34:34.640 --> 01:34:40.717
tools and a new set of vulnerabilities.
My worry is, if we had 100 years for this

01:34:40.800 --> 01:34:44.837
to happen all very slowly,
we'd get used to it.

01:34:44.920 --> 01:34:48.957
We've gotten used
to the presence of explosives

01:34:49.040 --> 01:34:56.040
in society or the presence of various new
weapons or the presence of video cameras.

01:34:58.320 --> 01:35:03.037
We would get used to it over 100 years and
we’d develop governance mechanisms. We'd

01:35:03.120 --> 01:35:07.517
make our mistakes. My worry is just
that this is happening all so fast.

01:35:07.600 --> 01:35:11.623
So maybe we need to do our thinking
faster about how to make these governance

01:35:11.707 --> 01:35:12.560
mechanisms work.

01:35:13.280 --> 01:35:15.900
It seems like in an
offense-dominant world,

01:35:15.983 --> 01:35:19.357
over the course of the next
century—the idea is that AI

01:35:19.440 --> 01:35:22.122
is making the progress that would happen
over the next century happen in some

01:35:22.206 --> 01:35:25.874
period of five to ten years—we
would still need the same mechanisms,

01:35:25.957 --> 01:35:29.197
or balance of power would
be similarly intractable,

01:35:29.280 --> 01:35:36.237
even if humans were the only game in town.
I guess we have the advice of AI.

01:35:36.320 --> 01:35:41.357
But it fundamentally doesn't seem like
a totally different ball game here.

01:35:41.440 --> 01:35:44.157
If checks and balances were going to work,
they would work with humans as well.

01:35:44.240 --> 01:35:47.437
If they aren't going to work,
they wouldn't work with AIs as well.

01:35:47.520 --> 01:35:50.480
So maybe this just dooms human
checks and balances as well.

01:35:51.440 --> 01:35:55.520
Again, I think there's some
way to make this happen.

01:35:58.240 --> 01:36:01.890
The governments of the world may have
to work together to make it happen.

01:36:02.400 --> 01:36:09.234
We may have to talk to AIs about
building societal structures in such a way

01:36:09.318 --> 01:36:10.157
that these

01:36:10.240 --> 01:36:15.302
defenses are possible. I don't know.
I don’t want to say this is so far ahead

01:36:15.385 --> 01:36:15.837
in time,

01:36:15.920 --> 01:36:20.437
but it’s so far ahead in technological
ability that may happen over a short

01:36:20.520 --> 01:36:23.642
period of time, that it's hard
for us to anticipate it in advance.

01:36:23.726 --> 01:36:25.517
Speaking of governments getting involved,

01:36:25.600 --> 01:36:31.597
on December 26, the Tennessee legislature
introduced a bill which said, "It would

01:36:31.680 --> 01:36:34.797
be an offense for a person to knowingly
train artificial intelligence to provide

01:36:34.880 --> 01:36:39.357
emotional support, including through
open-ended conversations with a user."

01:36:39.440 --> 01:36:46.440
Of course, one of the things that Claude
attempts to do is be a thoughtful,

01:36:46.815 --> 01:36:48.797
knowledgeable friend.

01:36:48.880 --> 01:36:52.077
In general, it seems like we're going
to have this patchwork of state laws.

01:36:52.160 --> 01:36:55.588
A lot of the benefits that normal people
could experience as a result of AI are

01:36:55.671 --> 01:36:56.077
going to be

01:36:56.160 --> 01:36:59.277
curtailed, especially when we get into the
kinds of things you discuss in "Machines

01:36:59.360 --> 01:37:02.797
of Loving Grace": biological freedom,
mental health improvements, et cetera.

01:37:02.880 --> 01:37:08.932
It seems easy to imagine worlds in which
these get Whac-A-Moled away by different

01:37:09.015 --> 01:37:09.917
laws, whereas

01:37:10.000 --> 01:37:14.850
bills like this don't seem to address
the actual existential threats that you're

01:37:14.933 --> 01:37:15.837
concerned about.

01:37:15.920 --> 01:37:20.237
I'm curious to understand, in the context
of things like this, Anthropic's position

01:37:20.320 --> 01:37:23.602
against the federal
moratorium on state AI laws.

01:37:23.685 --> 01:37:27.120
There are many different
things going on at once.

01:37:28.960 --> 01:37:34.557
I think that particular law is dumb.
It was clearly made by legislators

01:37:34.640 --> 01:37:38.317
who just probably had little idea
what AI models could do and not do.

01:37:38.400 --> 01:37:41.400
They're like, "AI models serving us,
that just sounds scary.

01:37:41.840 --> 01:37:46.720
I don't want that to happen."
So we're not in favor of that.

01:37:47.280 --> 01:37:50.185
But that wasn't the thing
that was being voted on.

01:37:50.269 --> 01:37:52.397
The thing that was being voted on is:

01:37:52.480 --> 01:37:59.156
we're going to ban all state regulation
of AI for 10 years with no apparent plan

01:37:59.240 --> 01:38:03.418
to do any federal regulation of AI,
which would take Congress to pass,

01:38:03.501 --> 01:38:04.960
which is a very high bar.

01:38:05.680 --> 01:38:09.710
So the idea that we'd ban states
from doing anything for 10 years…

01:38:09.793 --> 01:38:10.957
People said they had

01:38:11.040 --> 01:38:14.917
a plan for the federal government, but
there was no actual proposal on the table.

01:38:15.000 --> 01:38:15.357
There was

01:38:15.440 --> 01:38:21.222
no actual attempt. Given the serious
dangers that I lay out in "Adolescence

01:38:21.305 --> 01:38:22.317
of Technology"

01:38:22.400 --> 01:38:29.117
around things like biological weapons
and bioterrorism autonomy risk, and the

01:38:29.200 --> 01:38:34.957
timelines we've been talking about—10
years is an eternity—I think that's

01:38:35.040 --> 01:38:36.557
a crazy thing to do.

01:38:36.640 --> 01:38:42.077
So if that's the choice, if that's what
you force us to choose, then we're going

01:38:42.160 --> 01:38:47.677
to choose not to have that moratorium.
I think the benefits of that position

01:38:47.760 --> 01:38:51.357
exceed the costs, but it's not
a perfect position if that's the choice.

01:38:51.440 --> 01:38:55.451
Now, I think the thing that we should do,
the thing that I would support,

01:38:55.535 --> 01:39:00.101
is the federal government should step in,
not saying "states you can't regulate",

01:39:00.185 --> 01:39:05.438
but "Here's what we're going to do,
and states you can't differ from this."

01:39:05.521 --> 01:39:08.877
I think preemption is fine in the sense of

01:39:08.960 --> 01:39:11.740
saying that the federal government
says, "Here is our standard.

01:39:11.824 --> 01:39:12.877
This applies to everyone.

01:39:12.960 --> 01:39:16.077
States can't do something different."
That would be something I would support

01:39:16.160 --> 01:39:21.917
if it would be done in the right way.
But this idea of states, "You can't do

01:39:22.000 --> 01:39:25.799
anything and we're not
doing anything either,"

01:39:25.882 --> 01:39:29.597
that struck us as very
much not making sense.

01:39:29.680 --> 01:39:33.437
I think it will not age well, it
is already starting to not age well with

01:39:33.520 --> 01:39:38.240
all the backlash that you've seen.
Now, in terms of what we would want,

01:39:39.200 --> 01:39:44.832
the things we've talked about are starting
with transparency standards in order

01:39:44.915 --> 01:39:49.439
to monitor some of these autonomy
risks and bioterrorism risks.

01:39:49.523 --> 01:39:53.037
As the risks become more
serious, as we get more

01:39:53.120 --> 01:39:57.811
evidence for them, then I think we could
be more aggressive in some targeted ways

01:39:57.895 --> 01:40:04.317
and say, "Hey, AI bioterrorism is really
a threat. Let's pass a law that forces

01:40:04.400 --> 01:40:07.837
people to have classifiers."
I could even imagine… It depends.

01:40:07.920 --> 01:40:11.358
It depends how serious the threat it
ends up being. We don't know for sure.

01:40:11.442 --> 01:40:12.157
We need to pursue

01:40:12.240 --> 01:40:15.142
this in an intellectually honest
way where we say that ahead of time,

01:40:15.225 --> 01:40:16.397
the risk has not emerged yet.

01:40:16.480 --> 01:40:20.400
But I could certainly imagine,
with the pace that things are going at,

01:40:21.440 --> 01:40:24.093
a world where later
this year we say, "Hey,

01:40:24.176 --> 01:40:26.957
this AI bioterrorism
stuff is really serious.

01:40:27.040 --> 01:40:31.517
We should do something about it.
We should put it in a federal standard.

01:40:31.600 --> 01:40:34.817
If the federal government won't act,
we should put it in a state standard."

01:40:34.900 --> 01:40:35.917
I could totally see that.

01:40:36.000 --> 01:40:42.120
I'm concerned about a world where if you
just consider the pace of progress you're

01:40:42.203 --> 01:40:42.877
expecting,

01:40:42.960 --> 01:40:48.397
the life cycle of legislation...
The benefits are, as you say because

01:40:48.480 --> 01:40:55.480
of diffusion lag, slow enough that I
really do think this patchwork of state

01:40:55.920 --> 01:40:57.817
laws, on the current
trajectory, would prohibit.

01:40:57.901 --> 01:40:59.757
I mean if having
an emotional chatbot friend is

01:40:59.840 --> 01:41:03.288
something that freaks people out, then
just imagine the kinds of actual benefits

01:41:03.371 --> 01:41:05.942
from AI we want normal people
to be able to experience.

01:41:06.025 --> 01:41:08.237
From improvements
in health and healthspan and

01:41:08.320 --> 01:41:12.280
improvements in mental health and so
forth. Whereas at the same time,

01:41:12.364 --> 01:41:13.277
it seems like you

01:41:13.360 --> 01:41:18.116
think the dangers are already on the
horizon and I just don't see that much…

01:41:18.199 --> 01:41:19.517
It seems like it would

01:41:19.600 --> 01:41:23.997
be especially injurious to the benefits
of AI as compared to the dangers of AI.

01:41:24.080 --> 01:41:27.597
So that's maybe where the cost
benefit makes less sense to me.

01:41:27.680 --> 01:41:31.197
So there's a few things here.
People talk about there being

01:41:31.280 --> 01:41:34.317
thousands of these state laws.
First of all, the vast,

01:41:34.400 --> 01:41:40.957
vast majority of them do not pass.
The world works a certain way in theory,

01:41:41.040 --> 01:41:44.557
but just because a law has been passed
doesn't mean it's really enforced.

01:41:44.640 --> 01:41:48.797
The people implementing it may be
like, "Oh my God, this is stupid.

01:41:48.880 --> 01:41:54.800
It would mean shutting off everything
that's ever been built in Tennessee."

01:41:55.440 --> 01:42:01.625
Very often, laws are interpreted
in a way that makes them not as dangerous

01:42:01.708 --> 01:42:02.557
or harmful.

01:42:02.640 --> 01:42:06.023
On the same side, of course, you have
to worry if you're passing a law to stop

01:42:06.107 --> 01:42:06.557
a bad thing;

01:42:06.640 --> 01:42:13.640
you have this problem as well.
My basic view is that if we could

01:42:16.400 --> 01:42:21.128
decide what laws were passed and how
things were done—and we’re only one small

01:42:21.212 --> 01:42:21.437
input

01:42:21.520 --> 01:42:28.160
into that—I would deregulate a lot of the
stuff around the health benefits of AI.

01:42:29.840 --> 01:42:33.477
I don't worry as much
about the chatbot laws.

01:42:33.560 --> 01:42:37.197
I actually worry more
about the drug approval

01:42:37.280 --> 01:42:44.280
process, where I think AI models are going
to greatly accelerate the rate at which we

01:42:44.372 --> 01:42:47.843
discover drugs,
and the pipeline will get jammed up.

01:42:47.926 --> 01:42:50.797
The pipeline will not
be prepared to process

01:42:50.880 --> 01:42:57.880
all the stuff that's going through it.
I think reform of the regulatory process

01:42:58.720 --> 01:43:02.007
should bias more towards the fact
that we have a lot of things coming where

01:43:02.091 --> 01:43:02.637
the safety and

01:43:02.720 --> 01:43:09.262
efficacy is actually going to be really
crisp and clear, a beautiful thing,

01:43:09.345 --> 01:43:11.200
and really effective.

01:43:12.400 --> 01:43:18.808
Maybe we don't need all this
superstructure around it that was designed

01:43:18.891 --> 01:43:21.277
around an era of drugs that

01:43:21.360 --> 01:43:25.267
barely work and often have serious
side effects. At the same time,

01:43:25.351 --> 01:43:26.477
I think we should be

01:43:26.560 --> 01:43:33.560
ramping up quite significantly
the safety and security legislation.

01:43:35.520 --> 01:43:41.939
Like I've said, starting with transparency
is my view of trying not to hamper

01:43:42.022 --> 01:43:43.037
the industry,

01:43:43.120 --> 01:43:46.717
trying to find the right balance. I'm
worried about it. Some people criticize

01:43:46.800 --> 01:43:50.877
my essay for saying, "That's too slow.
The dangers of AI will come too soon

01:43:50.960 --> 01:43:52.717
if we do that." Well, basically,

01:43:52.800 --> 01:43:57.663
I think the last six months and maybe
the next few months are going to be about

01:43:57.746 --> 01:44:02.797
transparency. Then, if these risks emerge
when we're more certain of them—which

01:44:02.880 --> 01:44:07.837
I think we might be as soon as later this
year—then I think we need to act very fast

01:44:07.920 --> 01:44:10.437
in the areas where we've
actually seen the risk.

01:44:10.520 --> 01:44:13.037
I think the only way
to do this is to be nimble.

01:44:13.120 --> 01:44:20.120
Now, the legislative process is normally
not nimble, but we need to emphasize the

01:44:21.280 --> 01:44:23.769
urgency of this to everyone involved.
That's why I'm sending this message

01:44:23.852 --> 01:44:27.552
of urgency. That's why I wrote
Adolescence of Technology.

01:44:27.635 --> 01:44:30.797
I wanted policymakers,
economists, national

01:44:30.880 --> 01:44:35.214
security professionals, and
decision-makers to read it so that they

01:44:35.298 --> 01:44:36.797
have some hope of acting

01:44:36.880 --> 01:44:42.877
faster than they would have otherwise.
Is there anything you can do or advocate

01:44:42.960 --> 01:44:49.960
that would make it more certain that the
benefits of AI are better instantiated?

01:44:51.040 --> 01:44:54.157
I feel like you have worked
with legislatures to say, "Okay,

01:44:54.240 --> 01:44:56.865
we're going to prevent bioterrorism here.
We're going to increase transparency,

01:44:56.949 --> 01:45:00.413
we're going to increase whistleblower
protection." But I think by default,

01:45:00.497 --> 01:45:01.357
the actual benefits

01:45:01.440 --> 01:45:08.397
we're looking forward to seem very fragile
to different kinds of moral panics or

01:45:08.480 --> 01:45:12.877
political economy problems.
I don't actually agree that

01:45:12.960 --> 01:45:16.957
much regarding the developed world.
I feel like in the developed world,

01:45:17.040 --> 01:45:23.437
markets function pretty well.
When there's a lot of money to

01:45:23.520 --> 01:45:26.591
be made on something and it's clearly
the best available alternative,

01:45:26.674 --> 01:45:27.597
it's actually hard for

01:45:27.680 --> 01:45:32.080
the regulatory system to stop it.
We're seeing that in AI itself.

01:45:33.600 --> 01:45:37.440
A thing I've been trying to fight
for is export controls on chips to China.

01:45:38.640 --> 01:45:41.840
That's in the national
security interest of the US.

01:45:42.720 --> 01:45:49.618
That's squarely within the policy beliefs
of almost everyone in Congress of both

01:45:49.702 --> 01:45:50.535
parties.

01:45:52.320 --> 01:45:55.831
The case is very clear.
The counterarguments against it,

01:45:55.914 --> 01:45:57.840
I'll politely call them fishy.

01:45:59.280 --> 01:46:05.150
Yet it doesn't happen and we sell the
chips because there's so much money riding

01:46:05.233 --> 01:46:06.067
on it.

01:46:08.080 --> 01:46:12.160
That money wants to be made. In that case,
in my opinion, that's a bad thing.

01:46:13.120 --> 01:46:17.776
But it also applies
when it's a good thing.

01:46:17.860 --> 01:46:22.957
So if we're talking about
drugs and benefits of

01:46:23.040 --> 01:46:28.814
the technology, I am not as worried
about those benefits being hampered in

01:46:28.897 --> 01:46:34.914
the developed world. I am a little worried
about them going too slow. As I said,

01:46:34.997 --> 01:46:37.517
I do think we should work to speed

01:46:37.600 --> 01:46:41.117
the approval process in the FDA.
I do think we should fight against

01:46:41.200 --> 01:46:44.919
these chatbot bills that you're
describing. Described individually,

01:46:45.002 --> 01:46:45.997
I'm against them. I

01:46:46.080 --> 01:46:51.256
think they're stupid. But I actually think
the bigger worry is the developing world,

01:46:51.339 --> 01:46:51.757
where we

01:46:51.840 --> 01:46:57.068
don't have functioning markets and where
we often can't build on the technology

01:46:57.151 --> 01:47:01.277
that we've had. I worry more that
those folks will get left behind.

01:47:01.360 --> 01:47:04.877
And I worry that even if the cures are
developed, maybe there's someone in rural

01:47:04.960 --> 01:47:10.477
Mississippi who doesn't get it as well.
That's a smaller version of the concern

01:47:10.560 --> 01:47:14.157
we have in the developing world.
So the things we've been doing

01:47:14.240 --> 01:47:21.240
are working with philanthropists. We
work with folks who deliver medicine and

01:47:26.560 --> 01:47:32.881
health interventions to the developing
world, to sub-Saharan Africa, India,

01:47:32.964 --> 01:47:34.077
Latin America,

01:47:34.160 --> 01:47:39.197
and other developing parts of the world.
That's the thing I think that

01:47:39.280 --> 01:47:42.030
won't happen on its own.
You mentioned export controls.

01:47:42.960 --> 01:47:47.440
Why shouldn't the US and China both have
a "country of geniuses in a data center"?

01:47:48.640 --> 01:47:52.560
Why won’t it happen or why shouldn't
it happen? Why shouldn't it happen.

01:47:54.720 --> 01:48:01.720
If this does happen,
we could have a few situations.

01:48:02.400 --> 01:48:05.277
If we have an offense-dominant
situation, we could have a situation

01:48:05.360 --> 01:48:12.241
like nuclear weapons, but more dangerous.
Either side could easily destroy

01:48:12.325 --> 01:48:13.360
everything.

01:48:14.000 --> 01:48:19.037
We could also have a world where it's
unstable. The nuclear equilibrium is

01:48:19.120 --> 01:48:24.077
stable because it's deterrence.
But let's say there was uncertainty about,

01:48:24.160 --> 01:48:29.178
if the two AIs fought, which AI would win?
That could create instability.

01:48:29.262 --> 01:48:30.157
You often have

01:48:30.240 --> 01:48:33.715
conflict when the two sides have
a different assessment of their likelihood

01:48:33.798 --> 01:48:36.802
of winning. If one side is like,
"Oh yeah, there's a 90% chance I'll win,"

01:48:36.886 --> 01:48:38.136
and the other side thinks

01:48:40.160 --> 01:48:43.437
the same, then a fight is much more
likely. They can't both be right,

01:48:43.520 --> 01:48:46.797
but they can both think that. But this
seems like a fully general argument

01:48:46.880 --> 01:48:52.080
against the diffusion of AI technology.
That's the implication of this world.

01:48:52.800 --> 01:48:55.837
Let me just go on, because I think
we will get diffusion eventually.

01:48:55.920 --> 01:49:00.024
The other concern I have is that
governments will oppress their own people

01:49:00.107 --> 01:49:00.941
with AI.

01:49:04.160 --> 01:49:11.034
I'm worried about a world where you
have a country in which there’s already

01:49:11.117 --> 01:49:12.880
a government that's

01:49:16.240 --> 01:49:21.597
building a high-tech authoritarian state.
To be clear, this is about the government.

01:49:21.680 --> 01:49:24.317
This is not about the people.
We need to find a way for

01:49:24.400 --> 01:49:27.760
people everywhere to benefit.
My worry here is about governments.

01:49:30.080 --> 01:49:33.837
My worry is if the world gets carved up
into two pieces, one of those two pieces

01:49:33.920 --> 01:49:38.880
could be authoritarian or totalitarian in
a way that's very difficult to displace.

01:49:39.440 --> 01:49:42.660
Now, will governments
eventually get powerful AI,

01:49:42.743 --> 01:49:45.357
and is there a risk of authoritarianism?

01:49:45.440 --> 01:49:49.040
Yes. Will governments eventually get
powerful AI, and is there a risk of

01:49:52.080 --> 01:49:58.520
bad equilibria? Yes, I think both things.
But the initial conditions matter.

01:49:58.603 --> 01:50:02.690
At some point, we're going to need
to set up the rules of the road.

01:50:02.773 --> 01:50:05.357
I'm not saying that one
country, either the United

01:50:05.440 --> 01:50:09.124
States or a coalition of democracies—which
I think would be a better setup,

01:50:09.207 --> 01:50:09.677
although it

01:50:09.760 --> 01:50:16.706
requires more international cooperation
than we currently seem to want

01:50:16.790 --> 01:50:19.117
to make—should just say,

01:50:19.200 --> 01:50:22.397
"These are the rules of the road."
There's going to be some negotiation.

01:50:22.480 --> 01:50:26.589
The world is going to have
to grapple with this.

01:50:26.673 --> 01:50:30.957
What I would like is for
the democratic nations of

01:50:31.040 --> 01:50:38.040
the world—those whose governments
represent closer to pro-human values—are

01:50:38.214 --> 01:50:39.197
holding the

01:50:39.280 --> 01:50:44.077
stronger hand and have more leverage
when the rules of the road are set.

01:50:44.160 --> 01:50:47.037
So I'm very concerned about
that initial condition.

01:50:47.120 --> 01:50:51.677
I was re-listening to the interview from
three years ago, and one of the ways it

01:50:51.760 --> 01:50:55.431
aged poorly is that I kept asking
questions assuming there was going to be

01:50:55.514 --> 01:50:59.564
some key fulcrum moment two to three years
from now. In fact, being that far out,

01:50:59.647 --> 01:51:03.537
it just seems like progress continues,
AI improves, AI is more diffused,

01:51:03.621 --> 01:51:05.757
and people will use it for more things.

01:51:05.840 --> 01:51:09.172
It seems like you're imagining a world
in the future where the countries get

01:51:09.256 --> 01:51:12.142
together, and "Here's the rules
of the road, here's the leverage we have,

01:51:12.225 --> 01:51:13.517
and here's the leverage you have."

01:51:13.600 --> 01:51:17.440
But on the current trajectory,
everybody will have more AI.

01:51:18.080 --> 01:51:19.917
Some of that AI will be used
by authoritarian countries.

01:51:20.000 --> 01:51:22.317
Some of that within the authoritarian
countries will be used by private

01:51:22.400 --> 01:51:26.317
actors versus state actors.
It's not clear who will benefit more.

01:51:26.400 --> 01:51:28.468
It's always unpredictable
to tell in advance.

01:51:28.551 --> 01:51:30.237
It seems like the internet privileged

01:51:30.320 --> 01:51:33.120
authoritarian countries more
than you would've expected.

01:51:33.280 --> 01:51:38.717
Maybe AI will be the opposite way around.
I want to better understand what

01:51:38.800 --> 01:51:42.557
you're imagining here.
Just to be precise about it,

01:51:42.640 --> 01:51:47.197
I think the exponential of the underlying
technology will continue as it has before.

01:51:47.280 --> 01:51:51.442
The models get smarter and smarter, even
when they get to a "country of geniuses

01:51:51.525 --> 01:51:52.480
in a data center."

01:51:53.760 --> 01:51:55.997
I think you can continue
to make the model smarter.

01:51:56.080 --> 01:52:01.757
There's a question of getting diminishing
returns on their value in the world.

01:52:01.840 --> 01:52:05.840
How much does it matter after
you've already solved human biology?

01:52:07.520 --> 01:52:10.884
At some point you can do harder,
more abstruse math problems,

01:52:10.968 --> 01:52:12.637
but nothing after that matters.

01:52:12.720 --> 01:52:16.836
Putting that aside, I do think
the exponential will continue,

01:52:16.919 --> 01:52:18.557
but there will be certain

01:52:18.640 --> 01:52:24.397
distinguished points on the exponential.
Companies, individuals, and countries

01:52:24.480 --> 01:52:27.761
will reach those points
at different times.

01:52:27.844 --> 01:52:31.517
In "The Adolescence
of Technology" I talk about:

01:52:31.600 --> 01:52:38.157
Is a nuclear deterrent still
stable in the world of AI?

01:52:38.240 --> 01:52:42.477
I don't know, but that's an example
of one thing we've taken for granted.

01:52:42.560 --> 01:52:47.680
The technology could reach such a level
that we can no longer be certain of it.

01:52:50.160 --> 01:52:55.537
Think of others. There are points
where if you reach a certain level,

01:52:55.621 --> 01:52:57.437
maybe you have offensive

01:52:57.520 --> 01:53:03.765
cyber dominance, and every computer system
is transparent to you after that unless

01:53:03.848 --> 01:53:09.277
the other side has an equivalent defense.
I don't know what the critical moment

01:53:09.360 --> 01:53:11.648
is or if there's a single critical moment.

01:53:11.732 --> 01:53:14.077
But I think there will
be either a critical

01:53:14.160 --> 01:53:18.636
moment, a small number
of critical moments,

01:53:18.719 --> 01:53:22.877
or some critical window where AI confers

01:53:22.960 --> 01:53:29.960
some large advantage from the perspective
of national security, and one country or

01:53:30.320 --> 01:53:36.557
coalition has reached it before others.
I'm not advocating that they just say,

01:53:36.640 --> 01:53:40.880
"Okay, we're in charge now."
That's not how I think about it.

01:53:42.560 --> 01:53:45.357
The other side is always catching up.
There are extreme actions you're not

01:53:45.440 --> 01:53:52.440
willing to take, and it's not right
to take complete control anyway.

01:53:52.560 --> 01:53:57.495
But at the point that happens, people
are going to understand that the world has

01:53:57.578 --> 01:53:57.997
changed.

01:53:58.080 --> 01:54:05.080
There's going to be some negotiation,
implicit or explicit, about what the

01:54:05.520 --> 01:54:12.520
post-AI world order looks like.
My interest is in making that

01:54:14.080 --> 01:54:21.080
negotiation be one in which classical
liberal democracy has a strong hand.

01:54:24.080 --> 01:54:27.357
I want to understand what that better
means, because you say in the essay,

01:54:27.440 --> 01:54:31.866
"Autocracy is simply not a form
of government that people can accept in

01:54:31.949 --> 01:54:33.517
the post-powerful AI age."

01:54:33.600 --> 01:54:39.543
That sounds like you're saying the CCP as
an institution cannot exist after we get

01:54:39.626 --> 01:54:40.459
AGI.

01:54:41.040 --> 01:54:46.664
That seems like a very strong demand,
and it seems to imply a world where

01:54:46.747 --> 01:54:47.837
the leading lab

01:54:47.920 --> 01:54:54.157
or the leading country will be able to—and
by that language, should get to—determine

01:54:54.240 --> 01:55:00.000
how the world is governed or what kinds
of governments are, and are not, allowed.

01:55:02.400 --> 01:55:05.958
I believe that paragraph
said something like,

01:55:06.041 --> 01:55:09.520
"You could take it
even further and say X."

01:55:13.360 --> 01:55:16.110
I wasn't necessarily endorsing
that view. I was saying,

01:55:18.080 --> 01:55:23.939
"Here's a weaker thing that I believe. We
have to worry a lot about authoritarians

01:55:24.023 --> 01:55:27.300
and we should try to check
them and limit their power.

01:55:27.383 --> 01:55:30.317
You could take this much
further and have a more

01:55:30.400 --> 01:55:36.805
interventionist view that says
authoritarian countries with AI are these

01:55:36.888 --> 01:55:38.157
self-fulfilling

01:55:38.240 --> 01:55:42.930
cycles that are very hard to displace,
so you just need to get rid of them from

01:55:43.014 --> 01:55:43.837
the beginning."

01:55:43.920 --> 01:55:49.037
That has exactly all the problems you say.
If you were to make a commitment to

01:55:49.120 --> 01:55:53.357
overthrowing every authoritarian country,
they would take a bunch of actions now

01:55:53.440 --> 01:56:00.440
that could lead to instability.
That just may not be possible.

01:56:02.240 --> 01:56:08.435
But the point I was making that I do
endorse is that it is quite possible

01:56:08.518 --> 01:56:14.971
that... Today, the view, my view, in most
of the Western world is that democracy is

01:56:15.054 --> 01:56:16.237
a better form of

01:56:16.320 --> 01:56:20.957
government than authoritarianism.
But if a country’s authoritarian,

01:56:21.040 --> 01:56:25.280
we don’t react the way we’d react if
they committed a genocide or something.

01:56:27.360 --> 01:56:31.290
I guess what I'm saying is I'm
a little worried that in the age of AGI,

01:56:31.373 --> 01:56:35.073
authoritarianism will have a different
meaning. It will be a graver thing.

01:56:35.920 --> 01:56:39.837
We have to decide one way
or another how to deal with that.

01:56:39.920 --> 01:56:45.757
The interventionist view is one possible
view. I was exploring such views.

01:56:45.840 --> 01:56:47.597
It may end up being the

01:56:47.680 --> 01:56:53.014
right view, or it may end up being
too extreme. But I do have hope.

01:56:53.097 --> 01:56:55.197
One piece of hope I have is

01:56:55.280 --> 01:57:00.797
that we have seen that as new
technologies are invented,

01:57:00.880 --> 01:57:04.397
forms of government become obsolete.

01:57:04.480 --> 01:57:10.557
I mentioned this in "Adolescence
of Technology", where I said feudalism

01:57:10.640 --> 01:57:17.282
was basically a form of government,
and when we invented industrialization,

01:57:17.365 --> 01:57:21.831
feudalism was no longer sustainable.
It no longer made sense. Why is that hope?

01:57:21.914 --> 01:57:26.240
Couldn't that imply that democracy is no
longer going to be a competitive system?

01:57:26.880 --> 01:57:33.880
Right, it could go either way.
But these problems with

01:57:38.160 --> 01:57:44.237
authoritarianism get deeper.
I wonder if that's an indicator of

01:57:44.320 --> 01:57:49.648
other problems that authoritarianism
will have. In other words,

01:57:49.731 --> 01:57:52.397
because authoritarianism becomes

01:57:52.480 --> 01:57:58.160
worse, people are more afraid of it.
They work harder to stop it.

01:57:59.440 --> 01:58:03.538
You have to think in terms
of total equilibrium.

01:58:03.621 --> 01:58:07.197
I just wonder if it will motivate new ways

01:58:07.280 --> 01:58:13.597
of thinking about how to preserve and
protect freedom with the new technology.

01:58:13.680 --> 01:58:20.680
Even more optimistically, will it lead to
a collective reckoning and a more emphatic

01:58:22.240 --> 01:58:26.720
realization of how important some of the
things we take as individual rights are?

01:58:27.360 --> 01:58:32.637
A more emphatic realization that
we really can't give these away.

01:58:32.720 --> 01:58:36.160
We've seen there's no other
way to live that actually works.

01:58:39.520 --> 01:58:46.477
I am actually hopeful that—it sounds too
idealistic, but I believe it could be the

01:58:46.560 --> 01:58:49.833
case—dictatorships
become morally obsolete.

01:58:49.916 --> 01:58:52.877
They become morally unworkable forms of

01:58:52.960 --> 01:58:59.960
government and the crisis that that
creates is sufficient to force us to find

01:59:00.850 --> 01:59:02.080
another way.

01:59:03.680 --> 01:59:06.705
I think there is genuinely a tough
question here which I'm not sure how you

01:59:06.788 --> 01:59:07.622
resolve.

01:59:07.920 --> 01:59:11.070
We've had to come out one way
or another on it through history.

01:59:11.760 --> 01:59:15.277
With China in the '70s and '80s,
we decided that even though it's an

01:59:15.360 --> 01:59:17.057
authoritarian system,
we will engage with it.

01:59:17.140 --> 01:59:18.877
I think in retrospect
that was the right call,

01:59:18.960 --> 01:59:22.566
because it’s a state authoritarian
system but a billion-plus people are much

01:59:22.649 --> 01:59:25.197
wealthier and better off than
they would've otherwise been.

01:59:25.280 --> 01:59:27.197
It's not clear that it
would've stopped being

01:59:27.280 --> 01:59:30.397
an authoritarian country otherwise.
You can just look at North Korea

01:59:30.480 --> 01:59:33.760
as an example of that.
I don't know if it takes

01:59:34.400 --> 01:59:39.019
that much intelligence to remain
an authoritarian country that continues

01:59:39.102 --> 01:59:40.717
to coalesce its own power.

01:59:40.800 --> 01:59:44.477
You can imagine a North Korea with an AI
that's much worse than everybody else's,

01:59:44.560 --> 01:59:49.917
but still enough to keep power.
In general, it seems like we should just

01:59:50.000 --> 01:59:54.637
have this attitude that the benefits of
AI—in the form of all these empowerments

01:59:54.720 --> 02:00:00.237
of humanity and health—will be big.
Historically, we have decided it's good

02:00:00.320 --> 02:00:02.571
to spread the benefits
of technology widely,

02:00:02.654 --> 02:00:05.360
even to people whose
governments are authoritarian.

02:00:06.000 --> 02:00:08.557
It is a tough question,
how to think about it with AI,

02:00:08.640 --> 02:00:10.317
but historically we have said, "yes,

02:00:10.400 --> 02:00:14.480
this is a positive-sum world, and it's
still worth diffusing the technology."

02:00:15.280 --> 02:00:20.797
There are a number of choices we have.
Framing this as a government-to-government

02:00:20.880 --> 02:00:27.757
decision in national security terms is one
lens, but there are a lot of other lenses.

02:00:27.840 --> 02:00:31.520
You could imagine a world where we
produce all these cures to diseases.

02:00:32.400 --> 02:00:35.892
The cures are fine to sell
to authoritarian countries,

02:00:35.975 --> 02:00:38.077
but the data centers just aren't.

02:00:38.160 --> 02:00:44.317
The chips and the data centers aren't,
and the AI industry itself isn't.

02:00:44.400 --> 02:00:48.320
Another possibility I think
folks should think about is this.

02:00:49.360 --> 02:00:54.164
Could there be developments we can
make—either that naturally happen as

02:00:54.247 --> 02:00:59.757
a result of AI, or that we could make
happen by building technology on AI—that

02:00:59.840 --> 02:01:05.277
create an equilibrium where it becomes
infeasible for authoritarian countries

02:01:05.360 --> 02:01:09.600
to deny their people private use
of the benefits of the technology?

02:01:12.400 --> 02:01:16.664
Are there equilibria where we can give
everyone in an authoritarian country

02:01:16.747 --> 02:01:18.080
their own AI model that

02:01:19.600 --> 02:01:24.327
defends them from surveillance and there
isn't a way for the authoritarian country

02:01:24.411 --> 02:01:27.328
to crack down on this while
retaining power? I don't know.

02:01:27.412 --> 02:01:29.757
That sounds to me like
if that went far enough,

02:01:29.840 --> 02:01:34.268
it would be a reason why authoritarian
countries would disintegrate from

02:01:34.351 --> 02:01:38.125
the inside. But maybe there's a middle
world where there's an equilibrium where,

02:01:38.208 --> 02:01:39.197
if they want to hold on

02:01:39.280 --> 02:01:45.357
to power, the authoritarians can't deny
individualized access to the technology.

02:01:45.440 --> 02:01:49.680
But I actually do have a hope
for the more radical version.

02:01:50.880 --> 02:01:54.797
Is it possible that the technology
might inherently have properties—or

02:01:54.880 --> 02:02:01.117
that by building on it in certain ways
we could create properties—that have this

02:02:01.200 --> 02:02:04.640
dissolving effect
on authoritarian structures? Now,

02:02:04.724 --> 02:02:07.197
we hoped originally—think back to the

02:02:07.280 --> 02:02:13.117
beginning of the Obama administration—that
social media and the internet would have

02:02:13.200 --> 02:02:19.997
that property, and it turns out not to.
But what if we could try again with the

02:02:20.080 --> 02:02:21.882
knowledge of how many
things could go wrong,

02:02:21.966 --> 02:02:23.597
and that this is a different technology?

02:02:23.680 --> 02:02:26.280
I don't know if it would work,
but it's worth a try.

02:02:26.480 --> 02:02:30.077
It's just very unpredictable.
There are first principles reasons why

02:02:30.160 --> 02:02:35.197
authoritarianism might be privileged.
It's all very unpredictable. We just

02:02:35.280 --> 02:02:40.797
have to recognize the problem and come
up with 10 things we can try, try those,

02:02:40.880 --> 02:02:43.484
and then assess which
ones are working, if any.

02:02:43.568 --> 02:02:46.287
Then try new ones if
the old ones aren't working.

02:02:46.370 --> 02:02:51.357
But I guess that nets out to today, as you
say, that we will not sell data centers,

02:02:51.440 --> 02:02:56.006
or chips, and the ability to make
chips to China. So in some sense,

02:02:56.089 --> 02:02:58.157
you are denying… There would be

02:02:58.240 --> 02:03:01.206
some benefits to the Chinese economy,
Chinese people, et cetera,

02:03:01.289 --> 02:03:02.397
because we're doing that.

02:03:02.480 --> 02:03:05.128
Then there'd also be benefits
to the American economy because it's

02:03:05.211 --> 02:03:05.997
a positive-sum world.

02:03:06.080 --> 02:03:08.637
We could trade. They could have their
country's data centers doing one thing.

02:03:08.720 --> 02:03:14.717
We could have ours doing another.
Already, you're saying it's not worth that

02:03:14.800 --> 02:03:18.894
positive-sum stipend
to empower those countries?

02:03:18.978 --> 02:03:22.637
What I would say
is that we are about to be

02:03:22.720 --> 02:03:27.757
in a world where growth and economic
value will come very easily if we're

02:03:27.840 --> 02:03:34.840
able to build these powerful AI models.
What will not come easily is distribution

02:03:35.040 --> 02:03:39.680
of benefits, distribution
of wealth, political freedom.

02:03:40.720 --> 02:03:43.117
These are the things that are
going to be hard to achieve.

02:03:43.200 --> 02:03:49.695
So when I think about policy, I think that
the technology and the market will deliver

02:03:49.778 --> 02:03:52.648
all the fundamental benefits,
this is my fundamental belief,

02:03:52.731 --> 02:03:54.531
almost faster than we can take them.

02:03:55.840 --> 02:04:01.438
These questions about distribution and
political freedom and rights are the ones

02:04:01.522 --> 02:04:02.717
that will actually

02:04:02.800 --> 02:04:06.317
matter and that policy should focus on.
Speaking of distribution, as you were

02:04:06.400 --> 02:04:12.797
mentioning, we have developing countries.
In many cases, catch-up growth has been

02:04:12.880 --> 02:04:15.357
weaker than we would have hoped for.
But when catch-up growth does happen,

02:04:15.440 --> 02:04:18.397
it's fundamentally because
they have underutilized labor.

02:04:18.480 --> 02:04:21.197
We can bring the capital and know-how from
developed countries to these countries,

02:04:21.280 --> 02:04:26.557
and then they can grow quite rapidly.
Obviously, in a world where labor is no

02:04:26.640 --> 02:04:30.077
longer the constraining factor,
this mechanism no longer works.

02:04:30.160 --> 02:04:33.517
So is the hope basically
to rely on philanthropy from

02:04:33.600 --> 02:04:37.840
the people or countries who immediately
get wealthy from AI? What is the hope?

02:04:38.880 --> 02:04:44.317
Philanthropy should obviously play
some role, as it has in the past.

02:04:44.400 --> 02:04:49.920
But I think growth is always better
and stronger if we can make it endogenous.

02:04:50.640 --> 02:04:54.720
What are the relevant
industries in an AI-driven world?

02:04:58.080 --> 02:05:00.797
I said we shouldn't build data centers in
China, but there's no reason we shouldn't

02:05:00.880 --> 02:05:03.997
build data centers in Africa.
In fact, I think it'd be

02:05:04.080 --> 02:05:07.930
great to build data centers in Africa.
As long as they're not owned by China,

02:05:08.960 --> 02:05:13.600
we should build data centers in Africa.
I think that's a great thing to do.

02:05:16.320 --> 02:05:21.360
There's no reason we can't build a
pharmaceutical industry that's AI-driven.

02:05:22.960 --> 02:05:28.797
If AI is accelerating drug discovery, then
there will be a bunch of biotech startups.

02:05:28.880 --> 02:05:31.197
Let's make sure some of those
happen in the developing world.

02:05:31.280 --> 02:05:34.877
Certainly, during the transition—we can
talk about the point where humans have no

02:05:34.960 --> 02:05:39.151
role—humans will still have some
role in starting up these companies

02:05:39.234 --> 02:05:41.037
and supervising the AI models.

02:05:41.120 --> 02:05:44.077
So let's make sure some of those
humans are in the developing world

02:05:44.160 --> 02:05:46.051
so that fast growth can
happen there as well.

02:05:46.134 --> 02:05:48.157
You guys recently announced
that Claude is going

02:05:48.240 --> 02:05:50.995
to have a constitution that's
aligned to a set of values,

02:05:51.078 --> 02:05:53.037
and not necessarily just to the end user.

02:05:53.120 --> 02:05:55.997
There's a world I can imagine where
if it is aligned to the end user,

02:05:56.080 --> 02:05:59.040
it preserves the balance of power we have
in the world today because everybody gets

02:05:59.123 --> 02:06:03.597
to have their own AI that's advocating
for them. The ratio of bad actors to

02:06:03.680 --> 02:06:07.130
good actors stays constant.
It seems to work out for our world today.

02:06:07.360 --> 02:06:12.797
Why is it better not to do that, but to
have a specific set of values that the

02:06:12.880 --> 02:06:16.637
AI should carry forward?
I'm not sure I'd quite

02:06:16.720 --> 02:06:21.600
draw the distinction in that way. There
may be two relevant distinctions here.

02:06:22.400 --> 02:06:25.495
I think you're talking about
a mix of the two. One is,

02:06:25.578 --> 02:06:27.437
should we give the model a set of

02:06:27.520 --> 02:06:31.437
instructions about "do
this" versus "don't do this"?

02:06:31.520 --> 02:06:37.440
The other is, should we give the model
a set of principles for how to act?

02:06:44.080 --> 02:06:48.797
It's kind of purely a practical
and empirical thing that we've observed.

02:06:48.880 --> 02:06:52.877
By teaching the model principles,
getting it to learn from principles,

02:06:52.960 --> 02:06:56.807
its behavior is more consistent,
it's easier to cover edge cases,

02:06:56.890 --> 02:06:58.077
and the model is more

02:06:58.160 --> 02:07:05.160
likely to do what people want it to do.
In other words, if you give it a list of

02:07:09.600 --> 02:07:10.717
rules—"don't tell people how to hot-wire
a car, don't speak in Korean"—it doesn't

02:07:10.800 --> 02:07:14.320
really understand the rules,
and it's hard to generalize from them.

02:07:15.440 --> 02:07:20.720
It’s just a list of do’s and don’t’s.
Whereas if you give it principles—it

02:07:21.440 --> 02:07:25.277
has some hard guardrails like "Don't make
biological weapons" but—overall you're

02:07:25.360 --> 02:07:28.719
trying to understand what
it should be aiming to do,

02:07:28.803 --> 02:07:31.037
how it should be aiming to operate.

02:07:31.120 --> 02:07:34.906
So just from a practical perspective,
that turns out to be a more effective way

02:07:34.989 --> 02:07:38.895
to train the model. That's the rules
versus principles trade-off.

02:07:38.978 --> 02:07:42.157
Then there's another
thing you're talking about,

02:07:42.240 --> 02:07:49.240
which is the corrigibility versus
intrinsic motivation trade-off.

02:07:51.680 --> 02:07:58.680
How much should the model be a kind
of "skin suit" where it just directly

02:08:02.000 --> 02:08:05.850
follows the instructions given to it
by whoever is giving those instructions,

02:08:06.560 --> 02:08:11.124
versus how much should the model have an
inherent set of values and go off and do

02:08:11.207 --> 02:08:12.240
things on its own?

02:08:14.320 --> 02:08:20.103
There I would actually say everything
about the model is closer to the direction

02:08:20.187 --> 02:08:21.020
that

02:08:21.760 --> 02:08:24.477
it should mostly do what people want.
It should mostly follow instructions.

02:08:24.560 --> 02:08:29.277
We're not trying to build something that
goes off and runs the world on its own.

02:08:29.360 --> 02:08:32.238
We're actually pretty far
on the corrigible side. Now,

02:08:32.321 --> 02:08:34.157
what we do say is there are certain

02:08:34.240 --> 02:08:40.477
things that the model won't do.
I think we say it in various ways in the

02:08:40.560 --> 02:08:42.692
constitution, that under
normal circumstances,

02:08:42.776 --> 02:08:45.197
if someone asks the model
to do a task, it should do

02:08:45.280 --> 02:08:50.330
that task. That should be the default.
But if you've asked it to do something

02:08:50.413 --> 02:08:51.280
dangerous, or

02:08:54.320 --> 02:09:01.197
to harm someone else,
then the model is unwilling to do that.

02:09:01.280 --> 02:09:07.597
So I actually think of it as a mostly
corrigible model that has some limits,

02:09:07.680 --> 02:09:11.997
but those limits are based on principles.
Then the fundamental question is,

02:09:12.080 --> 02:09:15.517
how are those principles determined? This
is not a special question for Anthropic.

02:09:15.600 --> 02:09:18.724
This would be a question
for any AI company.

02:09:18.807 --> 02:09:22.077
But because you have
been the ones to actually

02:09:22.160 --> 02:09:25.277
write down the principles,
I get to ask you this question.

02:09:25.360 --> 02:09:29.437
Normally, a constitution is written down,
set in stone, and there's a process of

02:09:29.520 --> 02:09:34.477
updating it and changing it and so forth.
In this case, it seems like a document

02:09:34.560 --> 02:09:37.597
that people at Anthropic write,
that can be changed at any time,

02:09:37.680 --> 02:09:42.909
that guides the behavior of systems
that are going to be the basis of a lot

02:09:42.992 --> 02:09:44.480
of economic activity.

02:09:45.680 --> 02:09:49.600
How do you think about how
those principles should be set?

02:09:50.880 --> 02:09:57.880
I think there are maybe three sizes
of loop here, three ways to iterate.

02:09:58.160 --> 02:10:01.837
One is we iterate within Anthropic. We
train the model, we're not happy with it,

02:10:01.920 --> 02:10:05.600
and we change the constitution.
I think that's good to do.

02:10:06.480 --> 02:10:10.527
Putting out public updates to the
constitution every once in a while

02:10:10.610 --> 02:10:15.940
is good because people can comment on it.
The second level of loop is different

02:10:16.023 --> 02:10:19.501
companies having different constitutions.
I think it’s useful.

02:10:19.585 --> 02:10:21.427
Anthropic puts out a constitution,

02:10:21.510 --> 02:10:28.510
Gemini puts out a constitution, and other
companies put out a constitution.

02:10:28.800 --> 02:10:34.160
People can look at them and compare.
Outside observers can critique and say,

02:10:34.800 --> 02:10:39.200
"I like this thing from this constitution
and this thing from that constitution."

02:10:40.160 --> 02:10:45.677
That creates a soft incentive
and feedback for all the companies to

02:10:45.760 --> 02:10:49.880
take the best of each element and improve.
Then I think there's a third loop,

02:10:49.963 --> 02:10:50.317
which is

02:10:50.400 --> 02:10:57.400
society beyond the AI companies and beyond
just those who comment without hard power.

02:10:59.280 --> 02:11:02.322
There we've done some experiments.
A couple years ago,

02:11:02.405 --> 02:11:04.637
we did an experiment with the Collective

02:11:04.720 --> 02:11:10.653
Intelligence Project to basically poll
people and ask them what should be in

02:11:10.737 --> 02:11:12.320
our AI constitution.

02:11:15.200 --> 02:11:17.277
At the time, we incorporated
some of those changes.

02:11:17.360 --> 02:11:19.517
So you could imagine doing something
like that with the new approach we've

02:11:19.600 --> 02:11:23.757
taken to the constitution.
It's a little harder because

02:11:23.840 --> 02:11:28.560
it was an easier approach to take when the
constitution was a list of dos and don'ts.

02:11:29.120 --> 02:11:32.797
At the level of principles, it has
to have a certain amount of coherence.

02:11:32.880 --> 02:11:37.837
But you could still imagine getting
views from a wide variety of people.

02:11:37.920 --> 02:11:41.997
You could also imagine—and this
is a crazy idea, but this whole

02:11:42.080 --> 02:11:49.080
interview is about crazy ideas—systems
of representative government having input.

02:11:52.320 --> 02:11:55.437
I wouldn't do this today because
the legislative process is so slow.

02:11:55.520 --> 02:11:59.142
This is exactly why I think we should
be careful about the legislative process

02:11:59.225 --> 02:12:03.867
and AI regulation. But there's no
reason you couldn't, in principle, say,

02:12:03.950 --> 02:12:06.877
"All AI models have to have a constitution

02:12:06.960 --> 02:12:12.176
that starts with these things, and then
you can append other things after it,

02:12:12.259 --> 02:12:13.277
but there has to

02:12:13.360 --> 02:12:17.268
be this special section that takes
precedence." I wouldn't do that.

02:12:17.351 --> 02:12:18.960
That's too rigid and sounds

02:12:22.160 --> 02:12:26.237
overly prescriptive in a way that I
think overly aggressive legislation is.

02:12:26.320 --> 02:12:32.637
But that is a thing you could try to do.
Is there some much less heavy-handed

02:12:32.720 --> 02:12:36.240
version of that? Maybe.
I really like control loop two.

02:12:37.840 --> 02:12:41.890
Obviously, this is not how constitutions
of actual governments do or should work.

02:12:42.480 --> 02:12:46.877
There's not this vague sense in which
the Supreme Court will feel out how people

02:12:46.960 --> 02:12:50.510
are feeling—what are the vibes—and
update the constitution accordingly.

02:12:50.800 --> 02:12:54.480
With actual governments, there's
a more formal, procedural process.

02:12:55.040 --> 02:12:59.017
But you have a vision
of competition between constitutions,

02:12:59.100 --> 02:13:01.357
which is actually very reminiscent

02:13:01.440 --> 02:13:05.006
of how some libertarian charter
cities people used to talk,

02:13:05.089 --> 02:13:07.357
about what an archipelago of different

02:13:07.440 --> 02:13:10.397
kinds of governments would look like.
There would be selection among them of

02:13:10.480 --> 02:13:14.480
who could operate the most effectively
and where people would be the happiest.

02:13:15.200 --> 02:13:21.840
In a sense, you're recreating that
vision of a utopia of archipelagos.

02:13:23.360 --> 02:13:29.284
I think that vision has things to
recommend it and things that will go wrong

02:13:29.368 --> 02:13:30.201
with it.

02:13:31.760 --> 02:13:34.637
It's an interesting, in some ways
compelling, vision, but things will

02:13:34.720 --> 02:13:40.477
go wrong that you hadn't imagined.
So I like loop two as well,

02:13:40.560 --> 02:13:46.317
but I feel like the whole thing has got to
be some mix of loops one, two, and three,

02:13:46.400 --> 02:13:51.600
and it's a matter of the proportions.
I think that's gotta be the answer.

02:13:53.040 --> 02:13:57.718
When somebody eventually writes
the equivalent of The Making of the Atomic

02:13:57.802 --> 02:13:58.877
Bomb for this era,

02:13:58.960 --> 02:14:02.477
what is the thing that will be hardest
to glean from the historical record that

02:14:02.560 --> 02:14:06.877
they're most likely to miss?
I think a few things. One is,

02:14:06.960 --> 02:14:11.277
at every moment of this exponential,
the extent to which the world outside it

02:14:11.360 --> 02:14:15.162
didn't understand it. This is a bias
that's often present in history.

02:14:15.246 --> 02:14:17.277
Anything that actually happened looks

02:14:17.360 --> 02:14:24.157
inevitable in retrospect.
When people look back, it will

02:14:24.240 --> 02:14:30.924
be hard for them to put themselves in the
place of people who were actually making

02:14:31.007 --> 02:14:31.997
a bet on this

02:14:32.080 --> 02:14:36.677
thing to happen that wasn't inevitable,
that we had these arguments like

02:14:36.760 --> 02:14:38.237
the arguments I make for

02:14:38.320 --> 02:14:43.811
scaling or that continual
learning will be solved.

02:14:43.894 --> 02:14:48.717
Some of us internally
put a high probability

02:14:48.800 --> 02:14:55.800
on this happening, but there's a world
outside us that's not acting on that

02:14:55.898 --> 02:14:56.731
at all.

02:14:58.960 --> 02:15:04.960
I think the weirdness of it,
unfortunately the insularity of it...

02:15:07.520 --> 02:15:10.157
If we're one year or two
years away from it happening,

02:15:10.240 --> 02:15:12.014
the average person
on the street has no idea.

02:15:12.097 --> 02:15:14.077
That's one of the things
I'm trying to change with

02:15:14.160 --> 02:15:18.957
the memos, with talking to policymakers.
I don’t know but I think

02:15:19.040 --> 02:15:26.040
that's just a crazy thing.
Finally, I would say—and this

02:15:27.280 --> 02:15:32.868
probably applies to almost all historical
moments of crisis—how absolutely fast it

02:15:32.952 --> 02:15:33.837
was happening,

02:15:33.920 --> 02:15:39.517
how everything was happening all at once.
Decisions that you might think were

02:15:39.600 --> 02:15:42.637
carefully calculated, well actually
you have to make that decision,

02:15:42.720 --> 02:15:46.787
and then you have to make 30 other
decisions on the same day because it's all

02:15:46.870 --> 02:15:47.757
happening so fast.

02:15:47.840 --> 02:15:51.740
You don't even know which decisions are
going to turn out to be consequential.

02:15:52.320 --> 02:15:59.320
One of my worries—although it's also an
insight into what's happening—is that some

02:16:00.880 --> 02:16:05.084
very critical decision will be some
decision where someone just comes into

02:16:05.167 --> 02:16:05.837
my office and

02:16:05.920 --> 02:16:12.920
is like, "Dario, you have two minutes.
Should we do thing A or thing B on this?"

02:16:14.640 --> 02:16:19.808
Someone gives me this random half-page
memo and asks, "Should we do A or B?"

02:16:19.891 --> 02:16:23.098
I'm like, "I don't know.
I have to eat lunch. Let's do B."

02:16:23.181 --> 02:16:25.997
That ends up being the most
consequential thing ever.

02:16:26.080 --> 02:16:33.080
So final question. There aren't tech
CEOs who are usually writing 50-page memos

02:16:33.848 --> 02:16:35.437
every few months.

02:16:35.520 --> 02:16:39.997
It seems like you have managed to build
a role for yourself and a company around

02:16:40.080 --> 02:16:47.080
you which is compatible with this
more intellectual-type role of CEO.

02:16:47.280 --> 02:16:51.964
I want to understand how you
construct that. How does that work?

02:16:52.047 --> 02:16:53.677
Do you just go away for

02:16:53.760 --> 02:16:56.157
a couple of weeks and then you tell your
company, "This is the memo. Here's what

02:16:56.240 --> 02:16:59.597
we're doing"? It's also reported that
you write a bunch of these internally.

02:16:59.680 --> 02:17:02.380
For this particular one,
I wrote it over winter break.

02:17:04.320 --> 02:17:08.557
I was having a hard time finding
the time to actually write it.

02:17:08.640 --> 02:17:12.986
But I think about this in a broader way.
I think it relates to the culture of

02:17:13.070 --> 02:17:13.677
the company.

02:17:13.760 --> 02:17:18.498
I probably spend a third, maybe 40%, of my
time making sure the culture of Anthropic

02:17:18.581 --> 02:17:18.957
is good.

02:17:19.040 --> 02:17:25.557
As Anthropic has gotten larger, it's
gotten harder to get directly involved

02:17:25.640 --> 02:17:26.877
in the training

02:17:26.960 --> 02:17:30.797
of the models, the launch of the models,
the building of the products. It's 2,500

02:17:30.880 --> 02:17:37.880
people. I have certain instincts, but it's
very difficult to get involved in every

02:17:38.397 --> 02:17:39.680
single detail.

02:17:41.120 --> 02:17:45.721
I try as much as possible, but one thing
that's very leveraged is making sure

02:17:45.804 --> 02:17:46.877
Anthropic is a good

02:17:46.960 --> 02:17:50.795
place to work, people like working there,
everyone thinks of themselves as team

02:17:50.878 --> 02:17:54.547
members, and everyone works together
instead of against each other.

02:17:54.630 --> 02:17:57.197
We've seen as some
of the other AI companies

02:17:57.280 --> 02:18:01.933
have grown—without naming any names—we're
starting to see decoherence and people

02:18:02.016 --> 02:18:03.117
fighting each other.

02:18:03.200 --> 02:18:07.450
I would argue there was even a lot of that
from the beginning, but it's gotten worse.

02:18:08.160 --> 02:18:14.637
I think we've done an extraordinarily good
job, even if not perfect, of holding the

02:18:14.720 --> 02:18:17.650
company together, making
everyone feel the mission,

02:18:17.734 --> 02:18:19.837
that we're sincere about the mission,

02:18:19.920 --> 02:18:23.503
and that everyone has faith that everyone
else there is working for the right

02:18:23.587 --> 02:18:27.495
reason. That we're a team, that people
aren't trying to get ahead at each other's

02:18:27.579 --> 02:18:27.997
expense or

02:18:28.080 --> 02:18:32.280
backstab each other, which again, I think
happens a lot at some of the other places.

02:18:33.200 --> 02:18:36.477
How do you make that the case?
It's a lot of things. It's me,

02:18:36.560 --> 02:18:41.037
it's Daniela, who runs the company
day to day, it's the co-founders,

02:18:41.120 --> 02:18:44.317
it's the other people we hire,
it's the environment we try to create.

02:18:44.400 --> 02:18:51.400
But I think an important thing in the
culture is that the other leaders as well,

02:18:52.106 --> 02:18:53.757
but especially me,

02:18:53.840 --> 02:18:58.797
have to articulate what the company
is about, why it's doing what it's doing,

02:18:58.880 --> 02:19:02.665
what its strategy is, what its
values are, what its mission is,

02:19:02.748 --> 02:19:04.160
and what it stands for.

02:19:06.320 --> 02:19:09.837
When you get to 2,500 people,
you can't do that person by person.

02:19:09.920 --> 02:19:12.477
You have to write, or you have
to speak to the whole company.

02:19:12.560 --> 02:19:16.695
This is why I get up in front of the whole
company every two weeks and speak for

02:19:16.778 --> 02:19:17.612
an hour.

02:19:18.320 --> 02:19:22.317
I wouldn't say I write essays internally.
I do two things. One, I write this thing

02:19:22.400 --> 02:19:27.277
called a DVQ, Dario Vision Quest.
I wasn't the one who named it that.

02:19:27.360 --> 02:19:31.112
That's the name it received, and it's
one of these names that I tried to fight

02:19:31.195 --> 02:19:35.723
because it made it sound like I was going
off and smoking peyote or something.

02:19:35.806 --> 02:19:37.997
But the name just stuck. So I get up

02:19:38.080 --> 02:19:42.957
in front of the company every two weeks.
I have a three or four-page document,

02:19:43.040 --> 02:19:47.454
and I just talk through three or four
different topics about what's going

02:19:47.537 --> 02:19:48.400
on internally,

02:19:49.040 --> 02:19:53.997
the models we're producing, the products,
the outside industry, the world as a whole

02:19:54.080 --> 02:19:59.757
as it relates to AI and geopolitically
in general. Just some mix of that. I go

02:19:59.840 --> 02:20:04.111
through very honestly and I say,
"This is what I'm thinking,

02:20:04.195 --> 02:20:10.004
and this is what Anthropic leadership
is thinking," and then I answer questions.

02:20:10.087 --> 02:20:13.517
That direct connection
has a lot of value that

02:20:13.600 --> 02:20:18.240
is hard to achieve when you're passing
things down the chain six levels deep.

02:20:19.040 --> 02:20:26.040
A large fraction of the company comes
to attend, either in person or virtually.

02:20:27.360 --> 02:20:30.159
It really means that you
can communicate a lot.

02:20:30.243 --> 02:20:32.797
The other thing I do
is I have a channel in

02:20:32.880 --> 02:20:35.930
Slack where I just write a bunch
of things and comment a lot.

02:20:36.720 --> 02:20:42.122
Often that's in response to things I'm
seeing at the company or questions people

02:20:42.206 --> 02:20:43.039
ask.

02:20:44.320 --> 02:20:47.750
We do internal surveys and there are
things people are concerned about,

02:20:47.833 --> 02:20:49.133
and so I'll write them up.

02:20:50.640 --> 02:20:56.077
I'm just very honest about these things.
I just say them very directly.

02:20:56.160 --> 02:21:00.432
The point is to get a reputation of
telling the company the truth about what's

02:21:00.515 --> 02:21:04.551
happening, to call things what
they are, to acknowledge problems,

02:21:04.634 --> 02:21:07.597
to avoid the sort
of corpo speak, the kind of

02:21:07.680 --> 02:21:12.697
defensive communication that often
is necessary in public because the world

02:21:12.780 --> 02:21:18.002
is very large and full of people who
are interpreting things in bad faith.

02:21:18.085 --> 02:21:21.757
But if you have a company
of people who you trust,

02:21:21.840 --> 02:21:28.617
and we try to hire people that we trust,
then you can really just be entirely

02:21:28.700 --> 02:21:29.680
unfiltered.

02:21:31.600 --> 02:21:33.517
I think that's an enormous
strength of the company.

02:21:33.600 --> 02:21:37.357
It makes it a better place to work,
it makes people more than the sum of

02:21:37.440 --> 02:21:37.997
their parts,

02:21:38.080 --> 02:21:40.749
and increases the likelihood that we
accomplish the mission because everyone

02:21:40.832 --> 02:21:41.437
is on the same page

02:21:41.520 --> 02:21:44.779
about the mission, and everyone
is debating and discussing how best

02:21:44.863 --> 02:21:46.077
to accomplish the mission.

02:21:46.160 --> 02:21:49.917
Well, in lieu of an external Dario
Vision Quest, we have this interview.

02:21:50.000 --> 02:21:54.557
This interview is a little like that. This
has been fun, Dario. Thanks for doing it.

02:21:54.640 --> 02:22:00.960
Thank you, Dwarkesh.