WEBVTT

00:00:00.000 --> 00:00:02.637
live stream um to this conversation. I

00:00:02.720 --> 00:00:03.997
have to say I have been looking forward

00:00:04.080 --> 00:00:07.197
to for months. Uh I had was lucky enough

00:00:07.280 --> 00:00:09.837
to ch to moderate a conversation between

00:00:09.920 --> 00:00:13.597
Darede and Demis last year in Paris. Um

00:00:13.680 --> 00:00:17.437
which I'm afraid got most attention for
the fact that you two were squashed on a

00:00:17.520 --> 00:00:22.157
very small love seat while I sat
on an enormous sofa which was probably my

00:00:22.240 --> 00:00:26.237
screw- up. But I said at that point that
this was for me like you know chairing a

00:00:26.320 --> 00:00:29.837
conversation between the Beatles and the
Rolling Stones. and you have not had a

00:00:29.920 --> 00:00:33.837
conversation on stage since. So, this is,
you know, the sequel, the the the,

00:00:33.920 --> 00:00:35.517
you know, the bands get together again.

00:00:35.600 --> 00:00:37.997
I'm delighted. You need no introduction.

00:00:38.080 --> 00:00:42.557
Uh, the title of our conversation is the
day after AGI, which I think is perhaps

00:00:42.640 --> 00:00:45.277
slightly getting ahead of ourselves
because we should probably talk about

00:00:45.360 --> 00:00:49.117
how quickly and easily we will get there.
And I want to do a bit of a sort

00:00:49.200 --> 00:00:52.237
of update on that and then talk about
the consequences. So firstly on the

00:00:52.320 --> 00:00:56.877
timeline Dario you last year in Paris
said we'll have a model that can do

00:00:56.960 --> 00:00:58.717
everything a human could do at the level

00:00:58.800 --> 00:01:00.797
of a Nobel laureate across many fields

00:01:00.880 --> 00:01:05.917
by 2627 we're in 26.
Uh do you still stand by that timeline?

00:01:06.000 --> 00:01:09.117
>> So you know it's always hard to know
exactly when something will happen but

00:01:09.200 --> 00:01:11.037
but I don't I don't think that's going

00:01:11.120 --> 00:01:13.677
to turn out to be that far off. So um

00:01:13.760 --> 00:01:18.797
you know the the the mechanism whereby
I imagined it would happen is that we

00:01:18.880 --> 00:01:22.877
would make models that were good
at coding and good at AI research and we

00:01:22.960 --> 00:01:27.277
would use that to produce the next
generation of model and speed it up to

00:01:27.360 --> 00:01:29.757
create a loop that would that would uh

00:01:29.840 --> 00:01:31.917
increase the speed of model development.

00:01:32.000 --> 00:01:34.477
We are now in terms of you know the

00:01:34.560 --> 00:01:39.037
models that write code I have engineers
within enthropic who say I don't write

00:01:39.120 --> 00:01:41.277
any code anymore. I just I just let the

00:01:41.360 --> 00:01:43.757
model write the code. I edit it. I do

00:01:43.840 --> 00:01:46.397
the things around it. I think I don't

00:01:46.480 --> 00:01:49.357
know. We might be six to 12 months away

00:01:49.440 --> 00:01:51.837
from when the model is doing most maybe

00:01:51.920 --> 00:01:55.357
all of what SWES do end to end. And then

00:01:55.440 --> 00:01:57.117
it's a question of how fast does that

00:01:57.200 --> 00:02:02.397
loop close. Not every part of that loop
is something that can be sped up by AI,

00:02:02.480 --> 00:02:06.077
right? There's like chips, there's
manufacturer of chips, there's training

00:02:06.160 --> 00:02:09.517
time for the model. So it's, you know,
I I think there's a lot of uncertainty.

00:02:09.600 --> 00:02:15.117
It's easy to see how this could take
a few years. I don't I I it's very hard

00:02:15.200 --> 00:02:19.517
for me to see how it could take longer
than that. Um but if if I had to guess,

00:02:19.600 --> 00:02:23.437
I would guess that this goes faster
than people imagine. And that that key

00:02:23.520 --> 00:02:26.397
element of code and increasingly

00:02:26.480 --> 00:02:29.837
research going faster than we imagine.

00:02:29.920 --> 00:02:33.757
That's going to be the key driver.
It's it's really hard to predict again how

00:02:33.840 --> 00:02:37.517
much that exponential is going to speed
us up, but but something fast is going

00:02:37.600 --> 00:02:41.757
to happen. So you demis were a little
more cautious last year. You said a 50%

00:02:41.840 --> 00:02:45.917
chance of a system that can exhibit all
the cognitive capabilities humans can by

00:02:46.000 --> 00:02:48.637
the end of the decade. Um clearly in

00:02:48.720 --> 00:02:53.197
coding as Dario says it's been remarkable.
What is your sense of do you

00:02:53.280 --> 00:02:56.157
stand by your prediction
and what's changed in the past year?

00:02:56.240 --> 00:02:59.997
>> Yeah, look I I I I think I'm still on
the same kind of timeline. I think there

00:03:00.080 --> 00:03:01.917
has been remarkable progress. But I

00:03:02.000 --> 00:03:05.357
think some areas of uh uh um kind of

00:03:05.440 --> 00:03:09.997
engineering work, coding or so you could
say mathematics are a little bit easier

00:03:10.080 --> 00:03:13.757
to see how they would be automated
partly because they're verifiable what

00:03:13.840 --> 00:03:17.837
the output is. Um some areas of natural
science are much harder to do than that.

00:03:17.920 --> 00:03:21.197
You won't necessarily know if
the chemical compound you've built or this

00:03:21.280 --> 00:03:24.557
prediction about physics is correct.
It may be you may have to test it

00:03:24.640 --> 00:03:28.797
experimentally and that will all take
longer. So uh I also think there are

00:03:28.880 --> 00:03:31.277
some missing capabilities at the moment

00:03:31.360 --> 00:03:35.997
uh in terms of like not just solving
existing conjectures uh or existing

00:03:36.080 --> 00:03:39.677
problems but actually coming up with the
question in the first place or coming up

00:03:39.760 --> 00:03:43.117
with the theory or the hypothesis.
I think that's much much harder and I

00:03:43.200 --> 00:03:46.477
think that's the highest level
of scientific creativity and it's not

00:03:46.560 --> 00:03:49.277
clear. I think we will have those
systems so I don't think it's impossible

00:03:49.360 --> 00:03:53.277
but I think there may be one or two
missing ingredients. Um, it remains to

00:03:53.360 --> 00:03:56.397
be seen how, you know, first of all,
can this self-improvement loop that we're

00:03:56.480 --> 00:03:59.917
all working on actually close without
a human in the loop. I think there are

00:04:00.000 --> 00:04:02.797
also risks to that with to that kind
of system, by the way, which we should

00:04:02.880 --> 00:04:07.197
discuss and I'm sure we will. But the
the but but that could speed things up

00:04:07.280 --> 00:04:08.797
if that kind of system does work.

00:04:08.880 --> 00:04:11.437
>> We'll get to the risks in a minute.
But one other change I think of the past

00:04:11.520 --> 00:04:15.437
year has been a kind of change in the
pecking order of the race, if you will.

00:04:15.520 --> 00:04:19.037
This time a year ago, we just had
the deepseek moment and everyone was

00:04:19.120 --> 00:04:23.117
incredibly excited about what happened
there and there was still a sense, you

00:04:23.200 --> 00:04:28.557
know, that Google Deep Mind was kind of
lagging open AI. I would say that now uh

00:04:28.640 --> 00:04:31.597
it's looking quite different. I mean,
they've declared code red, right? Um

00:04:31.680 --> 00:04:33.917
it's been quite a quite a year. So, talk

00:04:34.000 --> 00:04:38.877
me through what specifically you've been
surprised by and how well you've done

00:04:38.960 --> 00:04:41.917
this year and whether you think and then
I'm going to ask you about the lineup.

00:04:42.000 --> 00:04:46.397
Well, look, I I think we were I
was always very confident we uh would get

00:04:46.480 --> 00:04:49.997
back to sort of the top of the the
leaderboards and and the soda type of

00:04:50.080 --> 00:04:54.157
models across the board because I think
we've always had like the deepest and

00:04:54.240 --> 00:04:57.837
broadest research bench and it was about
kind of marshalling that all together

00:04:57.920 --> 00:05:01.997
and um getting the intensity and focus
and the kind of startup mentality back

00:05:02.080 --> 00:05:04.077
to the whole organization and it's been

00:05:04.160 --> 00:05:09.117
a a lot of work and um but I think we're
and we're still a lot of work to do um

00:05:09.200 --> 00:05:13.357
but I think you can start seeing the the
the the the you know the the kind of um

00:05:13.440 --> 00:05:17.837
the progress that's been made in both
the models with Gemini 3 but also uh on

00:05:17.920 --> 00:05:22.317
the product side with Gemini app getting
increasing uh market share. So I feel

00:05:22.400 --> 00:05:27.357
like uh we're making great progress um
but there's a ton more work to do um and

00:05:27.440 --> 00:05:30.557
you know we're bringing to bear Google
deep mind's kind of like the engine room

00:05:30.640 --> 00:05:35.037
of Google where we're getting used to um
shipping our models more quickly into

00:05:35.120 --> 00:05:38.477
the product surfaces. One question
for you Daria on on this aspect of it

00:05:38.560 --> 00:05:41.997
because you've just or you're in the
process of you know a new round at an

00:05:42.080 --> 00:05:44.077
extraordinary valuation too. Um but you

00:05:44.160 --> 00:05:49.037
are unlike Demis a let's call it
an independent model maker and there is I

00:05:49.120 --> 00:05:52.077
think an increasing concern that the
independent model makers will not be

00:05:52.160 --> 00:05:56.317
able to continue for long enough until
you get to where the revenues come in.

00:05:56.400 --> 00:06:00.237
Um it's made very openly about open AI
but talk me through how you think about

00:06:00.320 --> 00:06:03.277
that and then we'll get to the AGI itself.
Yeah, I mean, you know, I think

00:06:03.360 --> 00:06:08.157
I think I think how we think about that
is, you know, as we've built better and

00:06:08.240 --> 00:06:12.717
better models, there's been a kind
of exponential relationship, not only

00:06:12.800 --> 00:06:16.477
between how much compute you put into
the model and how cognitively capable it

00:06:16.560 --> 00:06:20.557
is, but between how cognitively capable
it is and how much revenue it's able to

00:06:20.640 --> 00:06:24.797
generate. So, our revenues grown 10x
in the last three years from 0 to 100

00:06:24.880 --> 00:06:27.117
million in 2023, 100 million to a

00:06:27.200 --> 00:06:32.077
billion in 2024, and 1 billion to 10
billion in 2025. And so th those revenue

00:06:32.160 --> 00:06:35.277
numbers, you know, I don't know if that
curve will literally continue. It would

00:06:35.360 --> 00:06:40.157
be crazy if it did. Um, but those
numbers are starting to get not too far

00:06:40.240 --> 00:06:43.757
from, you know, the sca the scale
of the largest companies in the world. So

00:06:43.840 --> 00:06:46.157
there's there's there's always
uncertainty. You know, we're trying to

00:06:46.240 --> 00:06:50.477
bootstrap this from nothing. It's it's
a crazy thing, but but I have confidence

00:06:50.560 --> 00:06:55.437
that if we're able to produce the best
models in the things that we focus on,

00:06:55.520 --> 00:06:59.117
um, uh, then I think then I think things
will go well. And you know I I will I

00:06:59.200 --> 00:07:02.477
will generally say you know I think
I think it's been a good year for both

00:07:02.560 --> 00:07:07.357
both Google and Anthropic and I think
the thing we actually have in common is

00:07:07.440 --> 00:07:10.717
that they're you know they're both kind
of kind of kind of companies that are

00:07:10.800 --> 00:07:13.997
you know or the research part
of the company that are kind of led by

00:07:14.080 --> 00:07:19.117
researchers who focus on the models who
focus on solving important problems in

00:07:19.200 --> 00:07:20.637
the world right who have these kind of

00:07:20.720 --> 00:07:23.677
hard scientific problems as a as a north

00:07:23.760 --> 00:07:27.917
star and and and I think those are
the kind of companies companies that are

00:07:28.000 --> 00:07:31.437
going to succeed going forward and you
know I think I think we share that

00:07:31.520 --> 00:07:33.837
between us >> very much.
Uh I'm I'm going to resist

00:07:33.920 --> 00:07:36.477
the temptation to ask you what will
happen to the companies that are not led

00:07:36.560 --> 00:07:39.917
by researchers uh because
I know you won't answer it.

00:07:40.000 --> 00:07:45.277
But let's then go on to uh
the predictions area now and this we are

00:07:45.360 --> 00:07:48.157
supposed to be talking about the day
after AI. But let's talk about closing

00:07:48.240 --> 00:07:52.557
the loop. This the odds that you will
get models that will close the loop and

00:07:52.640 --> 00:07:55.997
be able to you know power themselves if
you will because that's the really the

00:07:56.080 --> 00:08:00.397
crux for the the winner takes all
threshold approach. Do you still believe

00:08:00.480 --> 00:08:04.797
that we are likely to see that or is
this going to be much more of a normal

00:08:04.880 --> 00:08:09.117
technology where followers
and catchup can can compete?

00:08:09.200 --> 00:08:11.677
>> Well, look, I definitely don't think
it's going to be a normal technology.

00:08:11.760 --> 00:08:15.517
So, I mean, there are aspects already
that as Dario mentioned that it's

00:08:15.600 --> 00:08:19.997
already helping with our coding
and and some aspects of research. The full

00:08:20.080 --> 00:08:23.677
closing of the loop though, I think
is an unknown. I mean, I think it's

00:08:23.760 --> 00:08:28.877
possible to do. you may need AGI itself
to be able to do that in some domains

00:08:28.960 --> 00:08:32.077
again where there these domains you
know where there's there's more messiness

00:08:32.160 --> 00:08:34.237
around them it's not so easy to verify

00:08:34.320 --> 00:08:39.437
your answer very quickly um there's
kind of MP hard domains so as soon as you

00:08:39.520 --> 00:08:43.357
start getting more and you know I also
include by the way for AGI physical AI

00:08:43.440 --> 00:08:46.557
robotics working all of these kind
of things and then you've got you know

00:08:46.640 --> 00:08:51.037
hardware in the loop uh that may uh
limit how fast the self-improvement

00:08:51.120 --> 00:08:55.357
systems can work but I think in coding
and mathematics and these kind areas. I

00:08:55.440 --> 00:08:57.997
can definitely see that working.
And then the question is more theoretical

00:08:58.080 --> 00:09:00.077
one is what is the limit of engineering

00:09:00.160 --> 00:09:03.197
and maths uh to solve uh the natural

00:09:03.280 --> 00:09:07.837
sciences. Dario, you um last year,
I think it was last year that you

00:09:07.920 --> 00:09:09.837
published Machines of Love and Grace um

00:09:09.920 --> 00:09:12.797
which was a very I would say upbeat

00:09:12.880 --> 00:09:17.277
essay about the potential that that you
were going to see unfold and you were

00:09:17.360 --> 00:09:19.597
talking about you know a a what was it a

00:09:19.680 --> 00:09:22.477
genius of data at country

00:09:22.560 --> 00:09:27.117
data center I'm told that you are
working on an update to this a new essay

00:09:27.200 --> 00:09:29.357
so you know wait for it guys it's not

00:09:29.440 --> 00:09:31.437
out yet but it is coming out but perhaps

00:09:31.520 --> 00:09:36.877
you can give us a sort of a sneak
preview of what a year later your big

00:09:36.960 --> 00:09:39.997
take is going to be. >> Yes.
So, you know, my take my take has

00:09:40.080 --> 00:09:43.917
not changed. It has always been my view
that, you know, AI is going to be

00:09:44.000 --> 00:09:47.037
incredibly powerful. I think Demis and I,
you know, kind of agree on that. It's

00:09:47.120 --> 00:09:51.437
just a question of exactly when um uh
and because it's incredibly powerful, it

00:09:51.520 --> 00:09:54.397
will do all these wonderful things like
the ones I talked about in Machines of

00:09:54.480 --> 00:09:58.237
Love and Grace. It, you know, will
help us cure cancer. It may help us to

00:09:58.320 --> 00:10:02.397
eradicate tropical diseases.
It will help us understand understand the

00:10:02.480 --> 00:10:07.277
universe. but that there are these, you
know, immense and grave risks that, you

00:10:07.360 --> 00:10:08.797
know, not that we can't address them.

00:10:08.880 --> 00:10:12.557
I'm not a doomer, but but that, you know,
we we we we we need to think about

00:10:12.640 --> 00:10:16.477
them and we need to address them. And I
wrote Machines of Loving Grace first. I'

00:10:16.560 --> 00:10:20.397
I'd love to give some uh a sophisticated
reason why I wrote that first, but it

00:10:20.480 --> 00:10:24.557
was just that the the positive essay
was easier and more fun to write than than

00:10:24.640 --> 00:10:28.797
the negative essay. Um, so, you know,
I finally spent some time on vacation and

00:10:28.880 --> 00:10:30.797
I was able to write an essay about the

00:10:30.880 --> 00:10:32.397
risks. And even when I'm writing about

00:10:32.480 --> 00:10:36.557
the risks, um, I I I try, you know, I I

00:10:36.640 --> 00:10:40.717
I'm like an optimistic person, right? So
even as I'm writing about these risks, I

00:10:40.800 --> 00:10:44.717
I I wrote about it in a way that was like,
how do we overcome these risks?

00:10:44.800 --> 00:10:48.717
How do we have a battle plan to fight
them? And and and the way I the way I

00:10:48.800 --> 00:10:53.117
framed it was, you know, there's
this scene from Carl Sean's Contact, the

00:10:53.200 --> 00:10:56.797
movie version of it, where, you know,
they they kind of discover alien life

00:10:56.880 --> 00:11:01.517
and this international panel that's like
interviewing um uh you know, people to,

00:11:01.600 --> 00:11:05.677
you know, to be humanity's
representative to meet the alien. Um uh

00:11:05.760 --> 00:11:09.117
and uh one one of the questions they
asked one of the candidates is, you

00:11:09.200 --> 00:11:12.637
know, if you could ask the aliens anyone
question, what it would what what what

00:11:12.720 --> 00:11:17.037
would it be? And one of one of the
characters says, "I would ask, how did

00:11:17.120 --> 00:11:21.997
you do it? How did you manage to get
through this technological adolescence

00:11:22.080 --> 00:11:26.157
without destroying yourselves? How did
you make it through?" And and and ever

00:11:26.240 --> 00:11:28.957
since I saw it, it was like 20 years ago,
I think I saw that movie, it's kind

00:11:29.040 --> 00:11:32.877
of stuck with me. And that that's
the frame that I used, which is which is

00:11:32.960 --> 00:11:35.437
that, you know, we we're we're we are

00:11:35.520 --> 00:11:37.997
knocking on the door of these incredible

00:11:38.080 --> 00:11:40.237
capabilities, right? the the ability to

00:11:40.320 --> 00:11:45.117
build basically machines out of sand,
right? I think I think it was inevitable

00:11:45.200 --> 00:11:47.037
that the instant we started working with

00:11:47.120 --> 00:11:50.957
fire. Um uh but but how we handle it is

00:11:51.040 --> 00:11:53.037
is not inevitable. And so I think the

00:11:53.120 --> 00:11:58.157
next few years we're going to be dealing
with, you know, how do we keep these

00:11:58.240 --> 00:11:59.837
systems under control that are highly

00:11:59.920 --> 00:12:02.637
autonomous and smarter than any human?

00:12:02.720 --> 00:12:05.917
How do we make sure that individuals

00:12:06.000 --> 00:12:09.757
don't misuse them? Right? I have worries
about things like bioteterrorism. How do

00:12:09.840 --> 00:12:13.757
we make sure that nation states don't
misuse them? That's why I've been so

00:12:13.840 --> 00:12:18.157
concerned about, you know, the CCP,
other authoritarian authoritarian

00:12:18.240 --> 00:12:21.437
governments. What are the economic
impacts? Right? I've talked about labor

00:12:21.520 --> 00:12:24.797
displacement a lot. And and you know,
what what haven't we thought of which

00:12:24.880 --> 00:12:28.477
which in many cases, you know, maybe may
be the the hardest thing to deal with at

00:12:28.560 --> 00:12:31.357
all. Um so, you know, I I'm I'm thinking

00:12:31.440 --> 00:12:34.397
through how to address those risks. And

00:12:34.480 --> 00:12:38.557
you know, for for each of these, it's
a mixture of things that we individually

00:12:38.640 --> 00:12:43.197
need to do as as leaders of the of of of
the companies and that we can do working

00:12:43.280 --> 00:12:46.397
together. And then there there's
going to need to be some role for wider

00:12:46.480 --> 00:12:50.637
societal institutions like the like
the government in in in addressing all of

00:12:50.720 --> 00:12:54.717
these. But, you know, I I I just feel
this urgency that, you know, every day,

00:12:54.800 --> 00:12:57.517
you know, there's there's all kinds
of crazy stuff going on in the outside

00:12:57.600 --> 00:13:00.557
world, outside AI, right? Um but but you

00:13:00.640 --> 00:13:03.197
know my my my view is this is happening

00:13:03.280 --> 00:13:08.397
so fast and is such a crisis we should
be devoting almost all of our effort to

00:13:08.480 --> 00:13:10.077
thinking about how to get through this.

00:13:10.160 --> 00:13:13.677
>> So I can't decide whether I'm more
surprised that you a take a vacation b

00:13:13.760 --> 00:13:17.597
when you take a vacation you think about
the risks of AI and c that your essay is

00:13:17.680 --> 00:13:21.997
framed in terms of are we going to get
through the technological adolescence of

00:13:22.080 --> 00:13:25.357
this technology without destroying
ourselves. So, I'm my head is slightly

00:13:25.440 --> 00:13:28.557
spinning, but you then and I can't wait
to read it, but you you you mentioned

00:13:28.640 --> 00:13:32.477
several areas that can guide the rest
of our conversation. Let's start with jobs

00:13:32.560 --> 00:13:35.277
um because you actually have been very
outspoken about that and I think you

00:13:35.360 --> 00:13:38.397
said that half of entry- level white
collar jobs could be gone within the

00:13:38.480 --> 00:13:39.997
next one to five years. But I'm going to

00:13:40.080 --> 00:13:43.517
turn to you Demis because so far we

00:13:43.600 --> 00:13:48.317
haven't actually seen any discernable
impact on the labor market. Um, yes,

00:13:48.400 --> 00:13:51.917
unemployment has ticked up in the US,
but all of the kind of economic studies

00:13:52.000 --> 00:13:56.157
I've looked at and that we've written
about suggest that this is overhiring

00:13:56.240 --> 00:13:57.837
post pandemic that it's really not

00:13:57.920 --> 00:14:00.797
AIdriven. If anything, people are hiring

00:14:00.880 --> 00:14:03.917
to build out AI capability.

00:14:04.000 --> 00:14:08.957
Do you think that this will be as you
know economists have always argued that

00:14:09.040 --> 00:14:12.317
it's not a lump of labor fallacy that
actually there will be new jobs created

00:14:12.400 --> 00:14:16.637
because so far the evidence seems
to suggest that? Yeah, I mean I I think in

00:14:16.720 --> 00:14:19.837
um the near term that is what will happen.
The kind of normal evolution

00:14:19.920 --> 00:14:21.757
when a breakthrough technology arrives.

00:14:21.840 --> 00:14:26.157
So some jobs will get disrupted but I
think new even more valuable perhaps

00:14:26.240 --> 00:14:28.237
more meaningful jobs will get created.

00:14:28.320 --> 00:14:32.477
Um I think we're going to see this year
the beginnings of maybe impacting the

00:14:32.560 --> 00:14:36.637
junior level entry level kind of jobs
internships this type of thing. I think

00:14:36.720 --> 00:14:40.637
there is some evidence I can feel
that ourselves maybe like a slowdown in

00:14:40.720 --> 00:14:44.077
hiring in that but I think that can be
more than compensated by the fact there

00:14:44.160 --> 00:14:48.317
are these amazing creative tools out
there pretty much available for everyone

00:14:48.400 --> 00:14:50.957
uh almost for free that if you know I

00:14:51.040 --> 00:14:53.757
was to talk to a a class of undergrads

00:14:53.840 --> 00:14:58.077
right now I would be telling them to get
really unbelievably proficient with

00:14:58.160 --> 00:15:01.757
these tools I think to the extent
that even those of us building it we're so

00:15:01.840 --> 00:15:05.517
busy building it it's hard to have also
time to really explore the almost the

00:15:05.600 --> 00:15:10.157
capability overhang even today's models
and products have let alone tomorrow's

00:15:10.240 --> 00:15:14.237
and I think that uh can be maybe better
than a traditional internship would have

00:15:14.320 --> 00:15:19.117
been in terms of you sort of leaprogging
uh yourself to be a useful uh in a

00:15:19.200 --> 00:15:22.797
useful in a profession so I think
there's that's what I see happening

00:15:22.880 --> 00:15:27.277
probably in the next 5 years um maybe we
again slightly differ on time scales on

00:15:27.360 --> 00:15:31.037
that but I think what happens after
AGI arrives that's a different question

00:15:31.120 --> 00:15:34.397
because I think really we would be
in uncharted territory at that point

00:15:34.480 --> 00:15:37.037
>> do you think it's going to take longer
than you thought last year when you said

00:15:37.120 --> 00:15:40.557
half of all white color jobs. >> I
have about the same view. I I actually

00:15:40.640 --> 00:15:45.037
agree with you and with Demis that at
the time I made the comment there was no

00:15:45.120 --> 00:15:47.917
impact on the labor market. I wasn't
saying there was an impact on the labor

00:15:48.000 --> 00:15:53.037
market at that moment. Um you know now
I think maybe we're starting to see just

00:15:53.120 --> 00:15:56.317
just the little beginnings of it you
know in software and coding. I even see

00:15:56.400 --> 00:15:59.117
it within within anthropic where you

00:15:59.200 --> 00:16:02.317
know I you know I can look forward I can

00:16:02.400 --> 00:16:06.477
kind of look forward to a time where on
the more junior end and then on the more

00:16:06.560 --> 00:16:10.557
on the more on the more on the more
intermediate end we actually need less

00:16:10.640 --> 00:16:13.437
and not more people and you know we're
thinking about how to deal with that

00:16:13.520 --> 00:16:16.397
within anthropic in a in a in a you know

00:16:16.480 --> 00:16:20.637
sense in a sensible way. Um I you know

00:16:20.720 --> 00:16:24.957
one to five years as of six months ago
I would stick with that. you know, if you

00:16:25.040 --> 00:16:29.037
kind of, you know, connect this to what
I said before, which is, you know, we we

00:16:29.120 --> 00:16:31.037
might have AI that's better than humans

00:16:31.120 --> 00:16:33.837
at at everything in, you know, maybe one

00:16:33.920 --> 00:16:36.157
to two years, maybe a little longer than

00:16:36.240 --> 00:16:38.797
that. The those don't seem to line up.

00:16:38.880 --> 00:16:43.917
The reason is that there's this there's
this lag and there's this replacement

00:16:44.000 --> 00:16:47.197
thing, right? I I know that the labor
market is adaptable, right? Just like

00:16:47.280 --> 00:16:51.517
you know 80% of people used to do
farming you know farming got automated

00:16:51.600 --> 00:16:55.517
and then they became factory workers
and then knowledge workers. So you know

00:16:55.600 --> 00:16:59.757
there is some level of adaptability here
as well right we should be economically

00:16:59.840 --> 00:17:03.917
sophisticated about how the labor
market works but my worry is as this

00:17:04.000 --> 00:17:08.157
exponential keeps compounding and I
don't think it's going to take that long

00:17:08.240 --> 00:17:12.557
again somewhere between between a year
and five years it will overwhelm our

00:17:12.640 --> 00:17:17.117
ability to adapt. I think I may be
saying the same thing Demis is just

00:17:17.200 --> 00:17:20.477
factored out of that that difference
we have about timelines which I think

00:17:20.560 --> 00:17:23.677
ultimately comes down to how how
fast you close the loop on CO and

00:17:23.760 --> 00:17:25.517
>> how much confidence do you have that

00:17:25.600 --> 00:17:28.637
governments get the scale of this and

00:17:28.720 --> 00:17:33.277
have are beginning to think about
what policy responses they need to have

00:17:33.360 --> 00:17:37.677
>> I don't think that that that it's
anywhere near enough work going on about

00:17:37.760 --> 00:17:40.637
this I'm I'm constantly surprised even
when I meet economists at places like

00:17:40.720 --> 00:17:44.557
this that they're not more of uh
professional economist, professors

00:17:44.640 --> 00:17:47.437
thinking about what happens um and not

00:17:47.520 --> 00:17:52.077
just sort of on the way to AGI but um
even if we get all the technical things

00:17:52.160 --> 00:17:55.437
right that Dario is talking about
and the job displacement is one question

00:17:55.520 --> 00:17:57.917
we're worried about the economics
of that but maybe there are ways to

00:17:58.000 --> 00:18:02.077
distribute this new productivity this
new wealth more fairly I don't know if

00:18:02.160 --> 00:18:05.037
we have the right institutions to do
that but that's what should happen at

00:18:05.120 --> 00:18:08.477
that point there should be you know we
may be in a post scarcity world but then

00:18:08.560 --> 00:18:11.357
there are even the things that keep
me up right now there are even bigger

00:18:11.440 --> 00:18:13.517
questions than that at that to do with

00:18:13.600 --> 00:18:18.637
meaning and um purpose and a lot of
the things that we get from our jobs not

00:18:18.720 --> 00:18:22.477
just economically that's one question
but I think that may be easier to solve

00:18:22.560 --> 00:18:27.197
strangely than uh what happens to the
human condition and humanity as a whole

00:18:27.280 --> 00:18:30.397
and I think I'm also optimistic we'll
come up with new answers there we do a

00:18:30.480 --> 00:18:32.957
lot of things today um from extreme

00:18:33.040 --> 00:18:35.037
sports to art that aren't necessarily

00:18:35.120 --> 00:18:38.237
directly to do with economic gain so I

00:18:38.320 --> 00:18:40.397
think we will find uh meaning and maybe

00:18:40.480 --> 00:18:42.797
there'll be even more sort sophisticated

00:18:42.880 --> 00:18:47.597
versions of those activities. Um, plus
I think we'll be exploring the stars. So,

00:18:47.680 --> 00:18:51.997
there'll be all of that to to factor
in as well for in terms of purpose. But I

00:18:52.080 --> 00:18:56.077
think it's really worth thinking now
even on my timelines of like 5 to 10

00:18:56.160 --> 00:18:59.517
years away. That isn't a lot
of time uh before this comes.

00:18:59.600 --> 00:19:01.837
>> How big do you think is the risk of a

00:19:01.920 --> 00:19:04.957
popular backlash against AI that will

00:19:05.040 --> 00:19:09.917
somehow kind of cause governments to do
what from your perspective might be

00:19:10.000 --> 00:19:13.757
stupid things? Because I'm just
thinking back to the era of you know

00:19:13.840 --> 00:19:18.157
globalization in the 1990s when when
there was indeed some displacement of

00:19:18.240 --> 00:19:20.397
jobs governments didn't do enough the

00:19:20.480 --> 00:19:25.277
public backlash was such that we've
ended up sort of where we are now. Uh do

00:19:25.360 --> 00:19:29.437
you think that there is a risk
that there will be a growing antipathy

00:19:29.520 --> 00:19:34.157
towards what you are doing and your
companies in the kind of body politic?

00:19:34.240 --> 00:19:37.917
>> Um I think there's definitely a risk.
I think um I think that's kind of

00:19:38.000 --> 00:19:41.277
reasonable. there's fear and there's
worries about these things like jobs and

00:19:41.360 --> 00:19:45.357
livelihoods. Um I think there's a couple
of things that I mean it's going to be

00:19:45.440 --> 00:19:49.197
very complicated the next few years
I think geopolitically but also the

00:19:49.280 --> 00:19:52.637
various factors here like we want to and
we're trying to do this with AlphaFold

00:19:52.720 --> 00:19:57.277
and our science work and isomorphic
our spinout company solve all disease cure

00:19:57.360 --> 00:20:01.037
diseases come up with new energy sources
I think as a society it's clear we'd

00:20:01.120 --> 00:20:05.197
want that I think maybe the balance of
what the industry is doing is not enough

00:20:05.280 --> 00:20:08.317
balance towards those types
of activities I think we should have a lot

00:20:08.400 --> 00:20:12.237
more examples I know Dario agrees with
me of like alpha fold like things that

00:20:12.320 --> 00:20:15.837
help sort of unequivocal good
in the world. And I think actually it's

00:20:15.920 --> 00:20:19.517
incumbent on the industry and and all
of us leading players to show that more,

00:20:19.600 --> 00:20:23.517
demonstrate that, not just talk about it,
but demonstrate that. Um and but

00:20:23.600 --> 00:20:27.677
then it's going to come with these other
intendent disruptions and um but I don't

00:20:27.760 --> 00:20:30.477
I think the other issue is the
geopolitical competition. There's

00:20:30.560 --> 00:20:33.757
obviously competition between
the companies but also US and China

00:20:33.840 --> 00:20:37.597
primarily. So unless there's
an international cooperation or or

00:20:37.680 --> 00:20:41.117
understanding around this um uh which I
think would be good actually in terms of

00:20:41.200 --> 00:20:44.637
things like minimum safety standards for
deployment I think Dario would agree on

00:20:44.720 --> 00:20:47.357
that as well. I think it's vitally needed.
This technology is going to be

00:20:47.440 --> 00:20:49.997
crossber border. It's going to affect
everyone. It's going to affect all of

00:20:50.080 --> 00:20:54.797
humanity. Um actually contact is one
of my favorite films as well. So funny

00:20:54.880 --> 00:20:59.677
enough, I didn't realize it was yours too,
Dario. But I I think um um you know

00:20:59.760 --> 00:21:03.757
those kind of things need to be worked
through. Um and and if we can maybe it

00:21:03.840 --> 00:21:07.437
would be good to have a bit of slow
a slightly slower pace than we're

00:21:07.520 --> 00:21:11.837
currently predicting even my timelines
so that we can get this right society.

00:21:11.920 --> 00:21:13.677
But that would require some coordination

00:21:13.760 --> 00:21:16.157
that is I I prefer your timelines.

00:21:16.240 --> 00:21:19.117
>> Yes, I will concede.

00:21:19.200 --> 00:21:23.357
>> But but Dario, let's turn to this now
because one thing since we last spoke uh

00:21:23.440 --> 00:21:27.597
in Paris, the geopolitical environment
has, if anything, I don't know,

00:21:27.680 --> 00:21:31.677
complicated, mad, crazy, whatever,
whatever phrase you want to use.

00:21:31.760 --> 00:21:36.077
Secondly, the US has a very different
approach now towards China. It's a much

00:21:36.160 --> 00:21:38.157
more it's a kind of no holds barred, go

00:21:38.240 --> 00:21:43.597
as fast as we can, but then sell chips
to China. Um and that is it. So you've

00:21:43.680 --> 00:21:48.317
got a different attitude towards
the United States. You've got a a very um

00:21:48.400 --> 00:21:51.597
strange relationship between the United
States and and Europe right now

00:21:51.680 --> 00:21:55.517
geopolitically against that. I mean I
hear you talk about it would be nice to

00:21:55.600 --> 00:21:59.117
have a CERN like organization. I mean
it's a million years from where we are

00:21:59.200 --> 00:22:02.637
from the real world. So in the real
world have the geopolitical risks

00:22:02.720 --> 00:22:07.277
increased and what if anything do you
think should be done about that? And and

00:22:07.360 --> 00:22:09.917
the administration seems to be doing
the opposite of what you were suggesting?

00:22:10.000 --> 00:22:12.637
Yeah, I mean, look, you know, we're
we're we're just trying to do the best

00:22:12.720 --> 00:22:15.437
we can to, you know, we're just we're
just one company and we're we're trying

00:22:15.520 --> 00:22:18.797
to operate in, you know, the the
environment that exists, no matter how

00:22:18.880 --> 00:22:22.717
no matter how crazy it is. But, you know,
I think I think at least my policy

00:22:22.800 --> 00:22:25.597
recommendations haven't changed that,

00:22:25.680 --> 00:22:28.957
you know, not selling chips is one of

00:22:29.040 --> 00:22:33.437
the, you know, one of the one of the
biggest things we can do um to, you

00:22:33.520 --> 00:22:37.917
know, make sure that we have the time
to handle this. Um, you know, you know, I

00:22:38.000 --> 00:22:40.397
said I said before, you know, I I I

00:22:40.480 --> 00:22:45.117
prefer Demis' timeline. I wish we had
five to 10 years, you know, so it's it's

00:22:45.200 --> 00:22:48.317
possible he's just right and I'm just
wrong, but but assume I'm right and it

00:22:48.400 --> 00:22:51.917
can be done in one to two years.
Why can't we slow down to to Demis'

00:22:52.000 --> 00:22:56.317
timeline? Well, no.
The but but but the reason the

00:22:56.400 --> 00:22:58.637
reason we the reason we can't do that is

00:22:58.720 --> 00:23:01.917
is you know because we have geopolitical

00:23:02.000 --> 00:23:04.877
adversaries building the same technology

00:23:04.960 --> 00:23:09.357
at a similar pace. It's very hard
to have an enforcable agreement where they

00:23:09.440 --> 00:23:11.437
slow down and we slow down and and so if

00:23:11.520 --> 00:23:14.397
we can just if we can just not sell the

00:23:14.480 --> 00:23:19.517
chips then this isn't a question
of competition between the US and China.

00:23:19.600 --> 00:23:22.477
This is a question of competition
between me and Demis which I'm very

00:23:22.560 --> 00:23:24.397
confident that we can work out.

00:23:24.480 --> 00:23:26.237
>> And what do you
make of the logic of the

00:23:26.320 --> 00:23:27.917
administration which as I understand it

00:23:28.000 --> 00:23:29.437
is we need to sell them chips because we

00:23:29.520 --> 00:23:33.597
need to bind them into US supply chains.

00:23:33.680 --> 00:23:37.677
>> So you know it's it's I I think it's I

00:23:37.760 --> 00:23:40.157
think it's a question not just of time

00:23:40.240 --> 00:23:41.917
scale but of the significance of the

00:23:42.000 --> 00:23:45.197
technology. Right? If this was telecom

00:23:45.280 --> 00:23:47.597
or something, then all this stuff about

00:23:47.680 --> 00:23:52.317
proliferating the US stack and you
know wanting to build our you know chips

00:23:52.400 --> 00:23:53.917
around the world to make sure that you

00:23:54.000 --> 00:23:57.677
know you know this c you know the you

00:23:57.760 --> 00:24:01.197
know these random countries in different
parts of the world you know build data

00:24:01.280 --> 00:24:03.357
centers that have Nvidia chips instead

00:24:03.440 --> 00:24:06.077
of Huawei chips you know I think of this

00:24:06.160 --> 00:24:08.317
more as like you know it's a decision

00:24:08.400 --> 00:24:11.357
are we going to you know sell nuclear

00:24:11.440 --> 00:24:14.157
weapons to North Korea Uh and you know

00:24:14.240 --> 00:24:18.477
because that produces some profit
for Boeing. Um you know where where we can

00:24:18.560 --> 00:24:22.477
say okay yeah these cases were made
by Boeing like the US is winning like this

00:24:22.560 --> 00:24:24.557
is great like I I I just you know that

00:24:24.640 --> 00:24:27.677
that analogy should just make clear how

00:24:27.760 --> 00:24:29.677
I see this trade-off that I just don't

00:24:29.760 --> 00:24:32.157
think it makes sense. Um and and we've

00:24:32.240 --> 00:24:37.437
done a lot of more aggressive stuff
to you know toward towards towards China

00:24:37.520 --> 00:24:40.957
and other players that that I think
is much less effective than this this one

00:24:41.040 --> 00:24:44.557
this one measure. One more area from me
and then I hope we'll have time for a

00:24:44.640 --> 00:24:47.037
question or two. The other area of

00:24:47.120 --> 00:24:49.197
potential risk that doomers worry about

00:24:49.280 --> 00:24:54.397
is a kind of all powerful malign AI.
Um and I think you've both been somewhat

00:24:54.480 --> 00:24:58.797
skeptical of the doomer approach
but in the last year we have seen you know

00:24:58.880 --> 00:25:00.477
these models showing themselves to be

00:25:00.560 --> 00:25:04.317
capable of deception duplicity. Uh do

00:25:04.400 --> 00:25:08.237
you think that do you think differently
about that risk now than you did a year

00:25:08.320 --> 00:25:12.637
ago? And is there something about
the way the models are evolving that we

00:25:12.720 --> 00:25:14.557
should put a little bit
more concern on that?

00:25:14.640 --> 00:25:17.757
>> Yeah, I mean you know since since
the beginning of anthropic we've kind of

00:25:17.840 --> 00:25:21.837
thought about this risk. I mean you know
our our our research at the beginning of

00:25:21.920 --> 00:25:25.837
it was very theoretical right? You know
we pioneered this idea of mechanistic

00:25:25.920 --> 00:25:29.917
interpretability which is looking inside
the model and and trying to understand

00:25:30.000 --> 00:25:33.677
looking inside its brain trying
to understand why it does what it does as

00:25:33.760 --> 00:25:37.597
it you know as as human neuroscientists
which we actually both have background

00:25:37.680 --> 00:25:40.477
in. um try try to understand try to

00:25:40.560 --> 00:25:44.637
understand the brain and I think as time
has gone on we've we've increasingly

00:25:44.720 --> 00:25:48.797
documented the you know bad behaviors
of the models when they emerge and are now

00:25:48.880 --> 00:25:53.277
working on trying to address them with
mechanistic interpretability. So I you

00:25:53.360 --> 00:25:56.877
know I think uh you know I I've always
been concerned about these these risks.

00:25:56.960 --> 00:26:00.797
I've talked to Demis many times. I
think he has also been um concerned about

00:26:00.880 --> 00:26:05.677
these risks. I think I have definitely
been and I I I would guess Demis as well

00:26:05.760 --> 00:26:10.157
although I'll let him speak for himself
skeptical of of doomerism which is you

00:26:10.240 --> 00:26:12.237
know we're doomed there's nothing we can

00:26:12.320 --> 00:26:14.957
do or this is the most likely outcome. I

00:26:15.040 --> 00:26:17.517
think this is a risk this is a risk that

00:26:17.600 --> 00:26:19.997
if we work all work together we can

00:26:20.080 --> 00:26:22.797
address we can learn through science to

00:26:22.880 --> 00:26:25.837
properly you know control and and direct

00:26:25.920 --> 00:26:27.837
these creations that we're building. But

00:26:27.920 --> 00:26:31.677
if we build them poorly, if we go, you

00:26:31.760 --> 00:26:36.397
know, if if if we're all racing and we
go so fast that there's no guard rails,

00:26:36.480 --> 00:26:38.237
then I think there is risk
of something going wrong.

00:26:38.320 --> 00:26:40.877
>> So, I'm going to give you a chance
to answer that in the context of of a

00:26:40.960 --> 00:26:42.637
slightly broader question, which is over

00:26:42.720 --> 00:26:44.797
the past year, have you grown more

00:26:44.880 --> 00:26:48.477
confident of the upside potential of the

00:26:48.560 --> 00:26:52.077
technology, science, all of the areas
that you have talked about a lot, or are

00:26:52.160 --> 00:26:56.477
you more worried about the risks that
we've been discussing? Look as I've been

00:26:56.560 --> 00:26:58.557
working on this for 20 plus years. So we

00:26:58.640 --> 00:27:00.877
we already knew the reason I've spent my

00:27:00.960 --> 00:27:05.677
whole career on AI is is the upsides
of solving basically the ultimate tool for

00:27:05.760 --> 00:27:09.117
science and understanding the universe
around us. I've I've sort of been

00:27:09.200 --> 00:27:13.277
obsessed with that since a kid and and
and building AI is the you know should

00:27:13.360 --> 00:27:16.317
be the ultimate tool for that if we
do it in the right way. The risks also

00:27:16.400 --> 00:27:19.597
we've been thinking about since the start
at least the start of deep mine 15

00:27:19.680 --> 00:27:23.997
years ago and um we kind of sort of foraw
that if you got the upsides it's a

00:27:24.080 --> 00:27:28.237
dual purpose technology so it could be
repurposed by say bad actors for harmful

00:27:28.320 --> 00:27:31.437
ends so we've needed to think about
that all the way through but I'm a big

00:27:31.520 --> 00:27:34.157
believer in human ingenuity um but the

00:27:34.240 --> 00:27:37.117
question is having the time and the

00:27:37.200 --> 00:27:41.597
focus and all the best minds
collaborating on it to solve these

00:27:41.680 --> 00:27:45.517
problems. I'm sure if we had that we
would solve the technical risk problem.

00:27:45.600 --> 00:27:48.957
It may be we don't have that and then
that will introduce risk because we'll

00:27:49.040 --> 00:27:52.477
be sort of it'll be fragmented. There'll
be different projects and people be

00:27:52.560 --> 00:27:55.917
racing each other. Then it's much harder
to make sure you know these systems that

00:27:56.000 --> 00:28:00.957
we produce will be technically safe. But I
I feel like that's a very tractable uh

00:28:01.040 --> 00:28:04.797
problem. >> If you if you
have the time I want to

00:28:04.880 --> 00:28:07.997
make sure there's one question gentlemen
keep it very short because we've got

00:28:08.080 --> 00:28:11.757
literally two minutes. Thanks for Hello.

00:28:11.840 --> 00:28:14.477
>> Yeah. No speak. >> Thanks very much.
I'm Philip, co-founder

00:28:14.560 --> 00:28:18.237
of Star Cloud Building Data Centers
in Space. Um, I wanted to ask a very

00:28:18.320 --> 00:28:19.997
slightly philosophical core question.

00:28:20.080 --> 00:28:23.357
The sort of strongest argument
for doomerism to me is the Fermy paradox,

00:28:23.440 --> 00:28:26.477
the idea that we don't see intelligent
life in our galaxy. I was wondering if

00:28:26.560 --> 00:28:28.237
you guys have any thoughts.
>> Yeah, I've thought a lot about that.

00:28:28.320 --> 00:28:29.837
That can't be the reason because we we

00:28:29.920 --> 00:28:32.957
we should see all the AIs that have So,

00:28:33.040 --> 00:28:36.957
just for everyone know the idea is well,
it's sort of unclear why that would

00:28:37.040 --> 00:28:40.157
happen, right? So if if the reason
there's a Fmy paradox there are no

00:28:40.240 --> 00:28:44.077
aliens because they get taken out by
their own technology we should be seeing

00:28:44.160 --> 00:28:48.237
paper clips coming towards us from some
part of the galaxy and apparently we

00:28:48.320 --> 00:28:50.397
don't we don't see any structures Dyson

00:28:50.480 --> 00:28:55.277
sphere is nothing whether they're AI
or natur or sort of biological so to me um

00:28:55.360 --> 00:28:57.677
there has to be a different answer
to fmy patterns I have my own theories

00:28:57.760 --> 00:28:59.117
about that but it's out of scope for the

00:28:59.200 --> 00:29:01.677
next minute but um you know I I just

00:29:01.760 --> 00:29:04.877
feel like uh that that I my prediction

00:29:04.960 --> 00:29:09.037
my feeling is that we're past the great
filter it probably multisellular life if

00:29:09.120 --> 00:29:13.837
I would have to guess was incredibly
hard for for biology to evolve that. Um

00:29:13.920 --> 00:29:16.957
so we're on you know there isn't
a comfort of like what's going to happen

00:29:17.040 --> 00:29:20.397
next. I think it's for us to write
as humanity what's going to happen next.

00:29:20.480 --> 00:29:23.197
>> This this could be a great discussion
but is out of scope for the next 36

00:29:23.280 --> 00:29:27.997
seconds. But what isn't 15 seconds each
what when we meet again I hope next year

00:29:28.080 --> 00:29:29.917
uh the three of us which I would love uh

00:29:30.000 --> 00:29:33.917
what will have changed by then? I well I

00:29:34.000 --> 00:29:36.317
think the biggest thing to watch is this

00:29:36.400 --> 00:29:40.157
issue of AI systems building AI systems

00:29:40.240 --> 00:29:42.157
how that goes whe that whether that goes

00:29:42.240 --> 00:29:45.357
one way or another that that will

00:29:45.440 --> 00:29:47.677
determine you know whether it's a few

00:29:47.760 --> 00:29:50.157
more years until we get there or or if

00:29:50.240 --> 00:29:53.357
we have you know you know if if we have

00:29:53.440 --> 00:29:57.757
wonders and and a great emergency
in front of us that we have to face

00:29:57.840 --> 00:30:00.717
>> AI systems building >> I agree
on that so we're we're keeping

00:30:00.800 --> 00:30:04.877
close touch about that um but also I
think um outside of that I think there

00:30:04.960 --> 00:30:08.637
are other interesting uh uh uh ideas
being researched like world models

00:30:08.720 --> 00:30:11.757
continual learning these are the things
I think that will need to be cracked if

00:30:11.840 --> 00:30:15.597
self-improvement doesn't sort of deliver
the goods on its own then we'll need

00:30:15.680 --> 00:30:19.197
these other things to work and then I
think things like robotics may have its

00:30:19.280 --> 00:30:22.077
sort of breakout moment >>
but maybe on the basis of what you've

00:30:22.160 --> 00:30:24.877
just said we should all be hoping that it
does take you a little bit longer and

00:30:24.960 --> 00:30:27.437
indeed everybody else to give us >>
I would prefer that I think that would

00:30:27.520 --> 00:30:28.637
be better for the world

00:30:28.720 --> 00:30:34.760
>> but you guys could do something
about that thank you So it's very MUCH