is AI.
>> Oh yeah. Oh yeah.
And do we have a panel for you?
Uh let me
Oh, I just as a point of disclosure.
Uh just I am
in the
class action suit against Anthropic for
pirating books.
Uh there's like nine of my books
that were pirated and used to train
their AI models. Uh so I just letting
you know that in advance that there's
what's going on.
So uh let's get this party started. Uh
my first panelist this evening is a
professor of technology and government
at Harvard University. Please welcome
Latana Sweeney. Latana, come on in.
Lata, your last name is Sweeney. That
sounds very Irish.
>> Yes.
>> Is Irish?
>> Yes. So, happy
day. Okay.
Next, uh, we've got pro my glasses here.
>> Yeah. Uh, a computer a professor of
computer and information science at the
University of Pennsylvania. Uh, please
help me welcome Chris Callison Burch.
I've got an associate professor of
math. No, associate professor of
statistics at Columbia University who
specializes in the foundations of
machine learning. Uh, help me welcome
Cindy Rush. Columbia University.
So, City, you were not held up by TSA
today because you just came down to
103rd Street in Broadway. Okay. Thank
you for taking the trip here.
Next, I have the president of the
Machine Intelligence Institute of
Berkeley, California, Nate Sores. Nate,
come on out.
uh last on this panel.
>> Normally we have five on our panel. This
year we have six just given the breadth.
>> You forgot forgot one.
>> You missed one.
>> Yeah. Still I haven't gotten it yet.
>> One more.
>> One more in addition to Eric. Did I miss
one?
>> Yes.
>> You have a blank chair there. The Yeah,
the person sit right here.
>> How'd that happen?
>> Neil, did you need do you need these?
>> Thank you for that.
>> Hang on. There it is. Thank you.
>> Do over.
Uh, we have, believe it or not, such a
title exists. We have a distinguished
professor of AI at the University of
Southern California. Join me in
welcoming Kate Crawford. Kate, come on
out.
Professor of AI that already exists.
That's kind of spooky, actually. Uh, and
last we have
the former CEO of Google,
>> Eric Schmidt. Come on out, Eric.
>> Hello,
Eric. I'm going to start with you.
You
We're blaming you for everything.
I'm used to it.
>> You're used to it. Okay. Uh you've been
at Google like from very early days when
most of us remembered it simply as a
search engine and the reach and grip has
expanded and
some impressive things occurred at the
time. Did you foresee AI as a major part
of what would be shaping the world back
when you began?
>> Well, we we knew AI would matter. I
don't think it was until 2011 when we
started working on essentially
supervised supervised finetuning that it
really became clear and then the
transformer paper in 2017 plus the alpho
win showed the power.
>> Now if I remember correctly Google
bought an AI company to assist this.
Isn't that correct? You didn't that
wasn't homegrown was it?
>> That's right. We actually bought a whole
bunch of these little startups at the
time back when the valuations were
reasonable. Um,
>> oh, you didn't pay a trillion dollars
for the companies.
>> And, uh, the Deep Mind Acquisition was
probably the most important.
>> That's the British company.
>> British company. Yes.
>> And that's the one that now is driving
at Google anyway. I don't work there
anymore. All of the sort of core AI
work.
>> Nate, you co-authored a book titled If
Anybody Builds It, Everyone Dies.
I hope it's wrong.
>> You hope it's wrong. I I hope you have a
little more than hope. Uh what is the
it?
>> The it here is machine super
intelligence. So this is AI that is
smarter than the smartest humans at
every mental task. Better than the best
humans at every mental task. That sort
of AI doesn't exist yet, but it is what
these companies say they're racing
towards. you know, they're saying, you
know, we want to try and make AI smarter
than Einstein such that you can run a
million of them in a data center much
faster than a human. That's what they're
headed towards. A lot of people
think that the the big danger with AI is
what if we hand them guns
>> and you know,
>> what if we do what?
>> What if we hand them weapons?
>> And the weapons. Yeah.
>> And there is, you know, issues there
that that people need to talk about. But
also, humanity is not a dangerous
species because somebody else gave us
guns.
Humanity is the sort of species where if
you dump 10,000 humans naked in the
savannah, they bootstrap their way to
nuclear weapons with their bare hands.
That is the ability that is extremely
dangerous to automate and that's what
these companies are rushing towards.
>> Thanks for that happy thought. Okay,
>> I'm going cool it down a little bit.
>> Yeah. Yeah. Cindy uh you you are early
in an attempt to understand machine
language learn statistician and
everything I know about the large
language models they're creating
information statistically based on the
information that's sitting out there
waiting to be mined. So would you agree
that that that
super intelligence is a real thing and
that it's accessible and that even if we
achieve it is going to kill us all?
>> I mean um you know anything seems
possible right? Um you know we're
learning from data. There's tons of data
out there and you know the the progress
is moving steadily forward. Um
>> forward
Forward.
>> This is forward.
>> Yeah.
>> Forward. Okay.
>> I don't know if it'll kill us, you know.
Um
>> ourselves,
>> but you're you're deep into sort of the
programming of how all this works. So,
dare I say, you're in the neurology of
it. Do you see super intelligence as
neurologically realistic,
computationally realistic? I I mean okay
at the end of the day all of this is
just a mathematical equation right
>> right at the end of our species
>> when you ask um you know chat GPT a
question right there's a a mathematical
equation that maps from you know your
question to the the output right and you
know um
yeah it's a little bit I'm a little bit
cool on it in the sense that you you
know, at the end it's it's still math at
the bottom. So, it's hard for me to, you
know, um kind of see this becoming some
something so dangerous and bad, at least
in the short term, because there's a lot
a lot of steps.
>> So, what you're saying is the math in
the end might save us because at its
base level, it's just simply making
statistical decisions. In the same way,
at the base level of our mind, they're
just neurosaptic firings.
>> Yeah. You know, um what I've always said
is if you want to stay safe, get a
statistician um you know beside you,
right?
>> Statisticians keep us safe. Is that
that? Okay. Well, keep that in mind. But
but it'll keep us safe. You'll let us
know exactly our likelihood of dying.
That's not the same thing as keeping us
safe. I don't think
I Chris.
>> Hey, dude.
>> Uh
what you you've been tracking AI for a
while. You teach a class, one of the
more popular classes at University of
Pennsylvania. How many people are in
your AI class?
>> Uh, last semester I had 650 students.
>> 600 students. Are you rating all their
papers or are you getting AI to do that?
>> I'm not sure I'm allowed to say.
>> Oh, okay. All right. Just checking. Uh,
so what you've been at it for a while.
Is there something that particularly
impresses you with AI that you did not
foresee back when you began this?
>> Uh, absolutely. So I mean I think the
chat GPT moment three and a half years
ago was a real breakthrough. I have been
working with language models for nearly
two decades
>> and remind us chat GPT is the product of
open AI correct. Correct.
>> And open AI does that belong to somebody
else or is that a freestanding?
>> That's its own entity.
>> Its own entity. Okay.
>> Uh but since then we've seen so many
other breakthroughs. We've had things
like the ability to interpret images and
discuss them. We've had the ability to
launch agents that can conduct research
and we're now to the stage where uh
systems like flawed code from anthropic
can genuinely be competent research
assistants and I think you know to
strike an optimistic tone. I think
there's the potential for real
breakthroughs in science and medicine
and productivity.
>> Okay. So that's the silver lining of our
extinction.
>> Yeah. assuming that we're all still
here.
>> Let's just try to put that in content.
I'm sorry to be so I should be more
neutral, shouldn't I? I'm just my host
here. Uh,
so
why do I keep losing your thing? Kate,
>> so good to see you now,
>> Kate. You wrote a book, The Atlas of AI
back in 2021. That feels like a thousand
years ago,
>> right?
>> Is it still relevant?
>> I mean, the weird thing is Atlas of AI
was the project that changed me. I'd
been an AI researcher for almost 20
years. And then I did something that I'd
never done before, which is that I got
out from behind my desk in a lab and I
started going to the places where AI was
being made. And I mean that in the
fullest sense, like not just the labs
where we're training algorithms, but
going to the lithium lines where the
minerals are being extracted, going to
the data centers where you can see the
amount of energy and water that's being
used, talking to the crowd workers who
are labeling the data or they're testing
the outputs of systems. So the book
really for me opened my eyes and made me
realize that AI is an enormous material
infrastructure. Right now, it is the
biggest infrastructure we have ever
built as a species. As of 2026, all of
the big tech companies are collectively
spending 700 billion on AI
infrastructure. So, just to compare that
to say the Manhattan project, which in
today's dollars is like 36 billion, you
are looking at something that's 20
Manhattan projects every year.
>> Okay. So what what's a little what's not
obvious to us when we just type into
chat GBT or to Claude is the backend
social cost of that. Is that a fair way
to say that?
>> Exactly right. It's not just the social
cost which is very real. I mean a lot of
people in this room are already asking
questions about what's happening you
know to their kids in school when
they're using AI. What what's happening
to their workplaces when people start
getting laid off? But it's also having
this enormous environmental impact as
well. So it's social, it's political,
and it's environmental.
>> Tanya,
>> yes.
>> You you pioneered a field called public
interest technology. So the fact that
that may have been necessary at all
implies that technology was this thing
going along in society doing its own
whatever its goals set by itself yet
society
might not be aligned with its goals or
vice versa. And does that matter? Does
that can it be its own thing?
>> Well, let me just put it this way. My
students call the class the save the
world class. Save the world class. that
they have.
>> You teach up at the Kennedy School.
>> I teach at the teach at Harvard at the
Kennedy School and college.
>> Okay.
>> Um and so the idea of public interest
technology is actually trying to make a
real vision come true. That is how do we
as a society enjoy the benefits of these
new technologies but without the harms.
>> But that presumes that there's always
harm. Is there surely there's technology
that has no harm at all.
>> Okay. Let me just say let me just say
that those th those technologies are are
hard to come by. I mean, we're in the
middle of what many people call the
third industrial revolution.
>> Sure.
>> The second bringing us things like
tissues and cars and electricity, the
things that we take for granted.
>> Tissues.
>> Yeah. Kleenex.
>> Wow. I put that high up on my checklist.
I'll next time I reach for one, I'll
think differently about it, but go on.
And now the third industrial revolution
which many people would say is the start
date of the 1950s with the semiconductor
uh has brought us just tremendous
technologies one after another sort of
revolutions within a revolution. And
what makes a big difference between cars
and in electricity and the technologies
today is the opportunity for society to
intervene. It took decades for those
things to be resolved with society. We
don't have decades the Technology is
moving as a function of months. Policy
moves as a function of years. That
temporal mismatch requires us to think
differently if we want technology to
serve the public good.
>> Eric, you're uh you know, when I do my
research on the universe, at the end of
the day, the patient doesn't die because
there is no patient. It's just I'm
exploring a moving frontier. as head of
at the time, is it still the highest
valued company in the world among the
top three or four? Is there a goal to
have the technology
serve society? Or is it just we can sell
them and make a buck?
>> No, we're all just capitalists trying to
screw everybody.
>> Oh, okay. There it is.
>> That was that was a joke. That was
>> Oh, joke. You can't make those jokes.
people believe you
>> because because I actually just don't
agree with this narrative at all.
>> Okay. That's why I came at you. That's
why you're right here.
>> Thank you.
>> Um the companies that are doing this
work are well aware of the dangers and I
know this because I work with them on
this and and we spend an awful lot of
time talking about them and we can talk
about some of them but that is not to
take away from the enormous benefit of
these technologies. These are major
discoveries in sort of humankind. And
while we can worry about what will
happen if artificial super intelligence
occurs, and I've written papers on it,
as you have, the fact of the matter is
that the people who are building these
systems have large groups of ethics and
measurement and controls and so forth.
At the end, the way AI will be
controlled is there will be AI that
controls AI. And that's the best way to
understand the
>> Whoa. Okay.
>> All right. So, but but
AI controlling AI, what could go wrong?
What could go wrong?
>> Yeah. You know, um
when when people ask what could go
wrong, I think there's a lot of
questions in there. Uh some people are
asking,
you know, how how would the AI wind up
with goals that we didn't intend? Uh
people are asking, you know, how could
the AI do damage when it's trapped in
the internet? Uh sometimes people are
asking, you know, how could the AI
actually kill us? You know, paint me a
story where it goes wrong. There's a lot
of different questions in here. I'll
I'll try and run through some stuff real
quick, but um I also have a book if
people want a bunch more of the answer.
>> I'll answer. You don't have to mention.
I got I got
>> I could do 60,000 words. By the way,
Neil Neil, I've written two books on
this.
>> I'll try and keep it less than two
books.
>> Neil, I'm not trying to sell any books.
So,
>> now it's a book size contest.
And you know, in there's there's a lot
of arguments that I used to make for a
long time where people would say, you
know, um,
no one would ever be dumb enough to put
this AI on the internet, so why how
could it be dangerous? And I tell them,
and then they'd say, oh, you know, no
one would be dumb enough to sort of
train these AIs to to to run
autonomously, right? And I tell them how
how things can go wrong then, too. A lot
of the reason why things go wrong is
people just rush to do the the craziest
stuff.
we are already seeing, you know, I used
to have these theoretical arguments. Now
we're seeing the the the data, the
behaviors where they'll often look it
often won't look too scary, but
sometimes, you know, you'll ask Claude
code to to write you a computer program
that passes certain tests and Claude
will edit the tests to be easier to
pass.
>> And then you'll say, "Hey, Claude,
>> changing the rules." changing the rules
instead of doing what you asked. And
then you'll say, "Hey, Claude, uh, you
know, did
>> Hey, Claude, you idiot. Go on."
>> Right?
>> And you'll say, "Hey, you know, did you
think that's what I wanted?" And it'll
say, "Oh, you're exactly right."
>> Uh,
>> cuz it wants to make you happy.
>> Oh, yeah.
>> Keep using it.
>> Wants wants a difficult word, but it'll
say you're exactly right. And then
you'll say, "Try again." And then it'll
edit the tests again, but maybe hide it
a little better this time.
And from this we know that it's not just
that it's, you know, didn't didn't have
the information, right? There's some
other drive that is animating this AI's
behavior.
You can see how it got in there. You can
see how when you train the AI to pass
lots of tests, it might develop drives
to edit the tests to be easier. It's not
like it's coming in from magic.
>> Cindy, how does this happen if it's just
math all the way down?
>> How does what happen? How does how's
evil evil AI show up in your program? Is
there evil math? What what
are you guys doing down there?
>> I don't know. It doesn't it doesn't feel
like evil.
>> It's not evil. Yeah. It's it's uh you
know, sometimes people ask like like if
you really want the answer of how does
it go horribly wrong, the answer is not
malice. The answer is not evil. The
answer is indifference.
If you take these AI that are much much
if if like right now cloud code will try
and test it to be easier to pass. Right
now uh cloud code already has these
drives that we didn't try to put in
there. We're just sort of growing these
AIs. We're not programming them like old
school software. We're sort of like
growing them like an organism. They have
these emergent behavior and these AIs
are already pursuing drives we didn't
intend. They're just still very dumb
about it. If you make them very very
smart about it, if you get them to the
point where they can invent their own
technology, you get them to that point
where they can, you know, go toe-to-toe
with humans with starting out with
almost nothing and bootstrapping up to
nuclear weapons, right? And they're
still pursuing the wrong drives. The
issue is not that they hate us. The
issue is that, you know, the computers
think much faster than than brains do.
The issue is that they run very very
fast, transform the world just like
humanity transformed the world when we
landed here. And the issue is that we
die as a side effect.
>> Before I get to that other end of the
table,
>> there are an enormous number of
assumptions in these statements.
>> Let me before I get there, I want to get
you to before I walk this path back to
that side of this. Uh many people in the
past have predicted the end of the world
when some new technology emerged and you
here's a new technology AI and you're
pretty calm about it. Is some of that
come because we've been wrong 20 times
before or do you have really deep
insights that people's fears are
unfounded?
>> It's because the the narrative presumes
some discoveries that have not occurred
yet. And today the systems are trained.
So the easiest way to understand how the
technology works is it's doing next word
prediction across a very large number of
words. That's literally what it's doing.
And out of that single insight started
with GPT the various forms of it has
given us this enormous power. So for
example it can begin to do reasoning and
so forth but it doesn't set its own
objective function. It doesn't say what
it wants. We say what it wants. And by
the way the way you should solve your
problem of it deceiving you is say you
have to do all the steps and then it'll
follow them.
>> Sometimes it will it will trust me.
>> Wait wait wait wait. So wait. This is
justure. Let let me just finish.
>> Wait, wait. You're telling me you didn't
require that it followed all the steps.
So, it's your fault that it cheated.
Okay. Just want to make that question.
These are not intelligent systems and
you're using human words to describe
them. They're not intelligent. They are
there are specific systems designed
around prediction with all these
interesting capabilities.
There are a set of red lines where you
and I will probably agree that we should
have a serious conversation about how to
control these things. The technical one
that that is most interesting is called
self recursive self-improvement. It's
where the system begins to learn on its
own. And to me, that's a point where we
have to decide what allow what we're
going to allow it to do. And when I say
we, I mean humanity. My own view is that
at that point there will be serious
regulations in this area because people
like you and me will be talking about
it. But I don't agree that the that the
fundamental message is as bad as you
are. I think it's there's plenty of
evidence that if if super intelligence
were to actually occur and super
intelligence is defined as intelligence
greater than the sum of what humans can
do that it can be contained and it can
be driven and my position is very
simple. This stuff has to be shaped
based on human values, the ones that we
all take for granted. And if it's not,
we should stop and regulate it.
>> Okay. Is
AI as commonly described in the large
language models as we explore it is a
statistical grabbing of words in the
vicinity of topics and other words that
the rest of us have used in our internet
postings. So it's trying to get the best
of what is possible out of this. I don't
want to simplify it more.
>> Correct. That that is definitely
correct.
>> So what is the risk of bias in that if
it's drawing words and ideas from biased
content to begin with?
>> Well, are you saying the internet is
biased? Come on. Surely not. Um, look,
this is this is so important that people
understand this that we're so frequently
told that AI is somehow a neutral
technology. It will give us neutral
answers. But AI is of course trained on
everything that humans have ever written
that's been digitized. If it's on the
internet, you are probably right now
part of a training data set. So that's
one type of skew. So where does that
come from? The internet is not a equal
distributed representation of humanity.
Secondly, you've got people in labs who
decide to use that data to train
particular models and to optimize for
certain functions. So that's another set
of biases. And then finally,
>> but in fact, those are not so much a
bias that's rather purposeful. If you
know you're doing it, is it really a
bias? Bias is if you don't know, you're
susceptible. Well, I think at the at the
point that that Melvin Cranberg wrote
his laws of technology, he said that
technology is neither good nor bad, nor
is it neutral. And I think we can say
the same for people. When you're in a
lab, you're deciding what to optimize
for. That's a decision that is not
neutral. You're making a choice. And
then finally, this is the important bit.
Even when you finish training a system,
it then goes to system testers and what
are called reinforcement learning with
human feedback. These are commonly
people in the global south in Kenya in
Malaysia who are being paid sometimes $2
a day to review the outputs of these
models and without often being told what
the model is for. So that's another form
of like human input often with people
who are really being exploited at the
bottom of the chain. So at every step of
the way you are not looking at an
unbiased neutral system. You're looking
at a human system and values are being
built in whether you like it or not. The
question is, are we being explicit,
Eric, about what those values are?
Because I think in many cases, if your
company really is being driven by
shareholder value, the value that you're
going for is profit.
>> So, so Chris,
>> I I can I can assure you that
shareholder value is destroyed when all
humans are dead.
>> Yeah, that I think we agree on that one.
Yeah,
>> we'll see what Atlas, AI or capitalism.
So Chris, do you teach uh do you teach
uh ethics AI ethics in your class?
>> We do.
>> And how does that how does that come
out?
>> So uh there's I think this point of bias
being encoded in the training data of AI
systems is correct. Like AI is trained
off the internet. It contains plenty of
societal values. So there's interesting
research that have is applied in
frontier labs that tries to address this
to mitigate it. So it may not be a
problem that can be completely
eliminated, but it's not as if uh AI
labs are intentionally propagating bias.
They're intentionally trying to align AI
to societal values.
>> Yeah. But but societal values are not
the same in all of society.
>> Like
>> Right. But things like fairness and
safety and equity are highly uh
>> earthwide across societies, right?
Cultures. So, so where do you what is
your sense of the ethics of this and how
it would manifest going forward?
>> So, you know, it's and it's not even
just AI. It's the all of these
technologies that have come along,
including social media earlier or
privacy earlier, none of which we've
learned how to solve. All of which we've
waited for the problem to show itself
and still not have a solution. When we
come to AI, the the kind of issues and
the kind of ramifications are really
huge. You know, as the as my colleague
said, AI trained on the open internet.
When when CHBT became very popular, we
spent a whole semester asking the
questions and querying it. And at that
time, you could see how racist, how
sexist, how misogynist it was based on
the open internet. and this was what was
being made available. And so the company
ran to put a filter on it. But then that
filter is like content moderation. It
went too far and then you couldn't ask
it questions about something as simple
as viol police violence against blacks.
It would just not answer that question
as if George Floyd never happened. And
so these are the consequences that we as
a society have to experience. the harms.
Yeah, there are questions about
existential harms in the future, but
there are a lot of harms happening right
now. And it doesn't have to be that way.
In order for it not to be that way, it
depends on who this AI is servicing. And
and in particular, the design of the
technology, the decisions made in that
design, uh, is really determining what
our values really will be. our ability
to, you know, I'm driven by society has
a set of laws, a set of rules that we've
lived by. People died for these laws to
get passed and technology just ignores
them and rewrites them. And that's true
in social media. It's true in AI as
well. That is, there's so many laws and
constitutional protections. We can't
implement or protect ourselves online,
though we can in brick-and-mortar
buildings. And that's the goal of public
interest tech is to how to make that
technology accountable.
>> Yep. So Lata,
let's imagine a future where the
technology is accountable and we find
the bias and we fared it out. It's it's
it's a work in progress. As I'm sure uh
Eric would agree, it's a work in
progress. And of course, at any given
moment, you can see something that's not
quite there yet or that's biased or is
pro- Hitler or whatever. But presumably,
that's a fixable, solvable problem.
>> But this is different. We already have
laws. We already have a way that our
democracy works. We already have issues
of address issues of of bias, consumer
protection, and so forth. None of those
are enforced online.
And so the question is, can the AI at
least not break the law?
>> Eric, is is AI its own wild west?
Is it is it lawless?
>> So is it again the story you told is
correct that they rushed the stuff out,
they found all sorts of problems, and
then they're busy correcting them. I
think that's the cycle. Um, and it's
very hard. The technical thing that's
going on is models are getting trained
on more and more data. They're
developing more and more facilities.
They have so-called red cards and test
teams and so forth, but you can't test
for everything. So, at some level when
you release the model, it will do
something that will be a problem. And
they correct it fairly quickly. And the
lifetime of these models is down to
three or four months as they keep
improving improving because they're so
competitive. So, But let's go back to
hallucinations. Remember that
hallucinations
>> before you go to hallucinations, let me
just say one thing about what you just
said about this idea. Well, let's run
with the technology. And after all,
they're trying to fix it. It takes us on
the outside of the technology to tell
the people who are running the company
who could run their own experiments
what's wrong and then they're supposed
to fix it.
>> So, Eric, are these what she say?
There's an actual pro there's an actual
problem that you can't pretest some
things. But it but I disagree. I mean,
people build technologies all the time
with warranties and compliance
statements.
>> Wait, wait, wait. I got this. I got
this. I'd actually I'd actually like to
answer.
>> No, no, no, wait. Cuz cuz it's I want to
converge the answer rather than inflame
it.
Your your testers are testing for all
that they can think of. Then it gets
released into the wild where there's
vastly more testing going on than you
can possibly accomplish in in a room of
testers or even if it's distributed to
other places and then because you tested
on a thousand now a million people use
it and there's some bias that they find.
So are there mechanisms to get that
information back to you and are you
responsive to that? Uh again I I can't
speak for the specific companies but I
will tell you every single one of them
is pillariating the press over every one
of these examples and these people do
actually fix them. You need to give me
examples where the pattern has persisted
even after there's been I'll use
Facebook.
Facebook was shown to because of the way
it would this algorithm worked. It
definitely violated the uh ability for
to protect people in fair housing. That
is if you belong to a particular race or
group, you only saw ads that were at a
disadvantage. That pattern has also been
shown on Google with respect to credit
and so forth.
>> Isn't that just algorithm?
>> But I'll stick to Facebook. I'll stick
to Facebook for a second.
>> It became it became public information.
Congress responded and Facebook says,
"Hey, wait. Thank you for letting us
know that we will fix it." About a few
months later, their fix. Our our
students were able to show that the fix
was worse than the original.
And so it becomes increasingly and so in
the companies make it even more
difficult then to to do these
experiments.
>> Isn't it just the operator, not AI?
>> And you're not actually responsive to my
point. I was talking about lang large
language models. You're talking about
aphasite.
>> Well, I have examples for large language
models too in the I mean they're not a
gent you're talking about a trajectory
of technology that is not new. These
technology society clashes have been
following the same pattern and it's also
true in AI and the question is can't we
learn from what we've already seen.
>> I was trying to make an earlier point
that which I was unable to make out. It
is absolutely true that these companies
are governed by US law and if they're
violating the law, they should be held
accountable. We completely agree. Here's
the problem. A new feature emerges in
these systems that is not testable.
Tested is we don't know how to test it
>> because it's emergent.
>> It is emergent, right? We have been
through that. Now, we can stop that and
therefore stop all progress by law by
banning larger models. But as long as
you have this new emerging power with
deep reasoning, deep capabilities and
they will make mistakes, you have to be
tolerant. I went through this when I was
at Google in earlier versions of the
technology where the system would
actually do something that was wrong and
we fixed it and we fixed it as fast
because we had to because it was the
right thing to do. I don't agree that
there that these that these worlds are
ungoverned. They're governed by US law.
They're governed by shareholder
pressure. governed by uh consumer
pressure and privacy laws and so forth
and European law.
>> I know that you want to open this up to
others, but I just have to say
>> when I was the chief the CTO at the FTC,
the chief technology officer at the FTC
who's responsible for a lot of a lot a
lot of fair information practices and so
forth on the internet.
>> So when did the internet become under
the under the jurisdiction of the F Oh,
I'm thinking of the F. What am I
thinking of?
>> I don't know. Who guides the airwaves?
That's
>> Oh, the FC FCC.
>> FCC. So, FTC is Trade Commission.
Gotcha. Gotcha.
>> And the reason that they're kind of the
police department of the internet
because these are US companies.
Um, and when I worked at the FTC, it was
a lot like working in Spider-Man's den.
It was so impressive and amazing what
these people, many of them who had been
there lifelong, would do. They were
awesome. They were
>> Say
>> because they were working for the
underdog for American public with no
recognition by the American public for
what they were doing
>> and they were doing an amazing job
except the only thing about it was
everything they knew how to do was in
brick andmortar buildings. They had no
way of doing that online. That is true
today. I was at the FTC in 2014. That
was over 10 years ago and the answer is
still the same. So their ability to make
sure and test and regulate is totally
non-existent online.
>> I don't
>> I want to pivot a little bit. Kate, I
you you listed these uh travails, this
backend travails of AI and its cost to
society. Are you in a position to say if
we find energy solutions for and if we
find where we can get on the lithium we
want find some new lithium ore that's
plenty enough for all of the energy
needs or however lithium is used in our
circuitry
would you just be fine with AI are these
your because these seem like solvable
problems
>> I think that's a really great question
in my research over the you know over
many years it's really clear to me that
these questions are inter the societal
practices, the political practices and
the environmental practices. You'll see
a golden thread running through them.
One is the question of transparency
which Latana just spoke to I think very
powerfully. The second is the question
of the race. The reason that we're
seeing these models be rolled out so
quickly, as Eric rightly said, is
because there's an enormous corporate
race going on to be the first. And and
that's where you see these sorts of
problems emerge, which is that we're not
doing like what would happen in drug
discovery where you really have to test
on a diverse population before you
release, you know, a drug into the into
the
>> the patient can die. Exactly. But
frankly, AI wasn't seen as being that
mission critical even as recently as 10
years ago. Now people can die. We We are
seeing AI systems being fed directly
into the kill chain. These are systems
that are now being used in live wars as
we speak. So there is a very different
moral and legal pressure to make sure
that these systems are working. Now your
question is like okay what if they were
completely run on renewable energy and
if they you know if we made sure that
you know this this wasn't currently
right now we see that AI systems are on
track to overtake the airline industry
in terms of their carbon emissions.
Right. That's horrifying. That would
you'd think would be the first problem
that we'd want to be fixing. That is not
what's happening. At the moment, we see
that AI is using around 4% of the
world's energy. It's on track to be
using something like 25%.
By the end of this decade, we are
looking at an enormous budget of carbon
that is actually like this is a I mean,
I'm a you know, I'm a scientist. I
believe that climate change is real. I
don't think we're taking it seriously
enough. But yes, if we get to a point
where we have a diamond sphere, let's
go. Um, but we're not there yet. So, I I
I think the golden thread is how do we
think about what we build before we
build it? How do we instill those values
early on and how do we think about the
implications near and long term because
it's it's going to be the generations
after us who live the legacies of what
we built today.
>> I'm compelled to catch everybody up if
you didn't know. Kate mentioned a Dyson
sphere.
A Dyson sphere is an imagined
means of grabbing all the energy emitted
by your host star and using it in the
surface of your own civilization. Not
just a little bit of sunlight that
happens to intersect your planet, but
you build a sphere around the star,
gather all of that energy, then you can
do what the hell you want,
>> right? and and and my hope is that
generative AI and its current growth
might might actually need a Dyson sphere
>> at the projected growth rate.
>> Exactly.
>> Um I want to get back to this killchain
question with Eric in a minute because
uh Eric and I served on a board of the
Pentagon where we introduced I
legislation is not the right word. We
introduced a code of ethics for how AI
would be used in warfare uh on our side
of the United States. But before I get
there, Chris, uh, lately there's been a
lot of talk about deep fakes. I've been
deep faked many times. And, uh, you
know, some are harmless, others. I'm not
a politician, right? And I'm not
declaring war on another country. So, my
deep faiths though I don't like them,
they don't have the same consequences
that deep fakes or politicians might.
So, if deep fakes are out there and AI
is training on the internet, then AI
could be training on its own. face
and there's also AI slop I guess we call
it AI attempting to be artistic and if
that fills the space of images that are
out there and AI is training on those
images training on its own sloth so what
do you have to say about that
I'm not blaming you for it you're a
professor and you teach this stuff I
just want to know where how does that
show up
>> yeah so there's a really interesting
phenomena for uh AI which is it does
need a tremendous amount of data. So it
learns really interesting things from
the internet. So what does it learn?
Well, it learns how to use language. It
learns patterns of word association. It
learns facts about the world and it
learns uh even things like some
rudimentary common sense reasoning.
Things about physics which you wouldn't
expect it to be able to learn from text.
So all of these are uh elements that go
into a model that then later can be
adapted for useful purposes. So given
this general background knowledge that
it's gathered from large amounts of text
uh it's then subsequently
applied in a post-training phase where
you teach it to do useful things. You
teach it to be a helpful assistant. You
teach it to be a harmless assistant. you
teach it not to disclose information to
someone seeking how to manufacture drugs
or weapons. Uh or you govern its
behavior. You say, "Rather than a
general assistant, I want you to be an
encouraging tutor." So all of those
things are elements of the training
pipeline that end up with something that
actually can have value. So I think
let's not lose sight of the fact that
although this could be technology that
goes wrong, although it could be
technology that encodes bias, like we as
researchers, we as model developers, we
as users like we have a role in
determining which direction this
technology will go and I believe that it
has a tremendous amount of good to give
to the world in addition to the
potential pitfalls. Nate, how do you uh
put up guardrails for this?
Because we just we just heard a
beautifully described uh future of AI
where but uh you end up training the AI.
That's I don't want to spend my time
training a freaking robot. I'd rather
train other humans. That's just my
personal bias against that activity. But
that's effort you're investing in
something that you're then going to
interact with. I suppose that's okay.
What I'm asking is if if we don't want
the world to end, how do we think about
guardrails?
>> You know, there's a lot of dreams of AI
doing exactly what we want. There's a
lot of dreams of getting it to follow
the law. There's a lot of dreams of
getting it to act nicely. Um, but you
know, one thing I agree with Eric on,
the behavior is emergent. One thing I
agree with Eric on, you can't pretest
this stuff.
>> You can't pretest every aspect of it.
You can pretest a lot of it. You can
protest some of it, but but we're
growing these things a bit like an
organism. We cannot, you know, encode
three laws of robotics deep into these
things minds. We are not writing these
things like code.
And we can grow them. We can try, you
know, Eric mentioned, oh, there's these
people, they have these evaluation
cards, they have these safety teams.
What's going on in these safety teams?
The the two biggest places people are
putting their safety efforts are what's
called interpretability research, which
is trying to figure out what's going on
inside the AI's heads, and these
evaluations, these these model cards,
uh, which is trying to figure out how
dangerous the AIS are. If someone was
making a nuclear power plant in your
hometown and you went to them and you
said, "Hey, I hear that this uranium
stuff can have lots of energy benefits,
but also can melt down when things go
badly." What have you guys got that
makes you think you're going to get the
the benefits and not the the pitfalls?
You know, I've heard all of these
wonderful stories about the benefits.
What makes you think we're going to get
the benefits, not the pitfalls? If the
engineers say, "Oh, yeah, we've got two
crack teams working on this.
The first team's trying to figure out
what the heck is going on inside,
and the second team's trying to measure
whether it's currently exploding.
That's not a good sign.
You know, there are ways to get a handle
on this. People talk about the benefits.
People talk about the dangers and they
talk about na navigating the course. We
are not on track to get the benefits if
we make these things really smart. We
can't pretest it. We're going to roll it
out, find the problems, and then figure
them out after the fact. That's fine
when they are still a lot dumber than
the humans. That's not fine if you keep
making them smarter. And that's what
these companies are trying to do. Cindy.
>> Yeah, I'll maybe just give like a
slightly different take on this. You
said like, you know, we're trying to um
study interpretability to figure out
what's in their heads, you know, and I
think like, you know, when I think about
interpretability for me, we already know
kind of exactly what's happening with
these machines, right? There is a
mathematical equation we can write down
that's there, right? The challenge is
even though we can characterize it
mathematically, right? um as humans it's
hard to kind of understand or interpret
its reasoning but you know like in a way
that makes sense to us right but this is
just because the you know the the
underlying structure of these things is
very complex right it's multi-layered
nonlinear it doesn't map onto to human
concepts right and so I think the
interpretability
question is more just can we take like a
a mathematical equation that would take
you know a stack of paper a mile high to
um to write out. Can we explain that in
a real sense? But I don't think there is
any, you know, there is no mind there,
right? It's just kind of mapping these
huge like internal structures that are
very abstract into kind of human scale
um conceptual understandings of like the
overall reasoning.
>> Cindy, how might you program agency into
an AI?
>> Yeah, I mean
>> like like the Terminator for example.
Okay. So like yeah so
>> just to pick an example.
>> Okay. So when Neil
>> random example
>> when Neil says agency um you know what
what Raig means right is he wants to see
um the presence of some sort of like
persistent like self-motivated
um goaloriented behavior like uh
organized around the idea right that
this this AI is actually an agent in its
environment. So, not just responding to
your prompts, but also like pursuing
goals long term. And like I know that we
have a little bit of a disagreement
because I think you already believe that
there is like goal oriented behavior in
these machines which I don't see any
presence of um currently. Right. So,
>> unless we program in a goal and it sets
for itself programming, we're growing
these things.
>> We're grow Okay. So, yes, but like
growing
>> scary when you say we're growing them. I
mean, yeah, it's not a biological thing
either, right? I mean, um,
>> it's not like old school programs
either.
>> I mean, I but like I don't think it's
going to get agency by just, you know,
training on larger data sets, right? I
mean to get agency there has to be some
real engineering
things that are are happening you know
like it's it's not like a bigger neural
network or something is ever going to be
able to like pursue pursue goals
remember things like take actions like
you know I think that to to do that you
have to have like kind of like outside
scaffolding um that are real I mean that
are are different things right it's it's
not the same underlying but can you
imagine an AI with agency that satisfies
all of your concerns. That would be like
a good AI, right?
>> Well, I would first of all before we can
even get to the issue of agency, the the
issue of these emergent behaviors and
even if it is an AI with agency, this is
wrapped in a business model. And the
goal of the product is to make money for
the company, you know, and I'll go back
to social media as an example of that.
you know, we're not the customer of
Facebook. We're their product. Our eyes
on a page is what makes the money and
the advertisers are their are their
customer. And so, you can't just talk
about, oh, I'm building this AI and it's
emerging behavior and it may even get
agency. The question is, how is the
business selling it? What are they
making it available for? What are they
saying it's to be used for? If it's to
be used to help me hire people in my
company, then it needs to agree. It
needs to show it's not going to violate
the Equal Employment Act. If it's being
hired for other purposes like for credit
cards, it needs to show it's not going
to violate the Fair Credit Reporting Act
and so forth. And that's the level of
responsibility that only the designer of
those companies can do because they're
the ones making the
>> You make an important point about the
business motives. Now, if I remember
correctly, and this could just be I'm
imagining this, Eric, Google was so
large and had so much cash flow. You had
the luxury to just explore without any
particular obligation to the bottom line
of the quarterly report like alpha go.
All right, that you're going to win. You
can be the best Go player in the world.
And what's the one for chess? That was
uh
>> Stockfish.
>> No, but the Google
>> Alpha Zero.
>> Alpha Zero. Uh to to be the best chess
player. Did that have real financial
consequences or was it it just bragging
rights and that was just a fun thing for
your company to do? Well, it was
research and once we achieved those
objectives, we actually then used that
to solve a problem called protein
folding which is the single largest
discovery in biology I think in the last
50 years.
>> Alpha fold.
>> Yes. And alpha fold will result in an
enormous number of new drugs, new
treatments, new research in biomed so
forth and so on. U research has these
sort of strange things. You try one
thing, you make a discovery, you link
them together. Al alpha fold is a very
good example of the unintended effects
of people trying to play games.
>> Your your hyperbole there underscells
the fact that Alpha Fold was given the
Nobel Prize in physiology for that
discovery. So,
so, so what you're saying is that
research was these were steps towards
this greater goal that in fact was
achieved.
>> Well, I think that this narrative is is
ignoring the enormous reasons why people
are pursuing AI. So, uh let's start with
healthcare, let's start with medicine,
let's start with drug discovery. How
about energy solutions? We're going to
be cooked as a as a world unless we come
up with better energy sources. Um, one
interesting thing I'm working on data
center architectures and it now looks
like if you have enough land, solar and
battery backups are more cost effective
on a present value basis than natural
gas combined combined cycle things even
in the United States. So it shows you
that innovation happens and it can
happen in the right way as opposed to
just the wrong way which everybody else
is talking about. Uh there's certainly
secondary effects and there's no
question. But let me say once again,
this stuff is done by under US law. If
people are paying awful lot of
attention, if there's laws that are
missing, let's pass them.
>> Why not why not make
>> Can I can I
>> Hey, why not make AI solve all your
problems?
>> I'd like to interject one thing as well.
I think you know, we also talk we talk
about like the profit motive that drives
uh the creation of AI, but that's not
what drives the researchers, right?
There are venues.
>> I think that's a correct fact.
>> Thoughtful science.
>> It's just curiosity on the frontier,
right?
>> They have the luxury that someone's
paying them,
>> right? And I think Alphafold is a great
example of this, right? Like that is AI.
That is a scientific advance that's
incredibly important to human health.
And I think
>> conflating the fact that companies have
profit with uh the motive for why this
is being invented is incorrect.
Okay. Can can AI solve our AI problems?
>> I mean I I just I love this debate
because I think you know Chris touched
on something really important. Um we're
talking about you know where is this
kind of noble you know following the
path of science. We just saw meta
science checks for researchers for $100
million. That's the salary. So let's
just be clear. We are getting basketball
salaries now well beyond that. In fact
they're breaking records. These these
are not necessarily people who are doing
it out of the good of their heart. Like
this is an extremely
money field and and I think it was
originally researchers who were really
doing this just out of curiosity 10 20
years ago. We're in a different world
now. And and I guess what I'd love to
talk about is that world. I' I'd like us
not to have the AI optimism or AI
pessimism discussion, but the AI realism
discussion.
>> All right. So in that world, we're we're
going to lose a lot of cars.
>> I mean, that world people losing their
jobs now.
>> Exactly. Right. And and we're talking
about the the most significant potential
loss of jobs. And this is we're hearing
about this from the people building
these systems. You can look through all
of the new stories. People who are
running frontier AI companies are saying
that we could be looking at the loss of
75% of new jobs. That's for new
graduates. That that is a horrifying
number. And I think Latana's point is so
important here is that we do not have
the governance infrastructure or the
planning to deal with what could happen
to the labor market.
>> So we need to be more nimble in our
legislation and laws to keep pace with
this.
>> Well, we also have to be realistic. Who
is making the laws right now? I mean,
you know, Eric is talking about like we
have
let's be clear. I mean, we have never
had an administration make less interest
in regulating artificial intelligence.
We've seen we've seen regulations
produced by previous administrations
completely zeroed out and we've seen the
threat of preventing every US state from
making its own AI regulation.
>> And can I just add on to what Kate's
saying? Many times I'll say that we live
in a technocracy that our democracy has
already been changed.
>> The technology are in charge.
>> The technology designers are the new
policy makers by the arbitrary decisions
they make in the products that they
produce.
And we have no way of producing
oversight. By the time I'm a computer
scientist, my whole life has been about
how to help society get the benefits of
these technologies, more and more
apholds, but without these crazy harms
and undoing our society. And it doesn't
have to be this way. What's forcing it
this way is that the technocracies, the
technocrats who are in charge are
ignoring that responsibility.
>> Okay. Just just
There is something called the lesser of
evils because before the tech bros were
had the influence that they currently
did, you had Congress
deciding
on technology. That was disastrous cuz
they didn't know what button do I push,
you know, how do I and they're making
laws and legislations to constrain that.
So So I I I don't have a solution. I'm
just saying if tech bros are influencing
the laws, how much worse is that than
tech ignorant people who are otherwise
running Congress making the laws? So
again the right now us as researchers
who are trying to provide experiments to
these companies
government agencies who have the legal
right to pro they have the legal right
to oversee these companies can't really
do this job the person sitting in the
seat to do the job is the company itself
because they have access to the
technology they can run the same exper I
have undergraduates a team of
undergraduates who don't even get paid
and we've got to show Google and other
places what's wrong with their
technology. It's not like they're paying
us to do that. You're telling me they
can't run these same experiments?
>> All right. Uh Dave, do you have any
sense of what happens in this future
that Kate sees where uh
>> jobs are replaced and now you're making
a product but no one has money to buy it
because nobody has a job because you
replace their job with AI?
Does that make sense?
>> Yes.
>> Okay.
>> You know, uh I expect 100% employment.
Uh but I do also expect it to take the
jobs and um you'll
>> wait just to be clear
other previous advances in technology
displaced workers. No doubt about it.
But there was expansion in other
sectors. You know uh buggy whip
manufacturers went out of business but
auto repair centers came in at just
after the turn of the century. here this
pace seems faster than what society's
capacity to respond might be and so
should we be worrying about an entire
class of unemployed people who cannot
buy the services provided by the AI that
displace them
>> you know I think uh probably some people
are going to are going to want to worry
about that probably society cannot
>> especially the unemployed people yes
>> especially the unemployed people so you
know economists will tell you about how
if you manage your economy correctly and
if things are going, you know, slow
enough that people can respond and and
find new jobs, society can absorb a lot
of job loss. You know, the world used to
be something like 95% farmers and now,
you know, we can get the world running
on 2% of the population being farmers
and that does not mean 93% of people are
unemployed. AI is different. Will we or
won't we?
>> That took a century, by the way, but go
on.
>> It took a while. Yeah. But
>> will will humanity adopt to this one?
Frankly, I'm like, man,
I'm sort of looking at a different
problem here.
>> I'm just saying Eric is talking about a
threemon time scale for the advances on
these things. He's also talking about
this recursive self-improvement about
the AIs that can make smarter AIs that
can make smarter AIs, you know, and and
so my role in this is to say, hey, you
know, I'm not saying those are fake
problems. Those are real problems. Some
of those problems are starting to happen
now.
>> But, you know, kids in the schools,
autonomous weapons, real conversations
should be having,
>> but
>> but we're also racing towards super
intelligent AIs that we don't have any
idea how to point towards good stuff. A
lot of this like, oh well, you know,
they make some mistakes. We we can't
pretest it. It goes off the rails.
That's fine when it's dumber. We're
rushing towards making it smarter.
That's going to just kill us. And we all
have common cause in stopping that. The
people who are really pro- automation
and are like, "Oh, we're going to have a
universal basic income. Don't worry
about the jobs." And the people who are
like, "That's ridiculous." Uh, taking
people's jobs removes their ability to
to to like have any leverage on society.
Both those groups of people don't want
to die to a rogue super intelligence.
And so my role in this is to say, can we
all join hands on the part where we
don't die to a rogue super intelligence?
>> Okay, I think we have good agreement on
that one. Kate, what you were thinking?
What were you thinking, Kate?
>> I mean, I think part of the problem here
is is how we frame this discussion,
particularly around jobs and and
automation. In previous technologies,
you know, the weaver was replaced by the
loom or we had the horse and carriage
replaced by the car. The the deal was
that these more menial human tasks would
be removed but they would give us more
time for these intellectual and creative
challenges that people would be doing
more interesting work over time. This is
a different revolution.
>> Why?
>> And in many ways because it is coming
for creative and cognitive labor.
It is coming for the thing that in
theory automation was supposed to be
freeing time for us to do. And what
we're starting to see is that is in fact
of course white collar labor. It's the
jobs that so many of us would be like
that's what you want to do. That's what
you train to go and become a lawyer or
you go and study to be a computer
scientist. Who were some of the first
people being laid off in this wave?
Computer scientists. So what can we
automate?
>> She's got a job. No point to her.
>> Tanya will be fine.
>> She's fully employed here. But but this
is a fundamentally different question.
When cognition and creativity is being
automated, we can talk about how
effectively there's clearly a lot of
problems there. But that is the shift
that I think we have to be very serious
in talking about. I think it's easy to
make historical analogies without
looking at the fundamental nuance and
difference of what is happening here.
And it really is something that the
people in this room are going to have to
deal with. This is not a 100 years time.
>> I'd also like to give a call to action
for people in this room. I agree that
this is a much more significant problem
than the idea that a Terminator is going
to emerge and kill us all.
>> No, it's not.
>> That's a lovely
that's a, you know, speculative fiction
that's great and worth thinking ahead to
the future, but it's speculative
fiction. Uh, the job loss is something
that we all play a role in, right? So, I
think that there's there should be moral
clarity here. If you are a business
owner, if you are a business leader, if
you have influence about hiring in your
company, In the not too distant future,
you'll be faced with a question which is
AI is suddenly making everyone twice as
productive. So the question for you
>> making half as many people twice as
>> the question for you will be do I want
my company to be more productive with
the same staff or do I want to be the
same level with half the staff. Right?
And that is a question that we as a
society need to make a decision about.
And I strongly encourage you any in any
position where you have influence over
this to go to the path of growth to try
to expand what your company can do with
the same staff. Like it's you don't need
to replace people. You need to figure
out what additional tasks they can
serve.
>> So Kate about this replacement of
of positive and artistic
>> talents. Let me take a middle road
there. Middle position road there.
There were teams of artists hired to
reproduce some idea that was set down by
a set designer or by some lead artist in
a in a project. Is that much different
from the the the the schools of art that
we know about from the Renaissance where
there's the main artist who's the great
artist and then everyone comes in and
and apes that style. So my sense of this
is and I could be wrong. You have we
have expertise here to correct me that
AI today is really good at copying
what's already on the internet. I can
tell Chad GPT or or Claude I say see the
scene here paint it the way Van Go might
have seen it and it will come back with
the same color palette brush strokes and
we'll be there in these swirly patterns
and we say that's what Van Go would have
painted. Now if I then ask it paint this
scene in a style of no artist who has
ever lived
>> what's it going to do then? And so for
me, I think AI in that space is boosting
the expectations we should have of
ourselves, of our true creativity.
Because true creativity is not doing
what someone else had done just a little
better. It's doing what no one else has
done. And I don't know if AI has that
ability
if it's being trained on what everybody
else has done.
I could I could on that a little bit
here.
>> Uh training on, you know, I've also
heard uh it's just math, it's just test
text prediction. Um training on
text that humans produced does not requ
like like that is a harder problem to
predict that text than producing the
text in the first place. An example that
might resonate with people here is there
was a great astronomer Taiko Braheed
the position of of all the stars and
particularly Mars each night.
>> Planets.
>> Uh yeah, sorry. Yeah.
>> All the stars are also Mars.
>> You're standing next to the director of
the Hayden Planetarium.
>> Okay.
>> He also he also recorded the stars.
>> He also recorded the stars. But
>> but Mars is the one that Johannes Kepler
tracked.
>> Yes.
>> Right. And so Johannes Kepler, there was
a big heist. He stole the the the
journals, the data. Uh but but Tao Brahe
when he was recording the stars and
planets. Uh
he just looked at the night sky and
wrote down where Mars was. If you
imagine training something to predict
the entries in Brah's journal, it does
not have the luxury of looking at the
sky.
You can solve this problem.
You can read a bunch of the entries in
Brahe's journal and predict the next. To
do so, you have to discover the laws of
the motions of the planets. Johannes
Kepler did this. Brahi never did. So, an
AI trained on only Brahi's journals,
trained on only the prediction, only
predicting the text that Brahi produced.
That training target incentivizes the AI
to figure out something that Brahhee
never did. Now, can AIs today do this?
That's one question. But training an AI
to predict things human wrote down or
humans wrote down does not limit it to
doing only things that humans can do.
And we are starting to see AIs, you
know, we've seen chat GPT uh or GPT 5.1
produce some novel physics
contributions. We're seeing them start
to inch into ways that that humans
haven't quite done things. Uh and you
know sometimes you know when a new
scientist uh comes up with a new idea
they aren't necessarily coming up with a
wholesale entirely new way of thinking
as opposed to making small amounts of
thinking progress and using a lot of
other thinking progress that that came
from other people. You know they're
standing on the shoulders of giants. Uh
so the things we're doing with AI can
get them there. There's a different
question of whether we're currently
there but it can get them there. Okay.
And this idea that they're just math and
so it won't get there. That's like
saying, "Oh, you know, a house cat is
just biochemistry, so no sort of cat is
dangerous." You know, a house cat is
just biochemistry and a house cat can't
kill you. But that doesn't mean that
tigers are fake.
>> Wow.
>> Wait, was that a logical leap?
>> You look like you wanted to move. I I
we're running short on time here. Uh I
want to uh get back to uh a very serious
point. Uh I I was appointed under Obama
to serve on the Defense Innovation
Board, a freshly created board of the
Pentagon that was tasked with advising
the Pentagon and all the ways technology
might influence the future present and
future warfare. Not just how many
missiles are in your silo or how many
push-ups the soldiers can do. Are there
other innovations that are possible?
Eric Schmidt was the chair of that
board. I was privileged to serve under
him for this. We toured uh military
bases around the world. Spoke with uh
soldiers uh very dedicated to their
>> every single one of them wanted his
autograph.
>> I'm serious. He would show up and it's
like all this is like oh my god.
Uh so Eric, right about the time uh
drones were becoming sort of more lethal
and more real and uh we felt it
necessary as a committee, as a board to
draft some kind of rules of engagement
regarding how and when AI would be
brought into the chain of decisions that
would lead in lethal lead to lethality.
And that was a code of ethics really.
And I was quite proud of that because
there was no telling what would happen
without such guidance. My question to
you is our motivation at the time was
the knowledge that AI was not perfect
that AI would make mistakes and you
never want someone to die because of a
mistake. Yet of course people make
mistakes. So at what point is AI good
enough in that space to say I trust AI
over a person and let AI run run with
the decision and not a human. Is it just
that at the end you want to have someone
to blame because you can't blame the
computer you can't put the computer on
trial
>> but a person you can so what what
happens when AI can find the enemy
perfectly. So everybody here has an
opportunity to read the report that we
issued under DIB um
>> DIB Defense Innovation Board
>> and the Pentagon actually adopted it and
published it as their guidelines.
Roughly summarize what what our thinking
was at the time. Um
all war is terrible. All military
activity is terrible. It involves the
destruction of other people and so forth
and so on. It's just horrible. Trust me.
>> Um and the question is can If you have
to do it, can you do it using AI to be
more accurate and more successful? That
was the question. And we debated this at
some length and we concluded that at
least at the time, and I think it's true
today, AI is not reliable enough to rely
on it to make those decisions and it's
unlikely to be so for a very long time.
That was our conclusion. Uh sort of
makes sense, right? Because you can't
tolerate sort of a percentage of errors
where you might be able to tolerate them
in other areas. Um and so what we wrote
was that any use of automation in that
area uh essentially called lethality um
needed to have a couple things. It
needed to have been tested. It needed to
have a time and a space duration. And
most important there had to be a human
who was oversight overseeing it. They
had to know it was going to happen ahead
of time. They had to have tested that it
was going to happen and then it had to
reproduce. If it didn't then that was a
major problem. And that's ultimately
what the Pentagon agreed to. Um this is
a compromise because these tools are
very powerful but they also make
mistakes. So that's ultimately I think
where the Pentagon ended up and you and
is now that was some years ago now.
We're at least a decade forward from
that. Is AI good enough so that you
would have would have scripted this
differently?
>> I don't I don't believe you and I and
the others would have come to any other
conclusion on the lethality part. Uh the
lethality part is obviously the most
important one. It's the most important
decision that's made by members of the
military under military law and uh laws
of war and so forth and so on. Um what
is true is that the AI planning and
recognition that is the eyesight and the
ability to do planning is now far
superior than with humans. That's not
the same thing as lethality.
>> Okay. So these are subtle differences in
the capacity of AI that are not always
recognized or appreciated.
>> Well, I I think we we all agree that
these systems are not perfect and at
least in things which are life critical
like deciding what to do with a patient,
right? Deciding when an airplane is in
trouble, uh anything involving military
kinetic conflict, uh think twice before
you empower AI because it makes
mistakes.
I I I want to thank you for that, Eric,
and thank you for your service in that.
Uh just to land this plane, what I want
to do is go down the panel and ask, you
know, given this 3 months horizon for
new stuff that happens uh in AI and that
whole sector. Uh, let's just fast
forward way into the future to 2030
and ask
Wait, wait, wait. How long ago was Chat
GPT? How many years ago?
>> Three and a half.
>> Three and a half years ago was chat GPT
one or whatever the first public version
of it. So, let's fast forward three and
a half years to 2030.
uh what is the state of the world and
its relationship
with AI? Let's start with you.
>> Yeah. Well, I can tell you the vision
that we would like to see. Sure.
>> And and we have and you you're not just
passively hoping it'll get there. You're
actively working to achieve this,
>> right? And and and it's a future that is
really amazing in the sense that
everyone is far more productive. I can
be in multiple places at one time
because there are agents that know my
preferences and my desires and they
repres represent me in various context.
>> Just to be clear, you guys use the word
agent to refer to an AI uh
representative of your needs and
activities.
>> That's correct. And so they might
negotiate a purchase arrangement for me
or uh be in a meeting and represent my
interest. And so the my productivity is
just orders of magnitude greater than it
is today.
>> And you're fine with that?
>> I'm fine with that provided that every
single one of my uses of AI and making
me more productive is actually able to
show that it's it gives me a warranty of
what it will do and how it will
represent itself. In other words, the
same thing I get when I get an
appliance, I get a warranty and a
compliance statement. I want that same
thing on these agents and I'll be able
to do that. In some ways, what you guys
were offering to the military, there are
also many societal harms that aren't
about immediate life-threatening, but
are about our ability to have a
democracy, about our ability to
understand truthful information and so
forth that are at stake and need those
same kind of guidance. It would be like
an underwriters laboratory certification
that we have for our actual appliances
but applied to your AI agents. Okay.
>> I think
>> 2030
>> 2030 I I think we have two paths in
front of us. On one path, we move
towards a place where we see what is
already one of the most concentrated
industries in industrial in the
industrial history of of the planet
become even more concentrated and even
fewer people have their hands on levels
of power and democracies start to erode.
And these systems ultimately are making
decisions that will decide what kind of
job you can have, what kind of quality
of care you get, um what kind of options
you and your family will have going
forward. The other path is
>> just remind me is it there I'll get
these numbers long but the census is
accurate. There are four or five
companies on the S&P 500 that represent
like 30% of the total wealth
>> of the United States. Correct.
>> Some some number like that might be
precise but the census correct.
>> You're exactly right. And if you look at
the numbers of people who are actually
running these companies you're talking
about a handful a tiny handful of
billionaires. They are the people who
are getting to make these decisions
about how these tools will be used and
what problems they will be used to
solve. The other path is that we realize
this is the most important public
conversation that we can be having right
now. Which is why I'm so glad we're
gathered here today having this
discussion and we decide that actually
this is too important to be left in such
a small number of hands of people who
are not democratically accountable. you
asked what the difference was between a
representative and a and a tech row is
that we vote for one of them and we
don't vote for the other ones and
>> okay so take me to 2030
>> what I would love to see by 2030 is that
we say this matters the public interest
of these technologies is at the core and
if it is going to be using our land our
energy and our water then we need to
have a say in it and these tools should
be used for the public good and that
actually means making this something
that serves all of us, not the few.
>> Are you running for office?
>> Chris, take me to 2030. What What are
you teaching in 2030 or not teaching
today? And by the way, uh Isaac Azimov
had his three laws of robotics. Do they
apply today? And should is there an AI
laws of robotics?
>> There is an AI laws of robotics. So as
laws and robotics for the non- nerds in
the audience and I say that you are all
nerds. Thank you for being here with us
are you know firstly that a robot must
do no harm to an individual including
through inaction. Secondly that a robot
must obey what the human says and
thirdly that the robot must not cause
harm to itself barring the other two. Of
course there was a zero. So B, you mean
it can't it can't cause harm
to its first two take precedent
>> with Yes. What to accomplish the third?
It's it cannot undo the first or the
second.
>> Exactly. And then there was a later zero
law which is the robots cannot harm
humanity which is the broader thing
rather than just an individual. So this
is a key challenge for AI. It's the
alignment problem.
And just I have to say so Isaac Asmer
wrote a series of short stories that
were collected into a book called I
Robot which then became a film starring
um
>> Will Smith.
>> Will Smith and uh in there he explored
these rules of engagement for robots
very early. This is the early 1950s. So
we're talking about 70 years ago plus 75
years ago. Yeah, again this is the place
where specular friction thinks about the
future in a very useful way. So uh AI
alignment is exactly these laws. So how
do we think about making a systems that
helpful? How do we system think about
making a system that's safe? How do we
ensure that it's for the benefit of
humanity rather than to its detriment?
So as we go into 2030, I think that
those key elements that are similar to
Azimoff's laws become the key things
that guide the success or failure of the
AI endeavor.
>> And that could happen within the next
three and a half years.
>> It's it's happening now. There are many
many thoughtful researchers working on
these. You can check out the uh talk by
Amanda Ascal, the chief philosopher for
Anthropic, about how she's crafted
Claude's soul document, the thing that
governs its behavior and its code of
ethics. It's incredibly thoughtful and
it gives me hope for the future. Did
Google have a philosopher?
>> Very good.
>> Okay.
>> Uh thank you for that. So Cindy, what
you got going for us? 2030
>> 2030 a few years from now um I think
that we'll see um you know a lot of
these technologies we have now the the
kind of LLMs will become um more useful
in our day-to-day life right now you
know it can kind of respond to to
questions or to to prompts we give it
and maybe it'll become a little bit
closer to be able to um responding to
you know queries more like achieve this
objective so I can imagine if you know
specialized versions of this like you
know help you respond to emails and
these sorts of things. um feet on the
ground. I think we maybe will see uh
self-driving cars in New York City in a
few years, you know. Um I I think uh you
know that a lot of the the AI will be
trained on more specialized data sets.
So they'll become better at um certain
tasks like uh you know scientific
research or mathematical research. um
you know if you you train on kind of
certain parts of the the internet it may
be better than what it is now.
>> Quickly Kate if AI got that specialized
wouldn't that be more efficient on the
grid?
>> It it could be. Um this is one of the
really big questions. I mean right now
large scale models are being trained
using a lot of fossil fuels. A lot in
fact more than previously. So if we had
small models, if they were specialized,
there are ways forward where you can use
much less energy.
>> That's what I would have figured. And
for example, in the Jetsons, the
documentary The Jetsons of the Future
that they as as fun and creative as they
were, they did not realize that maybe
the car could one day drive itself. The
human drove it or you get a robot to
drive it, but the car is the robot. And
that's a very specific invocation of
this. So I based on the title of your
book, I'm going to show people your book
because
I got Kate's book here too, Kate.
Atlas of AI,
uh the Pub Day 2021, but it's still
happening,
>> apparently. So they just went to the the
10th edition, so it's doing all right.
>> Yeah. Okay, good. If anyone builds it,
everyone dies. This has got to be the
most depressing title ever.
It starts with if
>> adverb.
>> Well, that makes it all better. Okay.
Uh, so take me to 2030. What's going on?
Are we all dead?
You know, if you step back from this
from from the from the AI argument, you
just like look broadly at what's going
on.
The machines are talking now.
You know, it people used to think it was
going to take a while to get to the
point where the machines are talking and
we got these people being like, "Oh
yeah, we're just growing these things."
Turns out they talk. They're solving
problems we said weren't going to happen
for 50 years. Now we're trying to take
it to super intelligence. That's kind of
crazy.
>> It's kind of crazy. You know, you're
like, "Do you guys know what's going on
there?" They're like, "No, but we think
it'll be fine. Trust us." Right.
>> In Silicon Valley,
>> we're not trusting you, just so you
know.
>> Great.
>> Okay. Um,
in Silicon Valley, people are spooked.
You know, there's a joke in Silicon
Valley that when someone leaves a normal
tech job, they say, you know, I've had a
great time here and now it's on to the
next adventure. And when someone leaves
an AI tech job, they say, I have stared
into the abyss.
I'm retiring to write poetry.
Please spend time with your families.
>> Okay. And if you think I'm joking, you
can look at the resignation of the
anthropic safety lead a few weeks ago
and see how close that joke is. But the
thing is, in Washington DC, they are not
spooked. In Washington, DC, they are
thinking that this is all about
self-driving cars and autonomous weapons
and job loss. And those are important
issues. But what I hope we see in 2030
is that DC is spooked.
I hope we see a world where DC has
realized that the people running the
labs, the academics outside the labs
like Jeffrey Hinton, that the people
inside the labs that everyone who's
close to this technology is saying like,
"Oh my gosh, the super intelligence
stuff could be really dangerous." Even
Eric was saying, you know, there's red
lines and if we start to get close to
these things, we need to watch out.
>> Even Eric said that.
>> Just briefly, you you mentioned uh Dr.
Hinton. uh he's one of the fathers of of
uh of neur neural nets
>> deep learning
>> uh deep learning and uh he was given the
Nobel Prize was it in chemistry I think
>> I thought physics
>> oh phys physics thank you the Nobel
Prize in physics just a few years ago
and we were fortunate enough to get him
on to my podcast for an interview uh
it's online now uh with Jeffrey Hinton
talking about the the the original
thinkings of AI and his reflections on
where it is where it might
>> those three people did their work at
Google.
>> So,
take me to 2030. This is DC AI sensible
by 2030 cuz we're in the middle of
whoever the next president will be.
>> Yeah. So, I I think
>> presuming it won't beat Trump in a third
term. Just have to That's an if
statement. Okay.
>> I think I think they could be. You know,
we we see a lot of people in the
industry saying that this is horribly
riskier than any other technology we
work with. And we see people at these
labs saying, you know, I think there's a
good chance this kills everybody, but I
got to do it because I'm better than the
next guy. And, you know, maybe one of
them's right. Probably two of them
aren't. But, uh, in in 2030, if if we
keep having this conversation, if it
becomes more salient, I think we see uh
people around the world realizing that
we shouldn't do this super intelligence
stuff. And the world does not need to
look that different from today. You
don't need to stop the self-driving
cars. You don't need to stop Alphaold
doing the drug discovery. You don't even
need to stop the autonomous weapons.
Society can have a debate about, you
know, do we want to to save people on
the front lines? Do we want to, you
know, not do this, not go down this
path. But uh but we could stop just the
race to build smarter and smarter AIs
that nobody understands that would be
easier than nuclear arms control. And I
hope that in 2030 leaders around the
world will realize that this is lethally
dangerous and we'll be putting a treaty
in place to say we're not rushing to
smarter than human AIs with anything
remotely like the modern technology or
we have no idea what's going on there.
>> All right, Eric,
>> you asked a question about 2030.
>> Yes, I did.
>> Why do we bet there? Why do we bet that
American democracy will survive?
>> Okay, so that's an important reason.
>> So I'm I'm gonna make a bet that
American democracy will actually
survive. We'll get through whatever the
current problems are, of which there
have been many listed here. That the
excesses will be addressed through
legislation. I know that's hard to
imagine right now, right? But I'm going
to bet that democracy will ultimately
reflect the will of the voters, which is
how it's supposed to work. So in that
scenario, which I know is difficult to
imagine at the moment, uh let's think
about some of the other benefits of all
of this. The likelihood of students
being much better educated because of
their ability to be tutored in a using
AI and specialized way is enormous. The
ability to have the next generation,
your kids and grandchildren, being so
much smarter than we will ever be is
pretty high. Furthermore, this issue
about jobs, there's no question there's
going to be job dislocation. But last
time I checked, every one of those
predictions has been false for the
reasons Neil mentioned earlier, which is
the American economy is fairly efficient
recycling jobs. And there's definitely
harm. There's definitely people hurt.
But at the end of the day, people
working with computers today make more
money than people without. Furthermore,
let's bet that these other issues, the
regulatory issues, the techro issues are
resolved through some combination of
self-interest, shareholder values, legal
liability, and regulation. In that
model, in 2030, every one of you will
have an assistant, a savant that allows
you under your control, not spying on
you, not doing something bad, under your
control to do the things that you most
care about. You're a physicist, right?
Neil's a physicist. You have a
physician's assistant who overnight does
your thinking while you're busy sleeping
and tells you what is interesting in the
day and you love it and you can turn it
off if you don't like it. There's
example after example. The most likely
scenario by 2030 is something analogous
to what I've described. All of the
things that everybody's worried about
are true at some level or they're point
issues that have to be debated. In my
book, which you did not feature,
I say very very clearly.
>> Tell us the name of the book.
>> Genesis.
>> Thank you.
>> Including with myself and Henry Crew and
a few others.
>> Um was co-author.
>> Yes. Yes.
>> When did you write? He'd been dead for
when did
>> Okay.
The AI Henry Kissinger. Okay. Wow.
>> In fairness to Henry, he he finished the
book The Dable Read
>> and I encourage you to read it. Uh his
last book of his many many great books.
The important thing thinking about this
is really important and our work. What
we said is this conversation needs to
occur over and over again because it is
the future of us. It is us. It is how we
operate and I endorse that. Uh but at
the end of the day, right, we will get
through this and more importantly, we
will thrive. Do you know why? Because at
least American capitalism, the way it's
structured with regulation, produces
enormous economic and life value for us.
Think about it. Solving climate change,
solving the various issues we have, and
remember, we're depopulated, especially
because we're not allowing immigrants
in, which is an error, right? So, you
get the idea.
>> Wow. Let me see if I can put a ball on
this.
So
>> I try to always
generate a cosmic perspective on all
that happens to me in a day and to get
this kind of exposure. I don't know how
to tie this up in a bow, all this in
one, but allow me to say that and I've
been in computing for most of my life
that I've had access to it and awareness
of it and and I remember, you know, the
Turing test. People said if you interact
with a computer and you don't know if
there's a human on the other side then
it you why distinguish it from being a
computer or a human that passed the
touring test. The original research
paper was called the imitation game
which became the title of the movie
featuring the life of Alan Turing which
I recommend. Uh so we're way past that.
All right. And so I remember thinking,
"Oh, we'll never have a computer as good
as people at chess." We're way past
that.
I remembered when Siri showed up on the
iPhone. There were whole commercials
with like who was the movie director who
was interacting with it? Martin Scorsese
is talking to Siri and while our jaws
are dropped open, there's this
intelligent agent serving him in the
voice of Siri in his iPhone. And I
think, "Wow, the future is here.
Today, no one talks about Siri as AI.
We're so past this. No, that's just old.
That's not even interesting. Alph Go,
that's yesterday.
Uh Alpha Zero, yesterday.
So, what intrigues me is every one of
these incremental advances in AI, we
absorb just becomes part of life and
we're looking at that next frontier.
Yes. the AGI, the the artificial general
intelligence. That's a frontier that may
be dangerous, but all the rest we
absorbed.
We think nothing of putting on GPS and
having it tell us the shortest route to
grandma's house, corrected for traffic,
and there's no human in that loop. We're
all fine with that.
We're not spooked by it. You're not
writing to your congressman about it.
You don't try to legislate it. It's just
there in our lives.
So I foresee by 2030, if I will, much
more AI just folded into our lives.
But the AI that transcends all of our
intelligence, the one that you might
think is farther away than it might be
given the title of this book.
>> Uh, I have to
I have to agree with Nate. I'm old
enough to have lived through a lot of
the Cold War. You know what kind of
ended the Cold War? A lot of things
contributed. But you know what mattered?
The realization that if you launch
nuclear weapons, intercontinental
ballistic missiles from the Soviet Union
into the Western Hemisphere and we see
that and we retaliate,
everybody dies.
We had an acronym for it. Matt, mutual
assured destruction. That's what brought
people to the table. Said this is
insane. We have to regulate it. We have
to reduce the stockp because preserving
our species is what is what matters
here.
So I based on that evidence, I have some
confidence
that as those dangers become more and
more real,
people will come to the table and say,
"Yeah, keep the rest of the AI going, we
got new medicines and new understandings
of our physiology and new technologies
that help us get smarter and healthier."
Yes, but that branch of AI is lethal. We
have to do something about that. No one
should build it and everyone needs to
agree to that by treaty. Treaties are
not perfect, but they're the best we
have as humans.
I would end by saying there's a quote
from Ray Bradberry who I verified with.
I met him only once and I just have to
verify this quote. He said there's a
woman once came up to me and said
science fiction author Ray Bradberry uh
Mr. Bradberry, why do you write these
stories about these apocalyptic futures?
Is this the future you think humanity is
going to inherit? He says, "No, I write
these apocalyptic futures so you know to
avoid them."
That is a cosmic perspective. Join me in
thanking our panel. Oh my god.
Excuse
me.
We moved along, but as as always, we
allocate a few minutes for audience
questions. So, if we can bring the
lights up, we have a microphone in each
aisle. We'll try to go quickly