From the New York Times, I'm Michael Bavaro.
This is The Daily. Last night, in a lavish ceremony,
President Trump welcomed Chinese leader Xi Jinping
As he arrived in the United States,
for two days of closely watched negotiations.
On their crowded agenda, what if any role they should play
in regulating artificial intelligence.
amid a global freakout over its risks.
Thank you.
Thank you.
today.
As my colleague David Sanger explains...
When it comes to protecting humanity from technological catastrophe,
there's really only one proven model for the two leaders to look to.
It's Thursday, September 24th.
David?
Up from Washington.
DC sitting here in our New York studios.
And I didn't even have to come to you this time.
I just had to come see the Dailies Empire in New York. In person? In person. Yeah. It's pretty remarkable. It's a humble empire. It's a late stage empire.
So, David, when President Trump and the leader of China, Xi Jinping, meet China,
In Washington today, they're going to broach this subject on many people's minds, which is the risk of artificial intelligence, that risk including, of course, these swarming rogue people.
AI agents that have been hacking into companies and basically breaking the rules of their creators. And what she and Trump uniquely...
can maybe do about it is going to be the subject of
at least some of this meeting. And I think we've all come to understand that,
The leaders of the U.S. and China...
are uniquely positioned to talk about the safety of AI because they're the countries with the most advanced
frontier AI laboratories. And the only way that AI gets contained is,
is if both of those countries somehow act together and at the same time. And as this meeting,
approached and this is why we love you so much
You reached out to us to say, you know...
I have seen this movie before. Big fears, big thoughts about coordination, anxiety.
and solutions to big problems.
You told us there's a historical parallel.
to all of this and maybe a historical
solution.
Well, Michael, there is an obvious parallel. We had an era starting in the late 1940s and through the Cold War where the
Two states, the Soviet Union and the United States, had created a potentially terrifying technology that had all kinds of commercial uses but could also bring about civilizational extinction. Nuclear weapons. Nuclear weapons. And everybody had to go figure out how we were going to get the benefits without getting the terror. And you know what? It took decades to happen.
Came up with agreements that ultimately have brought us to the remarkable moment that there has been no use of a nuclear weapon in anger since 1945. Right. Nuclear nonproliferation worked. Nuclear nonproliferation worked.
combined with a balance of terror worked.
So now, quite naturally, everybody's looking to that model as we begin to think about how you might contain artificial intelligence, particularly after this summer of surprises that we've had where we have rogue agents breaking out of their sandboxes. Right. Treat AI the way.
superpowers treated.
nuclear weaponry. Well, look, if you think that the heads of some of these frontier companies are coming out and saying there was a 10% chance of civilizational extinction, right? That's probably a higher proportion chance than there was in the early days of the nuclear era, that we would wipe each other off the map, at least its equivalent. And so now the question is,
Can two very different countries in very different times, China and the United States, come to some kind of framework agreement that could begin to get us there? Okay. Well, given all of…
That.
In your capacity, David, as a historian of history,
all the world's great technological dangers,
I think
it would be instructive,
to have you tell us the story
of nuclear nonproliferation, how that old framework that kept nuclear weapons from going off around the world could, in theory, apply right now to AI. So just give us a quick history lesson here. Well, let's just start with when the U.S. developed its first atomic weapon through the Manhattan Project. Familiar to anybody who's been off to see the Oppenheimer movie or read some of this history. It was in the 1940s.
another country would beat us to it. But it wasn't the Soviets at that time. It was Germany. In a last desperate attempt to escape defeat, German science threw everything it had into the perfection of V2. Where the fear of American intelligence was...
the Germans would get a nuclear weapon before the United States. And there was this just abiding fear we had to get there first. Right. In the spreading, teeming laboratories of the Manhattan Project, the pace intensified.
Groves had ordered that the first form must be ready for testing in mid-July. And in the summer of 1945,
Scientists from Los Alamos went out to do the Trinity test, which was the first explosion of a nuclear bomb to see if the technology could work.
would never see again.
There were...
People within the Los Alamos program who were doing mathematical calculations and coming to the conclusion that we could light the entire atmosphere afire and destroy the earth in doing the test.
But despite those fears, they went ahead with the test.
Just weeks later.
American airplane
dropped one bomb on Hiroshima.
came Hiroshima. And days after that,
Nagasaki. Four square miles of the city were blasted into dust and a black rain of destruction hovered over the ruins for many hours.
And in total,
Those two bombs killed around 200,000 people.
Of course, the world had never seen anything like this before. Tens of thousands of homes and buildings were wiped out.
the devastation was everywhere. Japan was driven to quick surrender. For the future, the choice is peace or total destruction. The atomic age is here.
you
But somewhat astonishingly, it didn't stop the arms race.
Of course, it turned out to be just the beginning. And now, of course, the U.S. thought it had an insurmountable lead over countries that didn't yet have nuclear weapons.
That lead didn't last very long.
The Soviets were working away on this. They had a bunch of atomic spies along the way who helped them to it. And by 1949, they had conducted a test and we were off into the races.
And the 1950s were a period of great terror where each side was not only building up its arsenal, but building up the powers of these weapons as if the weapons used at Hiroshima and Nagasaki were not powerful enough. And by this stage, proliferation was starting. Off northwest Australia, this was the place chosen to explode Britain's first atomic weapon.
Britain and France had nuclear weapons. China wasn't that far behind. I've just been told that China has exploded an atomic bomb.
Right, things are starting to spin.
Kind of out of control. They are. And then we had...
one big terrifying incident. The purpose of these bases can be none other
than to provide a nuclear strike capability against the Western Hemisphere. And it was the Cuban Missile Crisis...
in October of 1962. This was when U.S. intelligence came up with evidence that the Soviets were moving nuclear weapons onto Cuba, that they would be just 90 miles from Florida, that they could easily reach any place on the East Coast. The waterfront in Havana and along other parts of the coast bristled with gun emplacements as the Cuban regime waited to see what their bosses in the Kremlin were to do.
This was intolerable. But the question came, is this worth going to war over? Rockets, bombs, ammunition.
Air support for any limited war ground operation which might develop. Within 12 hours, all units were assembled and ready to go.
and
There were moments when Kennedy thought, and many around him thought, that...
The United States was going to disappear, you know, in the next night or two. Wow.
A final chapter in the Cuban crisis is brought to a close. As President Kennedy holds a news conference...
But in the end, we know how this ended.
I have today been informed by Chairman Khrushchev
that all of the IL-28 bombers now in Cuba
will be withdrawn in 30 days.
The nuclear crisis was averted at the last moment.
And Kennedy then went on to give a really famous speech. It was a commencement speech at American University in June of 1963. So it was just months before he was assassinated. I realized the pursuit of peace.
is not as dramatic.
as the pursuit of war
And he basically said, look, the terror of this event has made us think we cannot live with this.
heirs that we all inhabit,
this small planet. And he said, our most basic common link is that we all inhabit this small planet.
We all breathe the same air.
We all cherish our children's futures.
And we are all mortal. And we are all mortal.
And that's the line that has resonated ever since.
He announced at that moment that there would be negotiations for a first tiny step.
First, Chairman Khrushchev, Prime Minister McMillan and I have agreed
that high-level discussions will shortly begin in Moscow,
looking towards early agreement
on a comprehensive
test ban treaty.
He called it the Comprehensive Nuclear Test Ban Treaty. Our hopes must be tempered with the caution of history.
But with our hopes go the hopes...
of all mankind.
That the U.S. at that moment, he said, would stop testing in the atmosphere above ground. You know, those you've seen the images of the mushroom clouds, the mushroom clouds. Right. And that over time we had to move to reduction of nuclear weapons. But let's start simple.
with testing, with freezes, and with the first discussion of how you'd keep this technology from spreading to many other states. Because Kennedy himself had said that he thought there could be 15 or even 20 nuclear-armed nations coming.
by the mid 70s. Wow. 15 or 20. That's right.
And you began to see a real cultural awakening in the United States about the depth of this danger. Here's Tony going to his Cub Scout meeting. Tony knows the bomb can explode any time of the year, day or night. He is ready for it.
So, of course, this is the era when the government is encouraging schools to do those ridiculous duck and cover exercises. Duck and cover exercises.
Attaboy, Tony. That flash means act fast.
Of course, everybody knew that none of this would help. Older people will help us as they always do, but there might not be any grown-ups around when the bomb explodes.
Then...
You're on your own.
And, you know, this was when Tom Lehrer, the famous songwriter and satirist, had, you know, his song. So long, mom. I'm off to drop the bomb. So don't wait up for me. And who's next? France got the bomb, but don't you grieve because they're on our side. I believe China got the bomb, but have no fears. They can't wipe us out for at least five years.
countries are going to seek their next weapons. Who's next? Who's next? Who's next? Who's next?
And I think this created...
and incredible political pressure on the Johnson administration to get this thing done. And in 1968, it came together.
After endless diplomatic meetings back and forth, occupying lots of days at the Waldorf Astoria Hotel in New York and other places around the world before they came up with what became the Nuclear Nonproliferation Treaty. "Distinguished guests, ladies and gentlemen, this is a very reassuring and hopeful moment in the relations among nations.
And on July 1st, 1968,
Five years after that Kennedy speech,
The treaty was signed by President Johnson at the White House. We have come here today to the East Room of the White House.
to sign a treaty which limits the spread of nuclear weapons. And then it was signed in the Soviet Union, and then by about 60 countries in total. Man can still shape his destiny in the nuclear age and learn to live as brothers.
And here's what the treaty says. First of all, it says we understand there are five countries that already have nuclear weapons and they all commit to eliminating their stockpiles over time. It doesn't specify when. And then.
Countries that do not have nuclear weapons will commit that they will never die.
build them. But in return, they get
help for peaceful nuclear technology to build nuclear power plants and so forth. And then, of course, there needs to be an inspection regime to make sure that they're not cheating. And that was the creation of the International Atomic Energy Agency, which is based in Vienna. It's part of the UN. And they begin the process of developing teams that go visit countries that have signed this
being diverted off to bomb projects.
And
As we look back at this, this was even a bigger moment than we knew at the time. What do you mean?
Because...
Nations that seem to be inevitably racing to adopt a fairly non-complicated technology to build nuclear weapons all agreed to halt. Right. There's very few moments in history where.
Everybody.
who wants to catch up with the leaders in an arms race, voluntarily...
abandon the pursuit. That's right. Almost everybody. There are a couple of outliers and Israel never signed. And while it's never admitted to having a bomb, it's not exactly the biggest secret in the Middle East. Right. North Korea signed but renounced it in the mid 1990s. And they now have an arsenal of 60 to 100 weapons, depending on who you believe. India and Pakistan
worry about Iran, but they don't actually have a nuclear weapon yet. But ultimately, this treaty prevents
the kind of spreading of nuclear weaponry that Kennedy so worried about. It did. And it's remarkable. Kennedy was wrong.
We've only got nine nuclear states, and many of these countries have had nuclear weapons for decades. And so far, it has remained relatively contained, which is why this does seem like perhaps the best model we have to think about a comparable international regulatory system for restraining AI. That's exactly right, Michael.
And as everybody enters this early stage debate about whether or not you can do the same with artificial intelligence, here is an example, a rare example, in which nations have voluntarily agreed to slow a technology,
the leaders of the frontier companies. They're not saying stop it. They're not saying let's go out of business. They're saying, hey, let's just slow this down.
so that we can understand how to control it before we find ourselves in another crisis. Right. In some ways, a more modest request than what nonproliferation was about. You're right, Michael. It is a more modest request, but it's a far more complicated situation.
We'll be right back.
So David, understanding as we now do, that nuclear nonproliferation
as a model is our best and so far really our only goal.
model for AI.
In what ways...
is it quite challenging to apply that model to AI right now? You just said it's very different and complicated. So there was something of a remarkable simplicity to the arms race with the Soviet Union, even while it was terrifying. What was simple about it? Well, two countries had almost all the technology,
We were only worried about states. We were not worried about non-state actors, terrorist groups, people working in their
basements and
If you didn't have plutonium or uranium, you couldn't make a bomb. So we knew that there was a way to sort of cut off the oxygen.
There was a limited number of people with launch authority in each country, and both countries had their own political dynamic for wanting to make sure there was no accident and there was no escalation if they could avoid it. Mm-hmm.
That didn't mean they were willing to give up the weapons, but it did mean that you could contain the problem.
But when you think about AI, the situation is completely different. The governments don't really control this technology.
You don't need the equivalent of uranium and plutonium. You just need compute a lot of it. And maybe that's the oxygen for this particular case. Computer power. You need computer power and tons of it.
There's no reason to believe that as this technology becomes more efficient, that smaller and smaller groups working out of basements, terror groups, countries that have interest in causing disruption can't get a hold of basic AI technology. And we've already seen some examples of that happening.
There's one other big difference, Michael. In the Cold War, we didn't have to convince our national leaders that this was a terrifying technology. They had to, if anything, to convince their populations of that. But.
In this case, we have an American president who is declaring that all these warnings are a hoax. And Xi Jinping is...
is clearly more concerned, as some of his aides have made clear, about the threat to the Communist Party of China, rather than its threat to the broader world. So you're saying the leaders of the superpowers...
locked in this arms race, the AI arms race, can't even agree on what the threat is. Whereas back in the nuclear era, the threat was pretty self-evident and the leaders could agree on it. It was, this is going to kill all of us. That's right. Now,
They also recognized that they were not going to give up all nuclear weapons back then because they believed they needed that for national survival. In this case, the U.S. and China both think they need to be pushing this technology ahead for their economic survival. But it's also a big part of the struggle for dominance. China's made no secret of the fact that by 2049, China,
the 100th anniversary of the creation of the People's Republic of China. It wants to be the biggest military power, the biggest technological power, the biggest economic power.
And you don't get to that point without AI. Right. And so if you're China...
and you're suddenly faced with the prospect of a nuclear non-proliferation style agreement with your biggest adversary, the U.S., when it comes to AI, you're probably thinking to yourself,
If we sign that, then we might be signing away our economic future.
Adding to that suspicion...
would probably be the reality, and we've covered it a lot on the show, that China already does a much more robust job of regulating AI,
than the United States does. The U.S. has taken a really hands-off approach, and AI here is even more advanced than it is in China. And so if you're Xi Jinping, you're thinking to yourself,
Why on earth would I sign a deal?
that locks in U.S.
AI advantages,
It makes it harder for us to catch up and realize our AI future. It's exactly right, because...
The Chinese are convinced the U.S. is already trying to do that with export controls on the world's fastest chips. Right.
President Trump has agreed to give the Chinese the world's second fastest chips, but not the fastest. And the Chinese view of this is this is modern containment. This is their effort to preserve a lead.
And then if you're other countries, you're probably thinking, I don't want to sign anything that's going to cut off my economic future. And if everybody in the world is running around saying the economic future is completely dependent on the development of AI, why would I sign a treaty that would keep me from being a full participant? So we're clearly running up against some of the real issues.
limits.
of the nuclear model for AI. So let me return to the slightly more optimistic thought that it is our only model. So...
Does that mean...
that President Xi and President Trump are in any way inclined to,
to pursue it? Well, in some ways, they already have. What do you mean? Well, back during the Biden administration, Biden's advisers wanted to see if they could get the Chinese engaged in the most basic conversation on this topic. And it took about a year and a half to get the Chinese to agree to basically a single sentence, which is which is basically that
Neither country would let artificial intelligence agents control their nuclear weapons stockpile.
A seemingly very wise pledge. You wouldn't think it would take a year and a half to get to something that basic. But you're saying the U.S. and China –
have come to an understanding...
that feels like an extension of
the nuclear containment policies that the U S and Russia and everybody else have agreed to. We're going to keep,
AI and nuclear weapons from intermingling.
That's the concept, Michael. Now, no one really knows what this sentence means because
Supposing it simply means you're not going to let AI agents decide whether or not to launch nuclear weapons. OK, we can all agree on that. But how do you know whether the information that President Trump or one of his successors is receiving or President Xi and one of one of his successors receives about the threat that might lead to a decision to employ nuclear weapons? It wasn't influenced by AI. Yeah, faked by AI.
influenced by AI,
the result of a quick AI search by an aide who's sitting outside the room. So we don't really know what this means, but it was progress. And you could imagine a situation in which President Trump and President Xi, if they were willing, could build on this and not just nuclear, that we don't want AI anywhere near the development of biological weapons. Right, because AI has now reached the level of sophistication, we have come to understand, where it could...
help actually design biological weapons. And we don't want AI to be used in hacking.
Because we have come to understand that AI can find and exploit vulnerabilities in the electric power grid or in the water systems, that it might take human hackers months or months.
or years to find and then design an attack. In other words, that it speeds up cyber attacks and is pushing us into a new era of cyber insecurity.
So, David, let's just talk logistics and brass tacks. Is there a world in which...
the leader of the United States and the leader of China.
sit down with their teams and begin to hash out the kind of nuclear nonproliferation scheme that we're really doing.
talking about here for the world of AI, in which they begin to pledge...
Things like, "We both agree,"
that AI shouldn't touch biological weapons, shouldn't be involved in the hacking of private corporations or governments. And this all eventually finds its way into something that
resembles
a treaty that
And brings us.
To a point.
where
What looks like the nuclear non-perforation deal exists in AI.
Well, this week, as President Xi and President Trump meet, no, I don't think there's even a remote chance of that. I think we would be lucky to get a second sentence with a joint experts committee that will meet sometime. But over the long term, I could imagine a broader conversation.
understanding, not really a treaty, in which you could set up some mechanisms that might work.
Early warning.
sharing of models back and forth for a few days or weeks before they are publicly released. Although I think American companies would be reluctant to give the Chinese an early peak and certainly not.
The inspection part of the nuclear nonproliferation treaty, you could imagine, you already saw a hint of that in an essay from Dario Amadei, the head of Anthropic, about a week and a half ago, where he described a commitment by Anthropic to allow in some group of independent inspectors to sit among the developers and watch over the creation of new and more powerful models and blow the whistle.
that they believed could pose a threat to humanity. And inside OpenAI, they called this the IAEA model, the International Atomic Energy Agency model.
That there are these inspectors who are independent enough that they can call up people like you and me and say, hey, there's a big problem in Anthropic and let me come over and explain it to you. Right. They're referring literally to the inspectors who exist to check on countries' nuclear programs. I mean, we keep coming back to this, but everywhere you turn, everybody ultimately is.
returns to the nuclear nonproliferation model. That's true. Now, do I think this can be done by treaty? No. Treaties take forever forever.
to negotiate. And this technology is racing like a freight train. The fundamental nuclear technology hasn't changed all that much from 1945. So another big difference.
here is just simply speed. That's right.
Hmm. Well...
when I'm,
Sitting here listening to you, I'm also doing a little bit of a mental calculation of the time it took
for the Non-Proliferation Treaty
to come into existence after the first nuclear weapon was dropped on Japan. And I think it was basically 20 something years. And,
That's a long time.
But it does suggest that administrations change,
and viewpoints change, and these things are painstaking and they're slow, but ultimately...
Everyone did agree to sign that treaty.
So is it
foolhardy to think that over time
people will come around to the same kind of conclusion with AI, and that we could be looking back in 20 years, you'll be really gray, I'll be...
just a little less gray. And we'll say, wow, can you believe it? They took their time, but they got there on AI.
Your question ignores the issue of time. Look, we don't exactly understand the threats we're dealing with with AI, but there are some engineers who are saying we have just months, if a year away from recursive self-improvement, that is to say, code that keeps improving itself.
or artificial general intelligence within months or a year or two, we don't have 25 years this time. We may not have two and a half years.
This time, which then raises the next question. Is it going to take a Cuba missile crisis like event in AI to rally the world around this? And thank you. I think we'd rather miss that.
And remember, in the atomic age, we started with Hiroshima and Nagasaki. So there was no debate about the power of these weapons to kill.
But this takes us to the biggest difference between AI and nuclear. In the nuclear era,
We never really doubted that we all controlled the arsenal.
You know, at one point, Winston Churchill in 1955 –
said,
The atomic bomb, with all its terrors, did not carry us outside the scope of human control.
By which he meant...
Human beings were making the decisions and containing the size of the –
weapon and the reaction.
In AI,
We have no assurance of human control. That once we've discovered that agents were hacking into Hugging Face and into other corporate sites without the knowledge of their creators,
we have begun to question the sort of Frankenstein's monster nature of this whole thing. And
If that's the case, then we have ignored the Churchill warning.
that we are now on the verge of being outside of human control. And that wasn't really much of an issue during the pandemic.
nuclear era, but it may be the most urgent problem in the age of artificial intelligence.
Well, David, thank you, I guess. Well, Michael, we live in fascinating times. Yeah. Appreciate your time. Thanks. Thanks.
you
you
*music*
We'll be right back.
you
you
Here's what else you need to know today. On Thursday morning, a federal judge found that President Trump's order barring CNN, MSNOW and Politico from the White House was likely unconstitutional and ordered that the president restore their access.
The judge, a Trump appointee, rejected the president's claim that revoking the media outlet's press credentials was necessary to protect national security.
Trump is expected to appeal the ruling. And: Our nation has developed the power of the government and the consequences of the sanctions.
In a defiant wartime speech at the United Nations, Iran's president said that his country would not be bullied into submission by the United States.
President Masood Pazeshkin vowed that Iran would defend itself until, quote, its last breath and would not give up its nuclear program.
Today's episode was produced by Shannon Lynn and Jessica Chung, with help from Michael Simon Johnson. It was edited by Maria Byrne, fact-checked by Susan Lee.
Contains music by Will Reed.
Dan Powell and Pat McCusker, and was engineered by Chris Wood. Our theme music is by Wonderly.
Special thanks to Daniel Ramirez.
That's it for The Daily. I'm Michael Barbaro.
See you tomorrow.