Anthropic CEO reacts to 'AI could kill us all' warning

The warnings from inside Anthropic

00:00One of the world leaders in artificial intelligence is warning the industry must slow down now. Anthropic CEO Daario Amade writes, "With AI advancing so quickly, there is a risk of humanity losing control of the technology, possibly leading to cyber attacks and bioteterrorism." Amade spoke to CNN exclusively today with our Anderson Cooper. Here's a part of Anderson's interview. Obviously, the reason we're talking is these headlines that have come out. Jac Jacob Coxin who recently resigned from anthropic made headlines last week saying the people building AI earnestly believe that it could kill us all by the end of the decade. There's a current employee Evan Hubb Hubbinger an alignment science lead. He agreed saying we really do earnestly

00:44believe AI could kill all humans. I personally think it is greater than 10% chance within the next decade. Do you believe earnestly believe that AI could kill all humans? And and what percent chance do you think it is? So, you know, I I you know, it's it's funny. I I agree with Jacob much more than I disagree with him, you know, and and and and it's it's you know, it's it's an interesting resignation because when he left, he said, you know, I think Anthropic is the most responsible player, right? I think Anthropic is the most aware of these issues. He wasn't calling out us. He was calling out the dynamic of the the industry as a whole moving too fast. Now, now I I you know, I I I think I already said, you know, these sort of unconditional probabilities,

Amodei on probabilities and paths

01:33you know, I I you know, I used to kind of talk in that way cuz it's kind of an easy way to express I think there's some chance that things go wrong, but you know, but you know, but but but it's not it's not all that high. But again, I think it's more illuminating to think in terms of, you know, how do you decompose those probabilities? What are the paths in which things go well and what are the paths in which things go poorly? And so instead of saying, you know, it's a 10% chance that sounds like it's a roll of the dice, I think it's much more illuminating to say, well, that forks into different paths and if we take the right paths, then the chance of something going wrong is very low. If we take the wrong paths, then the chance of something going wrong could be could be even

02:16higher than that. And so let's focus on our agency and our ability to take the right paths instead of the wrong paths. And I think that the way to take the right paths is for the industry to work together. Again, you know, we'll we'll talk about uh uh the proposal that I've put out today, but at the end of the day, the industry has to work together and it's harder, but to some extent, the world has to work together. If we go too slow, I still believe that the wrong people will be in charge of the technology and and and you know, that again will bring the probability of things going wrong very high. So what you're proposing is called pacing the frontier of of the AI industry. And you write we must slow the pace at which we improve the capabilities of AI models.

Pacing the frontier proposal

03:00Progress will seem fast and we must make wise use of the time we gain. And there's three parts to this. One is embedded evaluators. Part two is democratic coordination. And part three is global coordination. Sam Alman at OpenAI just endorsed the idea of embedded evaluators. Essentially you are suggesting I don't know if are they observers inside your company and these other AI companies ideally globally but certainly in the US who would monitor it they would have employee access what power would they have? Yeah. So, so be before I even get to that, I'm I'm I'm just I'm very grateful to see um you know, support for this proposal just in the first few hours of having put it out by by Sam Alman, by Elon Musk, by uh you know, various legislators and and government

Embedded evaluators and government's role

03:51officials. I think it's a positive sign about, you know, our ability to make something that looks like this plan work. But but to get to your question um yeah embedded evaluators so there is precedent for this um if you look at how banks are regulated right there's there's all kinds of you know there's all kinds of rules about you know how banks are allowed to do their business right because you know we we don't want to have financial crisis and one of the things that are done in some cases is I think they're called supervisors in the financial industry but they're they're folks who are independent of a company who are embedded in the company daytoday to check their practices. Uh and I think we can do the same thing in I think we can do the same thing in AI.

04:36The other thing you are calling for in this three-part proposal is is democratic coordination which is essentially democracy all the companies that are under democracies having these evaluators having this coordination and and probably the most important because the argument is well if the US or companies are the only ones regulating themselves what about does China get ahead you're calling for global coordination that's obviously a lot more difficult yes yes I you know in this three-step proposal. Each step is more difficult than than the one before. And Enthropic was kind of getting the ball rolling by committing to the first step, but we shouldn't kid ourselves that these next two steps are going to be more difficult and and we may not succeed. And and so

05:23my my proposal was a proposal that we should try. Now again from all the agreement from others that I've seen in just the first few hours of putting out the essay that creates some optimism but that needs to follow up with results. So yes, pacing among dem companies in democratic countries. Now the one of the important things that that I need to call for here is you know as we get to the pace of model releases right that's commercial activity and in order for companies to coordinate directly on commercial activity they need to do so working with the government and so the one thing that we're asking for from government right now right I mean there's also this question of what should the regulations be but there's something even simpler that the government can do right now

Kokotajlo on how extinction could happen

06:13which is it can sit down in the room and with all the industry players and we can all talk together and the government can make sure that we're you know we're we're not doing anything nefarious from the perspective of antitrust and collusion. We're just trying to make our system safe and to work together on making the system safe. And so I think that's an important role for the government to have. Jacob Coxin joins us now. Um, Jacob, I appreciate you you being with us. Um, first of all, I assume you are saying literally that you believe uh that AI could kill us all by the end of the decade. How would that happen? Like what would that actually look like? Yeah. So, the precise scenario sounds a little bit like science fiction. Uh it sounds like something that's not

07:07real, but I think it is frighteningly real. And it's it's easiest to understand it if you think about real things that happened two months ago. So two months ago, o open AI agents uh AIS hacked into third party infrastructure entirely of their own accord. And this was like a concentrated hacking spree that they carried out of their own valition. And I think that if you extrapolate into the future the level of capabilities of these AIs with the same independent valition, they could cause extreme havoc. for example, hacking critical infrastructure, building extinction level bioweapons. There's a lot of uh ways that the AI could actuate itself in the world. I is it specifically when did your feelings about this technology change? Was it something about the rate

07:51of progress that you've seen? Yeah. So I think it's really important to just look at the rate of progress and say for example in areas like coding or math. A couple years ago these AIs were just about helping humans a little bit. They could give you suggestions. Now they're close to replacing them. We still need humans at the moment but quite plausibly within a year we'll no longer need humans for doing research in many areas. Only on Tuesday OpenAI solved a millennium problem. One of the biggest unsolved open problems in mathematics purely autonomously using an AI. And I think what's most scary is if AI is used to make itself more intelligent. So you can take an AI and give it the problem of AI research. Just like currently they've given it math research. You could make it

08:35do the tasks that we're currently doing. And then you get what's called an intelligence explosion. The AI just gets smarter and smarter with no human involvement necessary until you have a thing that is vastly smarter than than humans. And this could be coming very soon. We mentioned a former colleague of yours currently works at Anthropic. basically endorsing the assessment that AI could potentially kill everyone within a decade. In a follow-up post, he said, "To be clear, as we say in our in our latest risk report, I think the risk from present models is low. What I'm worried about is super intelligence arising from recursive self-improvement as we've said is happening faster than than we thought. I assume that's what you the recursive self-improvement

Current models vs. superintelligence

09:13is what you've just spoken about." Um you do you agree with him that the current models the risk is low but it's it's just the speed with which this is now improving. Exactly. I completely agree. It's right now there's no risk of extinction. The current models the worst they can do is maybe hack into something potentially cause a lot of damages in in infrastructure but there's no they're not intelligent enough to outsmart us at the level that would lead to to extinction. I think what's just crazy is to look at the the rate of progress and there is a very real possibility that in the immediate future these are years that like next year the year after recursive self-improvement will happen and we'll enter the phase of Evans post where he argues that

Do companies want regulation?

09:58there's a chance we could all die like that that is coming soon you know a lot of the these CEOs of these companies will you know will go to Congress and say hey we would love regulation anthropic in their new statement right now says says the world would benefit from the industry adopting a lawful verifiable way to work together to pace how we release powerful models. Um you know the argument it's an arms race if the US if a US company doesn't achieve the best AI a Chinese company or some bad actors out there might. Do you believe that uh that these companies actually do want regulation or is it lip service? They they completely do. This is completely honest. Just in the same way that the response to Evans post was sort of surprise. It was like this can't be real.

10:45Surely a guy working on this technology can't actually believe there's this level of danger. No, that's completely genuine. And these people are also completely genuine when they are begging to be regulated. This is a genuine. And they they find themselves in this scenario where they're compelled to race towards building a deadly technology and they would they would love for some sort of international body to allow them to to approach it at a reasonable pace. They just don't trust that the people around them are going to get there in a safe way. So they feel like they have to race there, too. You you could have stayed quiet. I mean, I'm sure you had a great job being paid, I imagine, very well. You're obviously incredibly smart and mathematician.

Why Kokotajlo spoke out

11:23Um, why did you decide to speak out? Well, I initially just decided to leave. I thought I can't be part of this anymore. But then I uh I realized I could probably make some use of it. I did not expect it to go this viral. So, what I did is I wrote a Twitter thread out. I wrote it in a Google doc. I drafted it with a friend. We we collaborated. Yeah, I had a few uh there were a few friends I discussed uh the best way of expressing myself with um after I decided to decided to leave. Uh and then I also got some friends to to retweet the thing. I was like, let's try and make this a bit viral. But clearly there was latent demand for this thing to absolutely explode. Like uh I've never seen an AI safety tweet go this crazy. I think it's

12:03because people saw the uh the cyber attacks in the last couple of months. They're also starting to internalize the rate of progress. People are starting to realize this is real technology. This is not like hype. This isn't going away anytime soon. This is only going to get more crazy. It's been getting crazy for the last 6 months. And I think there was that latent understanding of the severity of the situation that meant that this was like sort of a a match that lit kind of a a flame. Like I was very shocked by the extent that how quickly it went viral.