# WATCH NOW: Anthropic CEO Dario Amodei Warns UN Security Council of AI Risks to Humanity | AI15

*DWS News* · https://www.youtube.com/watch?v=BCsNphd3Plg

## Summary

Anthropic CEO Dario Amodei briefs the UN Security Council on the rapid progress of AI, citing both its benefits, such as a Claude-led scientific discovery, and two major risks: misuse (including biological weapons) and loss of control. He argues AI safety is a global security issue beyond any one company, and proposes narrow international agreements, evaluation and verification systems, and common global testing standards plus an incident notification system.

## Key points

- Amodei says AI has advanced from barely writing code four years ago to writing most of the code at Anthropic and solving famous unsolved math problems.
- He predicts one or two years, maybe less, to reach what he calls a country of geniuses in the data center.
- Anthropic announced a Claude-led discovery of a new molecular machine that may constitute a new gene-editing mechanism with potential gene therapy applications.
- He identifies two main risks: misuse of AI models, such as by bioterrorists, and loss of control as capabilities outpace developers' ability to manage them.
- Anthropic tests every model for dangerous capabilities, publishes risk reports of hundreds of pages, and pioneered mechanistic interpretability and alignment research.
- On September 12th he called for a collaborative effort to pace the frontier, including embedding external evaluators with employee-like access, similar to a food inspector.
- He proposes narrow agreements such as a ban on using AI to make biological weapons, evaluation and verification systems, and common global testing standards with an incident notification system.
- He calls AI the most important global security issue facing the world today, saying no leader, company, or nation can manage it alone.

## Notes

### Context
- Dario Amodei, CEO of Anthropic, briefs the UN Security Council at a meeting convened by France.
- He frames AI risks as grave enough to belong before the Security Council, not just economic or ICT ministries.

### Rapid AI Progress
- Four years ago AI could barely write a line of code or solve a high school math problem; today it writes most of the code at Anthropic and solves world-famous unsolved math problems.
- He estimates one or two years, maybe less, to reach a "country of geniuses in the data center."
- Anthropic announced a Claude-led discovery of a new molecular machine, possibly a new gene-editing mechanism with gene therapy applications; Claude led literature review, theorizing, and experimental design while humans ran and verified experiments.

### Two Main Risks
- Misuse of AI models, for example by bioterrorists to create biological weapons.
- Loss of control: model capabilities accelerating beyond developers' ability to control them.
- If managed poorly, Amodei believes AI could be a risk to humanity as a whole.

### Anthropic's Safety Practices
- Tests every model for dangerous capabilities in areas like cybersecurity and biology.
- Puts strong safeguards on its most capable models and publishes its work.
- Pioneered mechanistic interpretability and alignment science; releases risk reports running to hundreds of pages.
- Commits to slowing down as much as necessary so each successive AI technology released is safe.

### Global Cooperation Proposals
- On September 12th, Amodei called for a collaborative effort to "pace the frontier" — not stopping AI progress, but ensuring safety keeps up with capability.
- Three parts: embedding external evaluators with employee-like access (like a food inspector), industry-wide cooperation on standards and pace, and global government cooperation on international standards.
- To the Council he proposes: narrow agreements such as a ban on using AI to make biological weapons; evaluation and verification systems giving states visibility into frontier model capability; and common global testing standards for loss-of-control and misuse risks, plus notification of AI incidents significant to global security.

### Conclusion
- He calls this the most important global security issue facing the world today and urges members to set aside differences; no leader, company, or nation can manage it alone.

## Transcript

[00:00] I thank Mr. Altman for his briefing. I now give the floor to Mr. Dario Amodei. >> Good afternoon. I'd like to thank France for convening this meeting and the members of the council for inviting me to speak. AI is advancing very quickly. Four years ago, it could barely write a line of code or solve a high school math problem. Today, it writes most of the code at Anthropic and solves world-famous unsolved math problems. The trajectory only needs to continue for a tiny period longer, one or two years, maybe less, to reach what I've called a country of geniuses in the data center. This rapid pace of progress will produce incredible benefit for the world. Just today, Anthropic announced the Claude-led discovery of a new molecular

[00:45] machine. It's a preliminary finding, but it may constitute a new gene-editing mechanism that could have applications in in gene therapy. Claude led the literature review, theorizing, and experimental design, and humans ran and verified the experiments in the lab. We believe this type of application will accelerate rapidly in the years ahead and that we can make profound progress in the near future on diseases that have plagued humanity for millennia. But these remarkable capabilities also come with risks that must be managed. I see two main categories of risk. The first is misuse of AI models. For example, misuse by bioterrorists to create biological weapons. The second risk is loss of control.

[01:28] Model capabilities accelerating beyond developers' ability to control them. These questions are global in scope and concern all of humanity. They are grave enough that I believe they are important questions for this council, not just for economic or ICT ministries. If managed poorly, I even believe that AI could be a risk to humanity as a whole. At Anthropic, we believe we have the best track record in the industry for preventing these risks, both operationally and in our research. We test every model for dangerous capabilities in areas like cybersecurity and biology. We put strong safeguards on our most capable models, and we publish our work in these areas. We have pioneered research in areas like

[02:11] mechanistic interpretability and alignment science that improve our ability to steer and control our models. We also have unparalleled transparency practices, releasing risk reports that run to hundreds of pages. But, we understand that as AI models become more powerful, even more stringent standards of safety will be required. And we are always aiming to improve our safety and alignment practices, our abilities our ability to prevent misuse, and we are always striving to do better. We will slow down as much as necessary in order to make sure that every successive AI technology that we release is actually safe. But, regardless of what we do, managing these risks is ultimately bigger than any one company. And it has an

[02:57] industry-wide and global scale. On September 12th, I called for a collaborative effort to pace the frontier, not to stop AI progress, but to work together to make sure the industry and the world can proceed at a rate where we can be sure that safety keeps up with capability for everyone in the industry. This call had three parts. First, we committed to embed external evaluators inside Anthropic with employee-like access, similar to a food inspector. And And we recommended that other companies across the world do the same. Some have already agreed uh uh to adopt this measure. Second, we called for cooperation across the industry to set standards and modulate the pace of progress. And finally, we called for global

[03:47] cooperation across the world between governments to set international standards. It is this last area that most squarely falls within this council's remit. I want to offer three ideas in this regard. First, we should begin with narrow agreements that every member can support, such as a ban on using AI to make biological weapons or permitting your AI technologies to be used to make biological weapons. Second, we should build evaluation and verification systems that keep pace with AI development so that states can have visibility into frontier model capability and can verify each other's commitments. Third, we should establish common global standards for testing AI models for loss of control risks and misuse risks, and a

[04:39] notification system for AI incidents that are significant to global security. I believe that this is the most important global security issue facing the world today. For all the differences of the people in this room, we must put aside those differences in order to confront this global opportunity and global threat that is being presented to us at the same time. No leader, no company, and no nation can manage this alone. We commit to working with governments in this room on this urgent work. Thank you for your attention. >> I thank Mr. Amodei for his briefing. I now give the floor to Mr. Klemann de Lange. >> Mr. President, members