1 00:00:03,840 --> 00:00:07,757 Welcome everybody and welcome to those of you joining us on live stream um to 2 00:00:07,840 --> 00:00:09,677 this conversation that I have to say I 3 00:00:09,760 --> 00:00:11,997 have been looking forward to for months. 4 00:00:12,080 --> 00:00:14,797 Uh I had was lucky enough to ch to 5 00:00:14,880 --> 00:00:17,037 moderate a conversation between Darede 6 00:00:17,120 --> 00:00:22,157 and Deis Hassabos last year in Paris. Um which I'm afraid got most attention for 7 00:00:22,240 --> 00:00:26,477 the fact that you two were squashed on a very small love seat while I sat on an 8 00:00:26,560 --> 00:00:30,477 enormous sofa which was probably my screw up. But I said at that point that 9 00:00:30,560 --> 00:00:33,757 this was for me like, you know, chairing a conversation between the Beatles and 10 00:00:33,840 --> 00:00:37,917 the Rolling Stones. And you have not had a conversation on stage since. So this 11 00:00:38,000 --> 00:00:42,077 is, you know, the sequel, the the the, you know, the bands get together again. 12 00:00:42,160 --> 00:00:44,557 I'm delighted. You need no introduction. 13 00:00:44,640 --> 00:00:49,037 Uh the title of our conversation is the day after AGI, which I think is perhaps 14 00:00:49,120 --> 00:00:51,757 slightly getting ahead of ourselves because we should probably talk about 15 00:00:51,840 --> 00:00:55,677 how quickly and easily we will get there. And I want to do a bit of a sort 16 00:00:55,760 --> 00:00:58,717 of update on that and then talk about the consequences. So firstly on the 17 00:00:58,800 --> 00:01:03,437 timeline Dario you last year in Paris said we'll have a model that can do 18 00:01:03,520 --> 00:01:05,277 everything a human could do at the level 19 00:01:05,360 --> 00:01:07,357 of a Nobel laureate across many fields 20 00:01:07,440 --> 00:01:12,397 by 2627. We're in 26. Uh do you still stand by that timeline? 21 00:01:12,480 --> 00:01:15,597 >> So you know it's always hard to know exactly when something will happen but 22 00:01:15,680 --> 00:01:17,597 but I don't I don't think that's going 23 00:01:17,680 --> 00:01:20,237 to turn out to be that far off. So um 24 00:01:20,320 --> 00:01:25,357 you know the the the mechanism whereby imagined it would happen is that we 25 00:01:25,440 --> 00:01:29,437 would make models that were good at coding and good at AI research and we 26 00:01:29,520 --> 00:01:33,837 would use that to produce the next generation of model and speed it up to 27 00:01:33,920 --> 00:01:36,317 create a loop that would that would uh 28 00:01:36,400 --> 00:01:38,477 increase the speed of model development. 29 00:01:38,560 --> 00:01:40,957 We are now in terms of you know the 30 00:01:41,040 --> 00:01:45,517 models that write code I have engineers within anthropic who say I don't write 31 00:01:45,600 --> 00:01:47,757 any code anymore. I just I just let the 32 00:01:47,840 --> 00:01:50,317 model write the code. I edit it. I do 33 00:01:50,400 --> 00:01:52,957 the things around it. I think I don't 34 00:01:53,040 --> 00:01:55,917 know. We might be six to 12 months away 35 00:01:56,000 --> 00:01:58,397 from when the model is doing most maybe 36 00:01:58,480 --> 00:02:01,917 all of what sues do end to end. And then 37 00:02:02,000 --> 00:02:06,797 it's a question of how fast does that loop close. Not every part of that loop 38 00:02:06,880 --> 00:02:10,397 is something that can be sped up by AI, right? There's like chips, there's 39 00:02:10,480 --> 00:02:14,637 manufacturer of chips, there's training time for the model. So it's, you know, I 40 00:02:14,720 --> 00:02:16,077 I think there's a lot of uncertainty. 41 00:02:16,160 --> 00:02:21,677 It's easy to see how this could take a few years. I don't I I it's very hard 42 00:02:21,760 --> 00:02:25,997 for me to see how it could take longer than that. Um but if if I had to guess, 43 00:02:26,080 --> 00:02:29,997 I would guess that this goes faster than people imagine. And that that key 44 00:02:30,080 --> 00:02:32,877 element of code and increasingly 45 00:02:32,960 --> 00:02:36,317 research going faster than we imagine. 46 00:02:36,400 --> 00:02:40,237 That's going to be the key driver. It's it's really hard to predict again how 47 00:02:40,320 --> 00:02:43,997 much that exponential is going to speed us up, but but something fast is going 48 00:02:44,080 --> 00:02:48,317 to happen. So you demis were a little more cautious last year. You said a 50% 49 00:02:48,400 --> 00:02:52,397 chance of a system that can exhibit all the cognitive capabilities humans can by 50 00:02:52,480 --> 00:02:55,117 the end of the decade. Um clearly in 51 00:02:55,200 --> 00:02:59,757 coding as Dario says it's been remarkable. What is your sense of do you 52 00:02:59,840 --> 00:03:02,637 stand by your prediction and what's changed in the past year? 53 00:03:02,720 --> 00:03:06,317 >> Yeah, look I I I I think I'm still on the same kind of timeline. And I think 54 00:03:06,400 --> 00:03:08,237 there has been remarkable progress. But 55 00:03:08,320 --> 00:03:11,917 I think some areas of uh uh um kind of 56 00:03:12,000 --> 00:03:16,477 engineering work, coding or so you could say mathematics are a little bit easier 57 00:03:16,560 --> 00:03:20,237 to see how they would be automated partly because they're verifiable what 58 00:03:20,320 --> 00:03:24,317 the output is. Um some areas of natural science are much harder to do than that. 59 00:03:24,400 --> 00:03:27,677 You won't necessarily know if the chemical compound you've built or this 60 00:03:27,760 --> 00:03:31,117 prediction about physics is correct. It may be you may have to test it 61 00:03:31,200 --> 00:03:35,357 experimentally and that will all take longer. So uh I also think there are 62 00:03:35,440 --> 00:03:37,837 some missing capabilities at the moment 63 00:03:37,920 --> 00:03:42,477 uh in terms of like not just solving existing conjectures uh or existing 64 00:03:42,560 --> 00:03:46,157 problems but actually coming up with the question in the first place or coming up 65 00:03:46,240 --> 00:03:49,597 with the theory or the hypothesis. I think that's much much harder and I 66 00:03:49,680 --> 00:03:52,957 think that's the highest level of scientific creativity and it's not 67 00:03:53,040 --> 00:03:55,837 clear. I think we will have those systems. I don't think it's impossible 68 00:03:55,920 --> 00:03:59,837 but I think there may be one or two missing ingredients. Um, it remains to 69 00:03:59,920 --> 00:04:02,957 be seen how, you know, first of all, can this self-improvement loop that we're 70 00:04:03,040 --> 00:04:06,397 all working on actually close without a human in the loop. I think there are 71 00:04:06,480 --> 00:04:09,357 also risks to that to that kind of system, by the way, which we should 72 00:04:09,440 --> 00:04:13,677 discuss and I'm sure we will. But the the but but that could speed things up 73 00:04:13,760 --> 00:04:15,277 if that kind of system does work. 74 00:04:15,360 --> 00:04:17,997 >> We'll get to the risks in a minute. But one other change I think of the past 75 00:04:18,080 --> 00:04:21,917 year has been a kind of change in the pecking order of the race, if you will. 76 00:04:22,000 --> 00:04:25,597 This time a year ago, we just had the deepseek moment and everyone was 77 00:04:25,680 --> 00:04:29,677 incredibly excited about what happened there and there was still a sense, you 78 00:04:29,760 --> 00:04:35,037 know, that Google Deep Mind was kind of lagging open AI. I would say that now uh 79 00:04:35,120 --> 00:04:38,157 it's looking quite different. I mean, they've declared code red, right? Um 80 00:04:38,240 --> 00:04:40,397 it's been quite a quite a year. So, talk 81 00:04:40,480 --> 00:04:45,437 me through what specifically you've been surprised by and how well you've done 82 00:04:45,520 --> 00:04:48,477 this year and whether you think and then I'm going to ask you about the lineup. 83 00:04:48,560 --> 00:04:52,957 Well, look, I I think we were I was always very confident we uh would get 84 00:04:53,040 --> 00:04:56,557 back to sort of the top of the the leaderboards and and the soda type of 85 00:04:56,640 --> 00:05:00,717 models across the board because I think we've always had like the deepest and 86 00:05:00,800 --> 00:05:04,397 broadest research bench and it was about kind of marshalling that all together 87 00:05:04,480 --> 00:05:08,477 and um getting the intensity and focus and the kind of startup mentality back 88 00:05:08,560 --> 00:05:10,557 to the whole organization and it's been 89 00:05:10,640 --> 00:05:15,677 a a lot of work and um but I think we're and we're still a lot of work to do um 90 00:05:15,760 --> 00:05:19,837 but I think you can start seeing the the the the the you know the the kind of um 91 00:05:19,920 --> 00:05:24,397 the progress that's been made in both the models with Gemini 3 but also uh on 92 00:05:24,480 --> 00:05:28,877 the product side with Gemini app getting increasing uh market share. So I feel 93 00:05:28,960 --> 00:05:33,837 like uh we're making great progress um but there's a ton more work to do um and 94 00:05:33,920 --> 00:05:37,117 you know we're bringing to bear Google DeepMind's kind of like the engine room 95 00:05:37,200 --> 00:05:40,957 of Google where we're getting used to um shipping our models more and more more 96 00:05:41,040 --> 00:05:44,717 quickly into the product surfaces. One question for you Daria on on this aspect 97 00:05:44,800 --> 00:05:48,317 of it because you've just saw you're in the process of you know a new round at 98 00:05:48,400 --> 00:05:50,397 an extraordinary valuation too. Um but 99 00:05:50,480 --> 00:05:55,437 you are unlike them as a let's call it an independent model maker and there is 100 00:05:55,520 --> 00:05:58,637 I think an increasing concern that the independent model makers will not be 101 00:05:58,720 --> 00:06:02,797 able to continue for long enough until you get to where the revenues come in. 102 00:06:02,880 --> 00:06:06,797 Um it's made very openly about open AI but talk me through how you think about 103 00:06:06,880 --> 00:06:09,677 that and then we'll get to the AGI itself. Yeah, I mean that you know I 104 00:06:09,760 --> 00:06:11,437 think I think I think how we think about 105 00:06:11,520 --> 00:06:14,477 that is you know as we've built better 106 00:06:14,560 --> 00:06:19,197 and better models there's been a kind of exponential relationship not only 107 00:06:19,280 --> 00:06:22,957 between how much compute you put into the model and how cognitively capable it 108 00:06:23,040 --> 00:06:27,037 is but between how cognitively capable it is and how much revenue it's able to 109 00:06:27,120 --> 00:06:29,437 generate. So our revenues grown 10x in 110 00:06:29,520 --> 00:06:31,277 the last three years from 0 to 100 111 00:06:31,360 --> 00:06:34,077 million in 2023 100 million to a billion 112 00:06:34,160 --> 00:06:39,277 in 2024 and 1 billion to 10 billion in 2025. And so th those revenue numbers, 113 00:06:39,360 --> 00:06:41,997 you know, I don't know if that curve will literally continue. It would be 114 00:06:42,080 --> 00:06:47,117 crazy if it did. Um, but those numbers are starting to get not too far from, 115 00:06:47,200 --> 00:06:50,317 you know, the sca the scale of the largest companies in the world. So 116 00:06:50,400 --> 00:06:52,717 there's there's there's always uncertainty. You know, we're trying to 117 00:06:52,800 --> 00:06:57,037 bootstrap this from nothing. It's it's a crazy thing, but but I have confidence 118 00:06:57,120 --> 00:07:01,917 that if we're able to produce the best models in the things that we focus on, 119 00:07:02,000 --> 00:07:05,677 um, uh, then I think then I think things will go well. And you know, I I will I 120 00:07:05,760 --> 00:07:09,037 will generally say, you know, I think I think it's been a good year for both 121 00:07:09,120 --> 00:07:13,837 both Google and Anthropic. And I think the thing we actually have in common is 122 00:07:13,920 --> 00:07:17,277 that they're you know, they're both kind of kind of kind of companies that are, 123 00:07:17,360 --> 00:07:20,477 you know, or the research part of the company that are kind of led by 124 00:07:20,560 --> 00:07:25,677 researchers who focus on the models who focus on solving important problems in 125 00:07:25,760 --> 00:07:27,117 the world, right? Who have these kind of 126 00:07:27,200 --> 00:07:30,237 hard scientific problems as a as a north 127 00:07:30,320 --> 00:07:34,717 star. and and and I think those are the kind of companies that are going to 128 00:07:34,800 --> 00:07:38,717 succeed going forward and you know I think I think we share that between us 129 00:07:38,800 --> 00:07:41,597 >> very much. Uh I'm I'm going to resist the temptation to ask you what will 130 00:07:41,680 --> 00:07:45,117 happen to the companies that are not led by researchers 131 00:07:45,200 --> 00:07:46,477 uh because I know you won't answer it. 132 00:07:46,560 --> 00:07:51,757 But let's then go on to uh the predictions area now and this we are 133 00:07:51,840 --> 00:07:54,717 supposed to be talking about the day after AI but let's talk about closing 134 00:07:54,800 --> 00:07:59,037 the loop. This the odds that you will get models that will close the loop and 135 00:07:59,120 --> 00:08:02,557 be able to you know power themselves if you will because that's the really the 136 00:08:02,640 --> 00:08:06,957 crux for the the winner takes all threshold approach. Do you still believe 137 00:08:07,040 --> 00:08:11,277 that we are likely to see that or is this going to be much more of a normal 138 00:08:11,360 --> 00:08:15,597 technology where followers and catchup can can compete? 139 00:08:15,680 --> 00:08:18,237 >> Well, look, I definitely don't think it's going to be a normal technology. 140 00:08:18,320 --> 00:08:22,077 So, I mean, there are aspects already that as Dario mentioned that it's 141 00:08:22,160 --> 00:08:26,557 already helping with our coding and and some aspects of research. The full 142 00:08:26,640 --> 00:08:30,157 closing of the loop though, I think is an unknown. I mean, I think it's 143 00:08:30,240 --> 00:08:33,117 possible to do. you may need AGI itself 144 00:08:33,200 --> 00:08:37,197 to be able to do that in some domains again where there these domains you know 145 00:08:37,280 --> 00:08:40,717 where there's there's more messiness around them it's not so easy to verify 146 00:08:40,800 --> 00:08:45,917 your answer very quickly um there's kind of MP hard domains so as soon as you 147 00:08:46,000 --> 00:08:49,917 start getting more and you know I also include by the way for AGI physical AI 148 00:08:50,000 --> 00:08:53,037 robotics working all of these kind of things and then you've got you know 149 00:08:53,120 --> 00:08:57,597 hardware in the loop uh that may uh limit how fast the self-improvement 150 00:08:57,680 --> 00:09:01,837 systems can work but I think in coding and mathematics and these kind areas. I 151 00:09:01,920 --> 00:09:04,477 can definitely see that working. And then the question is more theoretical 152 00:09:04,560 --> 00:09:06,557 one is what is the limit of engineering 153 00:09:06,640 --> 00:09:09,677 and maths uh to solve uh the natural 154 00:09:09,760 --> 00:09:14,317 sciences. Daria, you um last year, I think it was last year that you 155 00:09:14,400 --> 00:09:16,317 published Machines of Love and Grace um 156 00:09:16,400 --> 00:09:19,357 which was a very I would say upbeat 157 00:09:19,440 --> 00:09:23,837 essay about the potential that that you were going to see unfold and you were 158 00:09:23,920 --> 00:09:26,157 talking about you know a a what was it a 159 00:09:26,240 --> 00:09:29,037 genius of data at country 160 00:09:29,120 --> 00:09:33,677 data center I'm told that you are working on an update to this a new essay 161 00:09:33,760 --> 00:09:37,917 so you know wait for it guys it's not out yet but it is coming out but perhaps 162 00:09:38,000 --> 00:09:43,437 you can give us a sort of a sneak preview of what a year later your big 163 00:09:43,520 --> 00:09:46,557 take is going to be. >> Yes. So, you know, my take my take has 164 00:09:46,640 --> 00:09:50,477 not changed. It has always been my view that, you know, AI is going to be 165 00:09:50,560 --> 00:09:53,517 incredibly powerful. I think Demis and I, you know, kind of agree on that. It's 166 00:09:53,600 --> 00:09:57,997 just a question of exactly when. Um, uh, and because it's incredibly powerful, it 167 00:09:58,080 --> 00:10:00,957 will do all these wonderful things like the ones I talked about in Machines of 168 00:10:01,040 --> 00:10:05,437 Loving Grace. It, you know, will help us cure cancer. It may help us to eradicate 169 00:10:05,520 --> 00:10:09,997 tropical diseases. It will help us understand understand the universe. but 170 00:10:10,080 --> 00:10:14,237 that there are these, you know, immense and grave risks that, you know, not that 171 00:10:14,320 --> 00:10:18,397 we can't address them. I'm not a doomer, but but that, you know, we we we we we 172 00:10:18,480 --> 00:10:21,677 need to think about them and we need to address them. And I wrote Machines of 173 00:10:21,760 --> 00:10:25,837 Loving Grace first. I' I'd love to give some uh a sophisticated reason why I 174 00:10:25,920 --> 00:10:29,597 wrote that first, but it was just that the the positive essay was easier and 175 00:10:29,680 --> 00:10:34,157 more fun to write than than the negative essay. Um, so, you know, I finally spent 176 00:10:34,240 --> 00:10:37,997 some time on vacation and I was able to write an essay about the risks. Even 177 00:10:38,080 --> 00:10:40,797 when I'm writing about the risks, um, I 178 00:10:40,880 --> 00:10:43,677 I I try, you know, I I I'm like an 179 00:10:43,760 --> 00:10:47,997 optimistic person, right? So, even as I'm writing about these risks, I I I 180 00:10:48,080 --> 00:10:51,517 wrote about it in a way that was like, how do we overcome these risks? How do 181 00:10:51,600 --> 00:10:56,077 we have a battle plan to fight them? And and and the way I the way I framed it 182 00:10:56,160 --> 00:11:00,157 was, you know, there's this scene from Carl Sean's Contact, the movie version 183 00:11:00,240 --> 00:11:04,077 of it, where, you know, they they kind of discover alien life and this 184 00:11:04,160 --> 00:11:07,997 international panel that's like interviewing um uh you know, people to, 185 00:11:08,080 --> 00:11:12,237 you know, to be humanity's representative to meet the alien. Um uh 186 00:11:12,320 --> 00:11:15,917 and uh one one of the questions they ask one of the candidates is, you know, if 187 00:11:16,000 --> 00:11:19,197 you could ask the aliens any one question, what it would what what what 188 00:11:19,280 --> 00:11:23,517 would it be? And one of one of the characters says,"I would ask,"How did 189 00:11:23,600 --> 00:11:28,557 you do it? How did you manage to get through this technological adolescence 190 00:11:28,640 --> 00:11:32,717 without destroying yourselves? How did you make it through?" And and and ever 191 00:11:32,800 --> 00:11:35,437 since I saw it, it was like 20 years ago, I think I saw that movie. It's kind 192 00:11:35,520 --> 00:11:39,357 of stuck with me. And that that's the frame that I use, which is which is 193 00:11:39,440 --> 00:11:41,997 that, you know, we we're we're we are 194 00:11:42,080 --> 00:11:44,557 knocking on the door of these incredible 195 00:11:44,640 --> 00:11:46,717 capabilities, right? the the ability to 196 00:11:46,800 --> 00:11:51,677 build basically machines out of sand, right? I think I think it was inevitable 197 00:11:51,760 --> 00:11:53,517 that the instant we started working with 198 00:11:53,600 --> 00:11:57,437 fire. Um uh but but how we handle it is 199 00:11:57,520 --> 00:11:59,597 is not inevitable. And so I think the 200 00:11:59,680 --> 00:12:04,637 next few years we're going to be dealing with, you know, how do we keep these 201 00:12:04,720 --> 00:12:06,317 systems under control that are highly 202 00:12:06,400 --> 00:12:09,197 autonomous and smarter than any human? 203 00:12:09,280 --> 00:12:12,477 How do we make sure that individuals 204 00:12:12,560 --> 00:12:16,317 don't misuse them? Right? I have worries about things like bioteterrorism. How do 205 00:12:16,400 --> 00:12:20,237 we make sure that nation states don't misuse them? That's why I've been so 206 00:12:20,320 --> 00:12:24,637 concerned about, you know, the CCP, other authoritarian authoritarian 207 00:12:24,720 --> 00:12:27,917 governments. What are the economic impacts? Right? I've talked about labor 208 00:12:28,000 --> 00:12:31,277 displacement a lot. And and you know, what what haven't we thought of which 209 00:12:31,360 --> 00:12:35,037 which in many cases, you know, maybe may be the the hardest thing to deal with at 210 00:12:35,120 --> 00:12:37,597 all. Um, so, you know, I I'm I'm 211 00:12:37,680 --> 00:12:42,477 thinking through how to address those risks. you know, for for each of these, 212 00:12:42,560 --> 00:12:46,557 it's a mixture of things that we individually need to do as as leaders of 213 00:12:46,640 --> 00:12:50,797 the of of of the companies and that we can do working together. And then there 214 00:12:50,880 --> 00:12:54,717 there's going to need to be some role for wider societal institutions like the 215 00:12:54,800 --> 00:12:58,877 like the government in in in addressing all of these. But, you know, I I I just 216 00:12:58,960 --> 00:13:02,397 feel this urgency that, you know, every day, you know, there's there's all kinds 217 00:13:02,480 --> 00:13:03,997 of crazy stuff going on in the outside 218 00:13:04,080 --> 00:13:07,117 world, outside AI, right? Um but but you 219 00:13:07,200 --> 00:13:09,677 know my my my view is this is happening 220 00:13:09,760 --> 00:13:14,877 so fast and is such a crisis we should be devoting almost all of our effort to 221 00:13:14,960 --> 00:13:16,637 thinking about how to get through this. 222 00:13:16,720 --> 00:13:20,237 >> So I can't decide whether I'm more surprised that you a take a vacation b 223 00:13:20,320 --> 00:13:24,157 when you take a vacation you think about the risks of AI and c that your essay is 224 00:13:24,240 --> 00:13:28,557 framed in terms of are we going to get through the technological adolescence of 225 00:13:28,640 --> 00:13:31,837 this technology without destroying ourselves. So, I'm my head is slightly 226 00:13:31,920 --> 00:13:35,117 spinning, but you then and I can't wait to read it, but you you you mentioned 227 00:13:35,200 --> 00:13:39,037 several areas that can guide the rest of our conversation. Let's start with jobs 228 00:13:39,120 --> 00:13:41,837 um because you actually have been very outspoken about that and I think you 229 00:13:41,920 --> 00:13:44,877 said that half of entry- level white collar jobs could be gone within the 230 00:13:44,960 --> 00:13:46,477 next one to five years. But I'm going to 231 00:13:46,560 --> 00:13:49,997 turn to you Demis because so far we 232 00:13:50,080 --> 00:13:54,877 haven't actually seen any discernable impact on the labor market. Um, yes, 233 00:13:54,960 --> 00:13:58,477 unemployment has ticked up in the US, but all of the kind of economic studies 234 00:13:58,560 --> 00:14:02,637 I've looked at and that we've written about suggest that this is overhiring 235 00:14:02,720 --> 00:14:04,397 post pandemic that it's really not 236 00:14:04,480 --> 00:14:07,277 AIdriven. If anything, people are hiring 237 00:14:07,360 --> 00:14:10,477 to build out AI capability. 238 00:14:10,560 --> 00:14:15,517 Do you think that this will be as you know economists have always argued that 239 00:14:15,600 --> 00:14:18,797 it's not a lump of labor fallacy that actually there will be new jobs created 240 00:14:18,880 --> 00:14:23,197 because so far the evidence seems to suggest that. Yeah, I mean I I think in 241 00:14:23,280 --> 00:14:26,397 um the near term that is what will happen. The kind of normal evolution 242 00:14:26,480 --> 00:14:28,317 when a breakthrough technology arrives. 243 00:14:28,400 --> 00:14:32,637 So some jobs will get disrupted but I think new even more valuable perhaps 244 00:14:32,720 --> 00:14:34,717 more meaningful jobs will get created. 245 00:14:34,800 --> 00:14:39,037 Um I think we're going to see this year the beginnings of maybe impacting the 246 00:14:39,120 --> 00:14:43,037 junior level entry level child of jobs internships this type of thing. And I 247 00:14:43,120 --> 00:14:47,117 think there is some evidence I can feel that ourselves maybe like a slowdown in 248 00:14:47,200 --> 00:14:50,637 hiring in that. But I think that can be more than compensated by the fact there 249 00:14:50,720 --> 00:14:54,877 are these amazing creative tools out there pretty much available for everyone 250 00:14:54,960 --> 00:14:57,437 uh almost for free that if you know I 251 00:14:57,520 --> 00:15:00,317 was to talk to a class of undergrads 252 00:15:00,400 --> 00:15:04,557 right now I would be telling them to get really unbelievably proficient with 253 00:15:04,640 --> 00:15:08,237 these tools. I think to the extent that even those of us building it, we're so 254 00:15:08,320 --> 00:15:11,997 busy building it, it's hard to have also time to really explore the almost the 255 00:15:12,080 --> 00:15:16,717 capability overhang even today's models and products have let alone tomorrow's 256 00:15:16,800 --> 00:15:20,797 and I think that uh can be maybe better than a traditional internship would have 257 00:15:20,880 --> 00:15:23,197 been in terms of you sort of leaprogging 258 00:15:23,280 --> 00:15:28,237 uh yourself to be useful uh in a useful in a profession. So I think there's 259 00:15:28,320 --> 00:15:32,397 that's what I see happening probably in the next five years. Um maybe we again 260 00:15:32,480 --> 00:15:35,997 slightly differ on time scales on that but I think what happens after AGI 261 00:15:36,080 --> 00:15:39,917 arrives that's a different question cuz I think really we would be in uncharted 262 00:15:40,000 --> 00:15:42,157 territory at that point. >> Do you think it's going to take longer 263 00:15:42,240 --> 00:15:45,037 than you thought last year when you said half of all white 264 00:15:45,120 --> 00:15:48,717 >> colors? I have about the same view. I I actually agree with you and with Demis 265 00:15:48,800 --> 00:15:52,557 that at the time I made the comment there was no impact on the labor market. 266 00:15:52,640 --> 00:15:56,877 I wasn't saying there was an impact on the labor market at that moment. Um, you 267 00:15:56,960 --> 00:16:00,637 know, now I think maybe we're starting to see just just the little beginnings 268 00:16:00,720 --> 00:16:02,317 of it, you know, in software in coding. 269 00:16:02,400 --> 00:16:04,797 I even see it within within anthropic 270 00:16:04,880 --> 00:16:07,997 where, you know, I you know, I can look 271 00:16:08,080 --> 00:16:12,237 forward I can kind of look forward to a time where on the more junior end and 272 00:16:12,320 --> 00:16:16,157 then on the more on the more on the more on the more intermediate end, we 273 00:16:16,240 --> 00:16:17,837 actually need less and not more people. 274 00:16:17,920 --> 00:16:21,277 And you know, we're thinking about how to deal with that within anthropic in a 275 00:16:21,360 --> 00:16:24,157 in a in a you know, sense in a sensible 276 00:16:24,240 --> 00:16:28,157 way. Um I, you know, one to five years 277 00:16:28,240 --> 00:16:32,397 as of six months ago, I would stick with that. You know, if you kind of, you 278 00:16:32,480 --> 00:16:36,077 know, connect this to what I said before, which is, you know, we we might 279 00:16:36,160 --> 00:16:40,557 have AI that's better than humans at at everything in, you know, maybe one to 280 00:16:40,640 --> 00:16:45,357 two years, maybe a little longer than that. The those don't seem to line up. 281 00:16:45,440 --> 00:16:47,517 The reason is that there's this there's 282 00:16:47,600 --> 00:16:52,397 this lag and there's this replacement thing, right? I I know the labor market 283 00:16:52,480 --> 00:16:56,797 is adaptable, right? Just like you know 80% of people used to do farming you 284 00:16:56,880 --> 00:16:59,757 know farming got automated and then they became factory workers and then 285 00:16:59,840 --> 00:17:04,957 knowledge workers. So you know there is some level of adaptability here as well 286 00:17:05,040 --> 00:17:08,077 right we should be economically sophisticated about how the labor market 287 00:17:08,160 --> 00:17:12,717 works but my worry is as this exponential keeps compounding and I 288 00:17:12,800 --> 00:17:17,197 don't think it's going to take that long again somewhere between between a year 289 00:17:17,280 --> 00:17:20,717 and five years it will overwhelm our ability to adapt. I think I may be 290 00:17:20,800 --> 00:17:25,597 saying the same thing Demis is just factored out of that that difference we 291 00:17:25,680 --> 00:17:28,797 have about timelines which I think ultimately comes down to how how fast 292 00:17:28,880 --> 00:17:32,077 you close the loop on CO. >> How much confidence do you have that 293 00:17:32,160 --> 00:17:35,197 governments get the scale of this and 294 00:17:35,280 --> 00:17:39,837 have are beginning to think about what policy responses they need to have? 295 00:17:39,920 --> 00:17:44,157 >> I don't think that that that it's anywhere near enough work going on about 296 00:17:44,240 --> 00:17:47,197 this. I'm I'm constantly surprised even when I meet economists at places like 297 00:17:47,280 --> 00:17:51,037 this that they're not more of uh professional economist professors 298 00:17:51,120 --> 00:17:53,997 thinking about what happens um and not 299 00:17:54,080 --> 00:17:58,557 just sort of on the way to AGI but um uh even if we get all the technical things 300 00:17:58,640 --> 00:18:01,917 right that Dario was talking about and the job displacement is one question 301 00:18:02,000 --> 00:18:04,397 we're worried about the economics of that but maybe there are ways to 302 00:18:04,480 --> 00:18:08,557 distribute this new productivity this new wealth more fairly I don't know if 303 00:18:08,640 --> 00:18:11,517 we have the right institutions to do that but that's what should happen at 304 00:18:11,600 --> 00:18:15,037 that point there should be you know we maybe in a post scarcity world. But then 305 00:18:15,120 --> 00:18:17,837 there are even the things that keep me up right now. There are even bigger 306 00:18:17,920 --> 00:18:19,757 questions than that at that point to do 307 00:18:19,840 --> 00:18:23,597 with meaning and um purpose and a lot of 308 00:18:23,680 --> 00:18:26,877 the things that we get from our jobs not just economically. That's one question. 309 00:18:26,960 --> 00:18:31,277 But I think that may be easier to solve strangely than uh what happens to the 310 00:18:31,360 --> 00:18:33,757 human condition and humanity as a whole. 311 00:18:33,840 --> 00:18:36,877 And I think I'm also optimistic we'll come up with new answers there. We do a 312 00:18:36,960 --> 00:18:39,437 lot of things today um from extreme 313 00:18:39,520 --> 00:18:41,517 sports to art that aren't necessarily 314 00:18:41,600 --> 00:18:44,717 directly to do with economic gain. So I 315 00:18:44,800 --> 00:18:48,557 think we will find uh meaning and maybe there'll be even more sort of 316 00:18:48,640 --> 00:18:52,717 sophisticated versions of those activities. Um plus I think we'll be 317 00:18:52,800 --> 00:18:57,117 exploring the stars. So there'll be all of that to to factor in as well for in 318 00:18:57,200 --> 00:19:01,037 terms of purpose. But I think it's really worth thinking now even on my 319 00:19:01,120 --> 00:19:05,517 timelines of like five to 10 years away that isn't a lot of time uh before this 320 00:19:05,600 --> 00:19:07,837 comes. How big do you think is the risk 321 00:19:07,920 --> 00:19:11,197 of a popular backlash against AI that 322 00:19:11,280 --> 00:19:16,317 will somehow kind of cause governments to do what from your perspective might 323 00:19:16,400 --> 00:19:20,237 be stupid things? Because I'm just thinking back to the era of, you know, 324 00:19:20,320 --> 00:19:24,717 globalization in the 1990s when when there was indeed some displacement of 325 00:19:24,800 --> 00:19:26,957 jobs. Governments didn't do enough. The 326 00:19:27,040 --> 00:19:31,757 public backlash was such that we've ended up sort of where we are now. uh do 327 00:19:31,840 --> 00:19:35,997 you think that there is a risk that there will be a growing antipathy 328 00:19:36,080 --> 00:19:40,637 towards what you are doing and your companies in the kind of body politic? 329 00:19:40,720 --> 00:19:44,477 >> Um I think there's definitely a risk. I think um I think that's kind of 330 00:19:44,560 --> 00:19:47,757 reasonable. There's fear and there's worries about these things like jobs and 331 00:19:47,840 --> 00:19:51,917 livelihoods. Um I think there's a couple of things that I mean it's going to be 332 00:19:52,000 --> 00:19:55,757 very complicated the next few years I think geopolitically but also the 333 00:19:55,840 --> 00:19:59,197 various factors here like we want to and we're trying to do this with AlphaFold 334 00:19:59,280 --> 00:20:03,837 and our science work and isomeorphic our spinout company solve all disease cure 335 00:20:03,920 --> 00:20:07,517 diseases come up with new energy sources I think as a society it's clear we'd 336 00:20:07,600 --> 00:20:11,677 want that I think maybe the balance of what the industry is doing is not enough 337 00:20:11,760 --> 00:20:14,877 balance towards those types of activities I think we should have a lot 338 00:20:14,960 --> 00:20:19,117 more examples I know Dary agrees with me of like alpha fold like things that help 339 00:20:19,200 --> 00:20:22,957 sort of unequivocal good in the world and I think actually it's incumbent on 340 00:20:23,040 --> 00:20:26,477 the industry and and all of us leading players to show that more demonstrate 341 00:20:26,560 --> 00:20:30,557 that not just talk about it but demonstrate that um and but then it's 342 00:20:30,640 --> 00:20:34,477 going to come with these other intendent disruptions and um but I don't I think 343 00:20:34,560 --> 00:20:37,277 the other issue is the geopolitical competition there's obviously 344 00:20:37,360 --> 00:20:41,837 competition between the companies but also US and China primarily so unless 345 00:20:41,920 --> 00:20:46,157 there's an international cooperation or or understanding around this um uh which 346 00:20:46,240 --> 00:20:49,757 I think would be good actually in terms things like minimum safety standards for 347 00:20:49,840 --> 00:20:52,477 deployment. I think Dario would agree on that as well. I think it's vitally 348 00:20:52,560 --> 00:20:55,357 needed. This technology is going to be crossborder. It's going to affect 349 00:20:55,440 --> 00:20:59,117 everyone. It's going to affect all of humanity. Um actually contact is one of 350 00:20:59,200 --> 00:21:01,357 my favorite films as well. So funny 351 00:21:01,440 --> 00:21:06,157 enough, I didn't realize it was yours too, Dario. But I I think um um you know 352 00:21:06,240 --> 00:21:10,237 those kind of things need to be worked through. Um and and if we can maybe it 353 00:21:10,320 --> 00:21:13,917 would be good to have a bit of slow a slightly slower pace than we're 354 00:21:14,000 --> 00:21:18,237 currently predicting even my timelines so that we can get this right society 355 00:21:18,320 --> 00:21:22,077 but it that would require some coordination that is I prefer your 356 00:21:22,160 --> 00:21:25,517 timelines. Yes, I think I battle concede. 357 00:21:25,600 --> 00:21:29,117 >> But but but Dario, let's turn to this now because the one thing since we last 358 00:21:29,200 --> 00:21:31,437 spoke uh in Paris, the geopolitical 359 00:21:31,520 --> 00:21:36,237 environment has, if anything, I don't know, complicated, mad, crazy, whatever, 360 00:21:36,320 --> 00:21:38,237 whatever phrase you want to use. 361 00:21:38,320 --> 00:21:42,557 Secondly, the US has a very different approach now towards China. It's a much 362 00:21:42,640 --> 00:21:44,877 more it's a kind of no holds bar, go as 363 00:21:44,960 --> 00:21:50,397 fast as we can, but then sell chips to China. Um, and that is so you've got a 364 00:21:50,480 --> 00:21:55,357 different attitude towards the United States. you've got a a very um strange 365 00:21:55,440 --> 00:21:59,517 relationship between the United States and and Europe right now geopolitically 366 00:21:59,600 --> 00:22:02,717 against that. I mean I hear you talk about it would be nice to have a CERN 367 00:22:02,800 --> 00:22:06,157 like organization I mean it's a million years from where we are from the real 368 00:22:06,240 --> 00:22:11,037 world. So in the real world have the geopolitical risks increased and what if 369 00:22:11,120 --> 00:22:14,317 anything do you think should be done about that and and the administration 370 00:22:14,400 --> 00:22:17,357 seems to be doing the opposite of what you were suggesting? Yeah, I mean, look, 371 00:22:17,440 --> 00:22:19,997 you know, we're we're we're just trying to do the best we can to, you know, 372 00:22:20,080 --> 00:22:22,797 we're just we're just one company and we're we're trying to operate in, you 373 00:22:22,880 --> 00:22:26,877 know, the the environment that exists, no matter how no matter how crazy it is. 374 00:22:26,960 --> 00:22:30,877 But, you know, I think I think at least my policy recommendations haven't 375 00:22:30,960 --> 00:22:33,517 changed that, you know, not selling 376 00:22:33,600 --> 00:22:36,317 chips is one of the, you know, one of 377 00:22:36,400 --> 00:22:38,637 the one of the biggest things we can do 378 00:22:38,720 --> 00:22:41,357 um to, you know, make sure that we have 379 00:22:41,440 --> 00:22:44,077 the time to handle this. Um, you know, 380 00:22:44,160 --> 00:22:45,757 you know, I said I said before, you 381 00:22:45,840 --> 00:22:51,117 know, I I I prefer Demis' timeline. I wish we had 5 to 10 years, you know, so 382 00:22:51,200 --> 00:22:54,397 it's it's possible he's just right and I'm just wrong, but but assume I'm right 383 00:22:54,480 --> 00:22:56,397 and it can be done in one to two years. 384 00:22:56,480 --> 00:22:58,957 Why can't we slow down to to Demis' timeline? 385 00:22:59,040 --> 00:23:02,797 >> Well, you could just slow down. Well, no. The but but but the reason the 386 00:23:02,880 --> 00:23:05,197 reason we the reason we can't do that is 387 00:23:05,280 --> 00:23:07,597 is you know because we have 388 00:23:07,680 --> 00:23:10,557 >> geopolitical adversaries building the 389 00:23:10,640 --> 00:23:14,717 same technology at a similar pace, it's very hard to have an enforcable 390 00:23:14,800 --> 00:23:16,797 agreement where they slow down and we 391 00:23:16,880 --> 00:23:19,517 slow down and and so if we can just 392 00:23:19,600 --> 00:23:22,317 >> if we can just not sell the chips, then 393 00:23:22,400 --> 00:23:26,557 this isn't a question of competition between the US and China. This is a 394 00:23:26,640 --> 00:23:29,757 question of competition between me and Demis which I'm very confident that we 395 00:23:29,840 --> 00:23:32,797 can work out. >> And what do you make of the logic of the 396 00:23:32,880 --> 00:23:34,477 administration which as I understand it 397 00:23:34,560 --> 00:23:35,997 is we need to sell them chips because we 398 00:23:36,080 --> 00:23:40,077 need to bind them into US supply chains. 399 00:23:40,160 --> 00:23:44,157 So you know it's it's I I think it's I 400 00:23:44,240 --> 00:23:46,637 think it's a question not just of time 401 00:23:46,720 --> 00:23:48,477 scale but of the significance of the 402 00:23:48,560 --> 00:23:51,997 technology right if this was telecom or 403 00:23:52,080 --> 00:23:56,717 something then all this stuff about proliferating the US stack and you know 404 00:23:56,800 --> 00:23:58,797 wanting to build our you know chips 405 00:23:58,880 --> 00:24:00,477 around the world to make sure that you 406 00:24:00,560 --> 00:24:04,237 know you know this c you know the you 407 00:24:04,320 --> 00:24:07,757 know these random countries in different parts of the world you know build data 408 00:24:07,840 --> 00:24:12,317 centers that have Nvidia chips instead of Huawei chips. You know, I think of 409 00:24:12,400 --> 00:24:14,077 this more as like, you know, it's a 410 00:24:14,160 --> 00:24:16,957 decision. Are we going to, you know, 411 00:24:17,040 --> 00:24:20,397 sell nuclear weapons to North Korea and, 412 00:24:20,480 --> 00:24:24,477 you know, because that produces some profit for Boeing. Um, you know, where 413 00:24:24,560 --> 00:24:27,997 where we can say, okay, yeah, these cases were made by Boeing, like the US 414 00:24:28,080 --> 00:24:31,917 is winning. Like, this is great. Like, I I I just, you know, that that analogy 415 00:24:32,000 --> 00:24:34,877 should just make clear how I see this 416 00:24:34,960 --> 00:24:39,997 trade-off that I just don't think it makes sense. Um and and we've done a lot 417 00:24:40,080 --> 00:24:43,757 of more aggressive stuff toward, you know, toward towards towards China, 418 00:24:43,840 --> 00:24:46,557 China and other players that that I think is much less effective than this 419 00:24:46,640 --> 00:24:50,317 this one this one measure. >> One more area for me and then I hope 420 00:24:50,400 --> 00:24:51,997 we'll have time for a question or two. 421 00:24:52,080 --> 00:24:54,557 The other area of potential risk that 422 00:24:54,640 --> 00:24:59,437 doomers worry about is a kind of all powerful malign AI. Um, and I think 423 00:24:59,520 --> 00:25:02,877 you've both been somewhat skeptical of the doomer approach, but in the last 424 00:25:02,960 --> 00:25:05,677 year we have seen, you know, these 425 00:25:05,760 --> 00:25:07,357 models showing themselves to be capable 426 00:25:07,440 --> 00:25:11,117 of deception, duplicity. Uh, do you 427 00:25:11,200 --> 00:25:14,797 think that do you think differently about that risk now than you did a year 428 00:25:14,880 --> 00:25:19,197 ago? And is there something about the way the models are evolving that we 429 00:25:19,280 --> 00:25:21,117 should put a little bit more concern on that? 430 00:25:21,200 --> 00:25:24,237 >> Yeah, I mean, you know, since since the beginning of Enthropic, we've kind of 431 00:25:24,320 --> 00:25:27,997 thought about this risk. I mean, you know, our our our research at the 432 00:25:28,080 --> 00:25:31,597 beginning of it was very theoretical, right? You know, we pioneered this idea 433 00:25:31,680 --> 00:25:35,277 of mechanistic interpretability, which is looking inside the model and and 434 00:25:35,360 --> 00:25:39,037 trying to understand looking inside its brain, trying to understand why it does 435 00:25:39,120 --> 00:25:43,357 what it does as it, you know, as as human neuroscientists, which we actually 436 00:25:43,440 --> 00:25:48,397 both have background in, um, try try to understand try to understand the brain. 437 00:25:48,480 --> 00:25:52,397 And I think as time has gone on, we've we've increasingly documented the you 438 00:25:52,480 --> 00:25:55,917 know bad behaviors of the models when they emerge and are now working on 439 00:25:56,000 --> 00:26:00,397 trying to address them with mechanistic interpretability. So I you know I think 440 00:26:00,480 --> 00:26:03,997 uh you know I I've always been concerned about these these risks. I've talked to 441 00:26:04,080 --> 00:26:08,557 Demis many times. I think he has also been um concerned about these risks. I 442 00:26:08,640 --> 00:26:12,717 think I have definitely been and I I I would guess Demis as well although I'll 443 00:26:12,800 --> 00:26:17,277 let him speak for himself skeptical of of dumerism which is you know we're 444 00:26:17,360 --> 00:26:21,597 doomed. there's nothing we can do or this is the most likely outcome. I think 445 00:26:21,680 --> 00:26:24,237 this is a risk. This is a risk that if 446 00:26:24,320 --> 00:26:26,477 we work all work together, we can 447 00:26:26,560 --> 00:26:29,357 address we can learn through science to 448 00:26:29,440 --> 00:26:31,837 properly, you know, control and and 449 00:26:31,920 --> 00:26:33,597 direct these creations that we're 450 00:26:33,680 --> 00:26:36,637 building. But if we build them poorly, 451 00:26:36,720 --> 00:26:40,157 if we go, you know, if if if we're all 452 00:26:40,240 --> 00:26:43,837 racing and we go so fast that there's no guardrails, then I think there is risk 453 00:26:43,920 --> 00:26:46,397 of something going wrong. So I'm going to give you a chance to answer that in 454 00:26:46,480 --> 00:26:49,997 the context of of a slightly broader question which is over the past year 455 00:26:50,080 --> 00:26:53,117 have you grown more confident of the 456 00:26:53,200 --> 00:26:57,117 upside potential of the technology science all of the areas that you have 457 00:26:57,200 --> 00:27:00,877 talked about a lot or are you more worried about the risks that we've been 458 00:27:00,960 --> 00:27:04,317 discussing I've been working on this for 20 plus 459 00:27:04,400 --> 00:27:08,397 years so we we already knew look the reason I've spent my whole career on AI 460 00:27:08,480 --> 00:27:13,197 is is the upsides of solving basically the ultimate tool for science and 461 00:27:13,280 --> 00:27:14,877 understanding the universal around us. 462 00:27:14,960 --> 00:27:18,637 I've I've sort of been obsessed with that since a kid and and and building AI 463 00:27:18,720 --> 00:27:21,517 is the you know should be the ultimate tool for that if we do it in the right 464 00:27:21,600 --> 00:27:25,037 way. The risks also we've been thinking about since the start at least the start 465 00:27:25,120 --> 00:27:29,757 of deep mind 15 years ago and um we kind of sort of foresaw that if you got the 466 00:27:29,840 --> 00:27:33,917 upsides it's a dual purpose technology so it could be repurposed by say bad 467 00:27:34,000 --> 00:27:37,277 actors for harmful ends. So we've needed to think about that all the way through 468 00:27:37,360 --> 00:27:39,037 but I'm a big believer in human 469 00:27:39,120 --> 00:27:41,677 ingenuity. Um but the question is having 470 00:27:41,760 --> 00:27:44,957 the time and the focus and all the best 471 00:27:45,040 --> 00:27:48,077 minds collaborating on it to solve these 472 00:27:48,160 --> 00:27:52,077 problems. I'm sure if we had that we would solve the technical risk problem. 473 00:27:52,160 --> 00:27:55,517 It may be we don't have that and then that will introduce risk because we'll 474 00:27:55,600 --> 00:27:59,037 be sort of it'll be fragmented. There'll be different projects and people be 475 00:27:59,120 --> 00:28:02,397 racing each other then it's much harder to make sure you know these systems that 476 00:28:02,480 --> 00:28:04,877 we produce will be technically safe. But 477 00:28:04,960 --> 00:28:09,437 I I feel like that's a very tractable uh problem if we have the time if we have 478 00:28:09,520 --> 00:28:13,517 the time and space. I want to make sure there's one question gentlemen. Keep it 479 00:28:13,600 --> 00:28:16,557 very short because we've got literally two minutes. 480 00:28:16,640 --> 00:28:19,597 >> Thanks for Hello. >> Yeah. No speak. 481 00:28:19,680 --> 00:28:20,957 >> Thanks very much. I'm Philip, co-founder 482 00:28:21,040 --> 00:28:25,037 of StarCloud building data centers in space. Um I wanted to ask a slightly 483 00:28:25,120 --> 00:28:29,037 philosophical question. The sort of strongest argument for doomerism to me 484 00:28:29,120 --> 00:28:31,757 is the firmy paradox, the idea that we don't see intelligent life in our 485 00:28:31,840 --> 00:28:34,237 galaxy. I was wondering if you guys have any thoughts. Yeah, I've thought a lot 486 00:28:34,320 --> 00:28:38,397 about that. That can't be the reason because we we we should see all the AIS 487 00:28:38,480 --> 00:28:43,117 that have So, just so everyone know the idea is well, it's sort of unclear why 488 00:28:43,200 --> 00:28:46,477 that would happen, right? So, if if the reason there's a Firmeny paradox, there 489 00:28:46,560 --> 00:28:50,237 are no aliens cuz they get taken out by their own technology. We should be 490 00:28:50,320 --> 00:28:54,157 seeing paper clips coming towards us from some part of the galaxy. And 491 00:28:54,240 --> 00:28:57,597 apparently, we don't. We don't see any structures. Dyson sphere is nothing 492 00:28:57,680 --> 00:29:02,317 whether they're AI or natur or sort of biological. So to me um there has to be 493 00:29:02,400 --> 00:29:04,637 a different answer to FMY powders. I have my own theories about that, but 494 00:29:04,720 --> 00:29:06,397 it's out of scope for the next minute. 495 00:29:06,480 --> 00:29:09,357 But um you know I I just feel like uh 496 00:29:09,440 --> 00:29:13,837 that that I my prediction my feeling is that we're past the great filter. It was 497 00:29:13,920 --> 00:29:17,757 probably multisellular life if I would have to guess. It was incredibly hard 498 00:29:17,840 --> 00:29:22,397 for for biology to evolve that. Um so we're on you know there isn't a comfort 499 00:29:22,480 --> 00:29:25,917 of like what's going to happen next. I think it's for us to write as humanity 500 00:29:26,000 --> 00:29:28,157 what's going to happen next. >> This this could be a great discussion 501 00:29:28,240 --> 00:29:31,917 but is out of scope for the next 36 sessions. But what isn't 15 seconds each 502 00:29:32,000 --> 00:29:36,477 what when we meet again I hope next year uh the three of us which I would love uh 503 00:29:36,560 --> 00:29:38,557 what will have changed by then 504 00:29:38,640 --> 00:29:41,597 >> I well I think the biggest thing to 505 00:29:41,680 --> 00:29:44,877 watch is this issue of AI systems 506 00:29:44,960 --> 00:29:47,757 building AI systems how that goes whe 507 00:29:47,840 --> 00:29:49,277 that whether that goes one way or 508 00:29:49,360 --> 00:29:52,797 another that that will determine you 509 00:29:52,880 --> 00:29:55,117 know whether it's a few more years until 510 00:29:55,200 --> 00:29:58,717 we get there or or if we have you know 511 00:29:58,800 --> 00:30:03,437 you know if if we have wonders and and a great emergency in front of us that we 512 00:30:03,520 --> 00:30:05,677 have to face. >> AI systems, building AI system. 513 00:30:05,760 --> 00:30:07,197 >> I agree on that. So, we're we're keeping 514 00:30:07,280 --> 00:30:11,357 close touch about that. Um but also I think um outside of that, I think there 515 00:30:11,440 --> 00:30:15,117 are other interesting uh uh uh ideas being researched like world models, 516 00:30:15,200 --> 00:30:18,317 continual learning. These are the things I think that will need to be cracked. If 517 00:30:18,400 --> 00:30:22,157 self-improvement doesn't sort of deliver the goods on its own, then we'll need 518 00:30:22,240 --> 00:30:25,757 these other things to work. And then I think things like robotics may have its 519 00:30:25,840 --> 00:30:29,037 sort of breakout moment. But maybe on the basis of what you've just said, we 520 00:30:29,120 --> 00:30:31,757 should all be hoping that it does take you a little bit longer and indeed 521 00:30:31,840 --> 00:30:33,917 everybody else to give us >> I would prefer that. I think that would 522 00:30:34,000 --> 00:30:35,117 be better for the world. 523 00:30:35,200 --> 00:30:41,240 >> Well, you guys could do something about that. Thank you both very much.