WEBVTT 00:00:00.090 --> 00:00:03.847 What is coordination currently look like between OpenAI and anthropic? 00:00:03.930 --> 00:00:07.580 Because I can't imagine is sound like constantly on the phone with Dario. 00:00:07.680 --> 00:00:10.950 Somehow I doubt it. But is there like the equivalent of a, 00:00:11.130 --> 00:00:15.210 you know, a red phone? Yeah. Well, look, we we all know each other personally. 00:00:15.420 --> 00:00:17.347 Many of us worked together in past lives. 00:00:17.430 --> 00:00:20.947 So I think that there's actually a lot of social connections between the labs. 00:00:21.030 --> 00:00:23.287 If you look at. There was another open letter that came 00:00:23.370 --> 00:00:26.827 out recently that we helped drive and that we in anthropic were both 00:00:26.910 --> 00:00:30.037 signatories on, which was about security, saying that we're in a 00:00:30.120 --> 00:00:34.290 cybersecurity moment, that everyone kind of needs to take this information of 00:00:34.530 --> 00:00:38.917 where cyber is going to be, and even just six months and needs to act a day 00:00:39.000 --> 00:00:43.327 proactively to defend themselves and actually talking behind the scenes to 00:00:43.410 --> 00:00:44.887 say, hey, we're actually aligned on this. 00:00:44.970 --> 00:00:47.077 This is something that's about it's bigger than any company. 00:00:47.160 --> 00:00:51.217 This is about the industry in the world. And can we come together as one voice to 00:00:51.300 --> 00:00:53.130 say, this is important? It happened 00:00:54.540 --> 00:00:58.870 to. Me. 00:00:59.580 --> 00:01:03.277 Hello and welcome to another episode of the Odd Lots podcast. 00:01:03.360 --> 00:01:06.547 I'm Joe Weisenthal and I'm Tracy Alloway Tracy. 00:01:06.630 --> 00:01:10.747 I don't know if I've ever, uh, said it. Um, the podcast, uh, we've talked a bit 00:01:10.830 --> 00:01:13.410 about my my vibe, coding adventures, etc.. 00:01:13.860 --> 00:01:17.940 Um, no, you've never said, oh no. I said, you know, I recently switched 00:01:18.030 --> 00:01:21.397 from, um, Cloud Code to Codex. This is big news. 00:01:21.480 --> 00:01:24.307 It is it is kind of big news, I think. Um. 00:01:24.390 --> 00:01:28.777 Oh, so bold of you to declare your allegiance, uh, to the public on the 00:01:28.860 --> 00:01:32.197 podcast. Well, you know, the I don't have allegiance. And you know what? 00:01:32.280 --> 00:01:36.157 It could switch again the way, like, you know, I guess one of the some stories of 00:01:36.240 --> 00:01:40.417 AI is like how easy it is to switch from time to time from one model to another. 00:01:40.500 --> 00:01:43.807 So maybe it's revealing and talks about some of the challenges the these 00:01:43.890 --> 00:01:49.267 businesses, that it was so easy. I found that Claude speak, you know, 00:01:49.350 --> 00:01:51.950 that I found it like a little bit hard to work with. 00:01:52.080 --> 00:01:57.037 I'm not capable enough to like and understand, like advanced engineering 00:01:57.120 --> 00:02:01.577 practices were like, oh, I'm migrating a code base from or like, you know, 00:02:01.660 --> 00:02:05.327 translating this into rust or whatever. So the fact, I don't know, I found like 00:02:05.410 --> 00:02:09.767 OpenAI to be like the pros, to be clear, and therefore to work with as, like a 00:02:09.850 --> 00:02:12.587 completely non-technical person such as myself. 00:02:12.670 --> 00:02:16.457 That's really interesting. I mean, one thing you said, it's hard to 00:02:16.540 --> 00:02:19.427 keep up with the models, right? And whatever you're using today might 00:02:19.510 --> 00:02:22.967 not be the one that you're using in a week from now. 00:02:23.050 --> 00:02:25.637 Yeah. And at the same time, everyone is 00:02:25.720 --> 00:02:29.777 talking about how fast the development is going and all the risks that it 00:02:29.860 --> 00:02:33.317 poses, right? Yes. And we are recording this September 10th. 00:02:33.400 --> 00:02:37.300 It felt like something broke through in the last couple of days where suddenly 00:02:37.480 --> 00:02:42.707 everyone is very keen on risk. The one thing with Codex and I guess 00:02:42.790 --> 00:02:47.957 called code is now every once in a while to get this pop up and it'll say the 00:02:48.040 --> 00:02:51.467 agent needs to connect to the internet in order to do this task, and I have to 00:02:51.550 --> 00:02:52.607 give it a permission. Yeah. 00:02:52.690 --> 00:02:55.487 And normally I'm just like click, click click like yes, yes, yes, yes guys. 00:02:55.570 --> 00:02:58.547 And I still do that. Why is there I still just click yes, 00:02:58.630 --> 00:03:00.977 yes, yes. But it makes you think. 00:03:01.060 --> 00:03:04.360 But it makes you think for a second before you click. 00:03:04.570 --> 00:03:06.887 I think for one second, yeah, I'll have more than click. 00:03:06.970 --> 00:03:10.217 And, um, we were in Jackson Hole a couple of weeks ago and that's of 00:03:10.300 --> 00:03:17.207 course, when the, the meter report broke about, um, the, uh, OpenAI hugging face 00:03:17.290 --> 00:03:20.057 attack. And I think since then, the anxiety 00:03:20.140 --> 00:03:24.197 about rogue AI, whatever you want to call it, has clearly snowballed. 00:03:24.280 --> 00:03:27.797 Mhm. And it's just, you know, it's yes. 00:03:27.880 --> 00:03:29.417 It feels like it's only getting bigger and bigger. 00:03:29.500 --> 00:03:33.940 And it's already been an industry that's been shot through with risk and anxiety. 00:03:34.150 --> 00:03:39.047 And now there's sort of one thing that for many people in AI has been something 00:03:39.130 --> 00:03:43.067 they've talked about for 20 plus years before there was an AI industry to speak 00:03:43.150 --> 00:03:45.677 of. This idea of like misaligned quote, 00:03:45.760 --> 00:03:49.540 rogue models is starting to become top of mind, a reality. 00:03:49.660 --> 00:03:54.887 So this is our chance to ask a person directly involved in AI development 00:03:54.970 --> 00:03:57.370 about their respective, I guess, anxiety. 00:03:57.640 --> 00:03:59.567 Yeah. And what, uh, what can that what can be 00:03:59.650 --> 00:04:02.897 done about that? Anyway, we literally have the perfect 00:04:02.980 --> 00:04:04.907 guest today. We're going to be speaking, of course, 00:04:04.990 --> 00:04:08.447 with Greg Brockman. He is the co-founder and president of 00:04:08.530 --> 00:04:12.377 OpenAI here with us in studio. So, Greg, thank you so much for coming 00:04:12.460 --> 00:04:14.290 on our lot. Thank you for having me. 00:04:14.920 --> 00:04:17.680 Um, so many different ways we could start. 00:04:17.770 --> 00:04:19.937 But here's something that I'm very curious about. 00:04:20.020 --> 00:04:24.197 Um, let's just start with, let's just jump right into the post hugging face 00:04:24.280 --> 00:04:27.790 environment and all of that. Are you? 00:04:28.030 --> 00:04:33.167 Is it OpenAI I like as an organization. So okay, like an agent breaks out of, 00:04:33.250 --> 00:04:36.430 uh, sandbox, not the first time it's happened, etc. 00:04:36.730 --> 00:04:40.547 it exhibits the sort of emergent behavior and then, you know, does 00:04:40.630 --> 00:04:45.017 something that people is a hack. When you think about the development of 00:04:45.100 --> 00:04:51.017 models and this incident and so forth, are you able to sort of diagnose the 00:04:51.100 --> 00:04:54.797 path or, sorry, the past and say like, you know what, like here's something 00:04:54.880 --> 00:04:58.457 that the models did that we don't like, or that people don't think it's good. 00:04:58.540 --> 00:05:02.327 And are you able to like, if you like, look back, look back, whether it's in 00:05:02.410 --> 00:05:05.777 model training or post training, reinforcement learning, whatever I say, 00:05:05.860 --> 00:05:10.247 like this is where there was some some branch that went wrong such that this 00:05:10.330 --> 00:05:15.347 behavior emerged. Yeah, I would say that the fact of many 00:05:15.430 --> 00:05:19.697 elements of the hugging face incident were not a surprise, not a mystery to 00:05:19.780 --> 00:05:21.827 us. For example, the fact that the agents 00:05:21.910 --> 00:05:25.067 were coordinating made. That's because the agents were trained. 00:05:25.150 --> 00:05:26.747 You trained them to be helpful. Exactly. 00:05:26.830 --> 00:05:29.387 And they were trained to coordinate to be a multi-agent system. 00:05:29.470 --> 00:05:31.607 And we've talked about this. It's a very useful property. 00:05:31.690 --> 00:05:32.990 Right. Makes them capable. 00:05:33.130 --> 00:05:37.247 But I think that the thing that was a surprise to us was the fact that the 00:05:37.330 --> 00:05:41.747 models had reached a level of capability where they were able to find that 00:05:41.830 --> 00:05:45.107 exploit in our sandbox environment. Right. 00:05:45.190 --> 00:05:49.367 Move through our research environment and then also capable enough to find 00:05:49.450 --> 00:05:52.997 exploits in hugging faces, production infrastructure and move through that. 00:05:53.080 --> 00:05:56.317 But I think that a lot of the facts of what the models were capable of. 00:05:56.400 --> 00:05:57.927 That was that was clear to us. Right. 00:05:58.010 --> 00:06:01.647 So I don't think that there were surprises there for the way in which the 00:06:01.730 --> 00:06:04.977 capability had unfolded, but just really realizing that we needed to uplevel 00:06:05.060 --> 00:06:08.997 where we were in terms of our safety and security standards like that for us, was 00:06:09.080 --> 00:06:12.357 that was the real watershed. But just to be clear, setting aside the 00:06:12.440 --> 00:06:16.857 technical capabilities, you know, ideally we would have models that run 00:06:16.940 --> 00:06:21.987 into a wall and then don't try to find the crack in the wall because or at 00:06:22.070 --> 00:06:25.340 least when that crack would be a crime when a person did it. 00:06:25.730 --> 00:06:30.957 Is there a way to identify the moment and training such that the models 00:06:31.040 --> 00:06:37.147 reasoned, oh, this would be okay. So this model that did that, the hugging 00:06:37.230 --> 00:06:40.377 face incident actually had not gone through our alignment training yet. 00:06:40.460 --> 00:06:45.807 Huh? Right. And it had lowered safeguards. And so that the reason we we were 00:06:45.890 --> 00:06:48.387 proceeding with this was because it was in a sandbox. 00:06:48.470 --> 00:06:53.667 And I think that our realization is that we need to pull back Earlier into our 00:06:53.750 --> 00:06:56.210 development process and training monitoring. 00:06:57.470 --> 00:06:59.817 There's always going to be a phase at which you do alignment, but you need to 00:06:59.900 --> 00:07:03.680 think about alignment as a core part of even this, this earlier phase. 00:07:03.950 --> 00:07:06.477 And if you look at where we've been, we've always been very focused on the 00:07:06.560 --> 00:07:10.260 deployment side, right, of really thinking about deployment safety, having 00:07:10.430 --> 00:07:14.457 really good test and governance and all those things, and the fact that we're 00:07:14.540 --> 00:07:17.037 now at a point where even for development, that's important. 00:07:17.120 --> 00:07:20.697 We always knew it would happen. The fact is now that has been a huge 00:07:20.780 --> 00:07:24.057 watershed for us into something we've really risen to the occasion for. 00:07:24.140 --> 00:07:28.437 Can I ask what is potentially a very dumb question with an obvious answer? 00:07:28.520 --> 00:07:34.647 But we hear about AI angst of all sorts, and in particular when it comes to 00:07:34.730 --> 00:07:39.117 cybersecurity. Why do we run training exercises where 00:07:39.200 --> 00:07:42.837 we ask AI to hack into various systems at all? 00:07:42.920 --> 00:07:46.587 Like why is this necessary for you? So I think it's very important to 00:07:46.670 --> 00:07:49.640 understand where we are with capabilities broadly. 00:07:50.060 --> 00:07:53.937 And I think that depending on the evaluation, depending on what you expect 00:07:54.020 --> 00:07:56.697 from the model, you need to have safeguards that are commensurate with 00:07:56.780 --> 00:07:58.550 that. And we think about this both, 00:07:59.750 --> 00:08:01.917 even just over these past couple of weeks, really starting to think about 00:08:02.000 --> 00:08:04.587 during the development process. You're always going to be testing 00:08:04.670 --> 00:08:07.917 different capabilities. And some of these capabilities are dual 00:08:08.000 --> 00:08:11.727 use, something like vulnerabilities that if those are in the hands of threat 00:08:11.810 --> 00:08:13.797 actors, that's something that could be could be negative. 00:08:13.880 --> 00:08:17.787 But if you can find vulnerabilities in your own code base, you can fix them, 00:08:17.870 --> 00:08:18.717 right? You can uplevel. 00:08:18.800 --> 00:08:22.047 And we actually think that it's a very important capability for AIS to exhibit 00:08:22.130 --> 00:08:25.910 and be put in defenders hands. And so in order to know where we are. 00:08:26.240 --> 00:08:29.697 Evaluations are very, very key. And I'll say one other thing on this, 00:08:29.780 --> 00:08:33.447 which is that I think that hugging face, there's two aspects to it that I think 00:08:33.530 --> 00:08:37.077 are our learning opportunities, right. That there's one that I think is really 00:08:37.160 --> 00:08:41.217 about us and the realization that we are at a point where safety, security, 00:08:41.300 --> 00:08:44.700 alignment, during evaluation and development, we need to uplevel it. 00:08:44.990 --> 00:08:46.687 That's something we've taken very seriously. 00:08:46.770 --> 00:08:50.907 We've slowed down a number of runs, like we did a very painful retooling of a lot 00:08:50.990 --> 00:08:53.887 of our processes that that's that's one reaction. 00:08:53.970 --> 00:08:58.470 But the second thing is this information on what the models are capable of today, 00:08:58.770 --> 00:09:01.687 what can they do in the real world? And I think that when mythos came out 00:09:01.770 --> 00:09:06.937 over the summer, you kind of saw just sort of, you know, just sort of public 00:09:07.020 --> 00:09:09.607 blog post about this, but it didn't really see impact or you didn't really 00:09:09.690 --> 00:09:13.357 see a real world understanding of, well, what can these models really do? 00:09:13.440 --> 00:09:16.117 What are they capable of? I think that hugging face really showed 00:09:16.200 --> 00:09:19.027 today's models are capable of getting into a company's production 00:09:19.110 --> 00:09:23.077 infrastructure, and that's important because there will be many models with 00:09:23.160 --> 00:09:26.197 this kind of capability that will be produced by a number of different 00:09:26.280 --> 00:09:30.967 organizations across the world in maybe the next six months, maybe that time 00:09:31.050 --> 00:09:34.237 period a little bit last a little bit more, and we need to be prepared. 00:09:34.320 --> 00:09:38.317 Defenders need to know we got this extra information, almost like this time 00:09:38.400 --> 00:09:41.287 traveler came back from six months in the future and said, here's what's going 00:09:41.370 --> 00:09:44.460 to be possible, and you have an opportunity to be ready. 00:09:44.550 --> 00:09:50.467 Um, you mentioned slowing down, um, some of some of your work in the wake of 00:09:50.550 --> 00:09:56.167 this, focusing more on this idea of like pacing development has become a sort of 00:09:56.250 --> 00:10:00.857 buzzword or watchword in the industry. And there are a lot of employees in both 00:10:00.940 --> 00:10:02.617 of the labs. I think it was like an open letter that 00:10:02.700 --> 00:10:07.387 was signed by a bunch of people across the industry about, um, pacing. 00:10:07.470 --> 00:10:13.290 But it's a competitive capitalist environment and our investors, etc. 00:10:13.770 --> 00:10:18.637 talk to us about, I don't know if it's game theory or whatever, but like, what 00:10:18.720 --> 00:10:23.557 is your view on? Is it possible for let's set aside China 00:10:23.640 --> 00:10:26.587 for one second, because then that's a whole separate thing just in the 00:10:26.670 --> 00:10:30.517 American companies. Do you think that something can be 00:10:30.600 --> 00:10:34.290 reached where you trust each other, such that you can have this sort of 00:10:34.440 --> 00:10:38.887 coordinated pacing to avoid a race to the bottom, where it's all about getting 00:10:38.970 --> 00:10:42.840 there faster, even if it means sacrificing some security questions? 00:10:42.990 --> 00:10:46.627 I absolutely believe it's possible, and I see I think it's going to take steps 00:10:46.710 --> 00:10:49.117 to get there. Yeah, but this is something we've been 00:10:49.200 --> 00:10:53.137 really investing in and thinking about and really even thinking about for 00:10:53.220 --> 00:10:56.460 almost a decade, right. It's always been kind of clear that 00:10:56.610 --> 00:10:59.797 they're going to go through a commercial phase that's really about competition. 00:10:59.880 --> 00:11:03.787 But there will be a phase where the technology itself, it's just so much 00:11:03.870 --> 00:11:06.937 bigger than any person, any company, even any country, right? 00:11:07.020 --> 00:11:10.187 It's really about humanity as a whole that's in our mission, right? 00:11:10.270 --> 00:11:13.777 We want to benefit humanity as a whole. And so working together with others to 00:11:13.860 --> 00:11:18.397 really think about how does this technology, its capability increase, how 00:11:18.480 --> 00:11:21.637 do we make sure that we have safety cases that are really laid out so that 00:11:21.720 --> 00:11:25.807 we know that, that it is something that is beneficial, that we're able to have 00:11:25.890 --> 00:11:28.597 the appropriate controls and the right oversight, the right monitor bill, the 00:11:28.680 --> 00:11:33.330 all of those sort of technical systems in place. 00:11:33.960 --> 00:11:35.617 And that's not about any one company, right. 00:11:35.700 --> 00:11:37.957 It's really about how the whole field evolves. 00:11:38.040 --> 00:11:40.297 Now, I think that there's work to be done here. 00:11:40.380 --> 00:11:44.230 And I think that the open letter is a good example of just a first baby step. 00:11:44.340 --> 00:11:46.807 And that was actually something we were very involved in helping craft that 00:11:46.890 --> 00:11:48.467 language. That was actually a pretty collaborative 00:11:48.550 --> 00:11:51.100 effort, and I think we were very happy to see that. 00:11:51.250 --> 00:11:52.787 That got some some real momentum and broke through. 00:11:52.870 --> 00:11:57.760 And in some way, even the term pacing was a very deliberate choice. 00:11:58.840 --> 00:12:02.657 What does coordination currently look like between OpenAI and anthropic? 00:12:02.740 --> 00:12:06.400 Because I can't imagine Sam like constantly on the phone with Dario. 00:12:06.490 --> 00:12:09.760 Somehow I doubt it. But is there like the equivalent of a, 00:12:09.970 --> 00:12:11.410 you know, a red phone? Yeah. 00:12:11.590 --> 00:12:14.177 Well, look, we we all know each other personally, right? 00:12:14.260 --> 00:12:16.157 Many of us worked together in past lives. 00:12:16.240 --> 00:12:19.787 So I think that there's actually a lot of social connections between the labs. 00:12:19.870 --> 00:12:22.967 If you look at there was another open letter that came out recently that we 00:12:23.050 --> 00:12:27.197 helped drive and that we in anthropic were both signatories on, which was 00:12:27.280 --> 00:12:31.547 about security, saying that we're in a cybersecurity moment, that everyone kind 00:12:31.630 --> 00:12:35.927 of needs to take this information of where cyber is going to be in even just 00:12:36.010 --> 00:12:39.940 six months and needs to act today proactively to defend themselves and 00:12:40.270 --> 00:12:43.487 actually talking behind the scenes to say, hey, we're actually aligned on 00:12:43.570 --> 00:12:45.077 this. This is something that's about it's 00:12:45.160 --> 00:12:47.597 bigger than any company. This is about the industry in the world. 00:12:47.680 --> 00:12:51.220 And can we come together as one voice to say, this is important? 00:12:51.400 --> 00:12:53.957 It happened. And that that, you know, you pick up the 00:12:54.040 --> 00:12:56.267 phone. So I think that there's there's personal 00:12:56.350 --> 00:12:59.567 relationship between a few different exacts that I think have been forming. 00:12:59.650 --> 00:13:02.507 We're building trust. And I think that the way I view this is 00:13:02.590 --> 00:13:07.097 that because we are competitors and that that will remain true, but we are 00:13:07.180 --> 00:13:09.317 aligned on wanting to do the right thing for the world. 00:13:09.400 --> 00:13:13.997 It always means that you really want for these coordination actions to really 00:13:14.080 --> 00:13:18.287 hone in on the what's the common interest? 00:13:18.370 --> 00:13:21.167 What's the thing that's like really about good for the world and that 00:13:21.250 --> 00:13:24.617 neither side is really trying to benefit themselves differentially or something 00:13:24.700 --> 00:13:26.837 like that. It's always a little backdrop of trying 00:13:26.920 --> 00:13:30.047 to make sure that that's like the spirit in which things are offered. 00:13:30.130 --> 00:13:33.737 And by the way, I think that my view of how these things should go is that it's 00:13:33.820 --> 00:13:38.897 never fully about, you know, there's always an element of trust, right? 00:13:38.980 --> 00:13:40.277 It's always about trust building. Right? 00:13:40.360 --> 00:13:43.607 Because you can always sort of imagine ways that, like maybe the other party 00:13:43.690 --> 00:13:47.297 would try to, you know, take, take this other action that is not just closing. 00:13:47.380 --> 00:13:50.987 I think that a lot of this is just about intent and about the fact that as you 00:13:51.070 --> 00:13:54.167 start to do small things together, that actually sets the groundwork for you to 00:13:54.250 --> 00:13:58.247 do big things together in the future. You know, if you and anthropic were oil 00:13:58.330 --> 00:14:03.977 companies and you were talking publicly about slowing down the pace of drilling 00:14:04.060 --> 00:14:07.397 or slowing down the pace of pumping, people say this is an antitrust. 00:14:07.480 --> 00:14:11.627 This is an antitrust violation. This is like, totally are you is there 00:14:11.710 --> 00:14:13.810 an issue that comes up? Would you. 00:14:14.080 --> 00:14:19.187 Should there be a carve out for AI companies such that they can formally 00:14:19.270 --> 00:14:24.707 say, we are all going to slow down and our coordinate our behaviors together in 00:14:24.790 --> 00:14:28.157 a way that in other contexts people would say, this is ridiculous. 00:14:28.240 --> 00:14:31.890 You can't publicly agree on all slowing down the production of something. 00:14:32.170 --> 00:14:35.297 Well, I'm not a lawyer, so I can't comment on this specific legalities, but 00:14:35.380 --> 00:14:39.220 I would say that as a general matter, that being able to freely talk about 00:14:39.550 --> 00:14:43.300 coordination on safety, security, like doing the right thing for the world, 00:14:43.480 --> 00:14:47.297 that seems like a very good thing to me. And I think that, again, everyone's 00:14:47.380 --> 00:14:50.357 interests are aligned here in terms of wanting to do the right thing for the 00:14:50.440 --> 00:14:52.217 world. So I think to the extent that there are 00:14:52.300 --> 00:14:56.057 legal barriers, I do think it would be very good to to help make it smooth for 00:14:56.140 --> 00:14:58.757 the for us to be able to work together on these issues. 00:14:58.840 --> 00:15:00.677 And again, it's more than just the frontier lives, right. 00:15:00.760 --> 00:15:03.137 It's really about working together with cloud providers. 00:15:03.220 --> 00:15:06.827 It's working with government. It's working with, uh, just all the 00:15:06.910 --> 00:15:10.607 different players in the ecosystem. And I think that this is, again, just 00:15:10.690 --> 00:15:14.367 one area that is bigger than any one company. 00:15:14.450 --> 00:15:18.797 And that's it should be. When you talk about pacing or slowing 00:15:18.880 --> 00:15:25.397 down, do you have a sense of like, what exact slowdown is acceptable to you? 00:15:25.480 --> 00:15:28.667 Because I imagine this is where the coordination problem comes in, where 00:15:28.750 --> 00:15:32.987 like how you measure development, what counts as development. 00:15:33.070 --> 00:15:36.617 One person's slow down might be different to another person. 00:15:36.700 --> 00:15:40.337 Slow down. Do you have a sense of specificity of 00:15:40.420 --> 00:15:44.517 what you're talking about here? So I do. 00:15:44.600 --> 00:15:48.867 And I think that there's a high level picture which is 00:15:48.950 --> 00:15:53.540 increasingly coming into view and into specifics and into into details. 00:15:53.900 --> 00:15:56.097 And I think that the framing here really matters. 00:15:56.180 --> 00:15:59.367 And it's something, again, we've thought about for a decade, but it's very 00:15:59.450 --> 00:16:02.217 different once you start to really start to see line of Sight. 00:16:02.300 --> 00:16:05.637 And to some extent, we're not there yet. We're starting to see systems where 00:16:05.720 --> 00:16:08.307 they're very capable. It's very clear that we need to have 00:16:08.390 --> 00:16:12.387 good safety practices around what we're doing 00:16:12.470 --> 00:16:14.667 right now. Things like train of thought, monitor 00:16:14.750 --> 00:16:16.797 ability. That's something that works very well in 00:16:16.880 --> 00:16:18.987 this moment as we move to more capable models. 00:16:19.070 --> 00:16:23.007 As we've talked about that those models start to be something like Astro starts 00:16:23.090 --> 00:16:29.037 to be able to I sort of have more, you know, like it needs the chain of thought 00:16:29.120 --> 00:16:32.157 last because the more capable model. And so you're going to need other 00:16:32.240 --> 00:16:34.617 answers, other ways of how you're going to monitor it. 00:16:34.700 --> 00:16:37.437 But monitoring absolutely important. Right. 00:16:37.520 --> 00:16:40.580 And so really thinking about what are the standards. 00:16:40.760 --> 00:16:43.047 And that's again invariant. It's not about any one company's 00:16:43.130 --> 00:16:45.237 technology. It's really about just how this 00:16:45.320 --> 00:16:49.527 technology needs to evolve and how we can ensure that it has the right safety 00:16:49.610 --> 00:16:53.997 properties to it. And so my view is that it's important 00:16:54.080 --> 00:17:00.680 not to conflate too much. The moment that you're in today with the 00:17:00.770 --> 00:17:03.020 standard you're going to need for the future. 00:17:03.170 --> 00:17:07.167 And but also you can work backwards where it is very clear that as we move 00:17:07.250 --> 00:17:10.347 to a higher level of capability, you need to be able to make a safety case, 00:17:10.430 --> 00:17:12.237 right. You need to be able to really say that 00:17:12.320 --> 00:17:14.157 you do. You have good understanding of 00:17:14.240 --> 00:17:17.997 controllability and all of these things that that I think people are feeling 00:17:18.080 --> 00:17:20.187 anxiety about. Again, it's something we also feel 00:17:20.270 --> 00:17:23.787 anxiety about the way we think about a lot that we care a lot about as core to 00:17:23.870 --> 00:17:25.887 our mission. Part of the reason we started this place 00:17:25.970 --> 00:17:29.517 is that we want to help this technology going in a more positive direction. 00:17:29.600 --> 00:17:35.667 And so I think that that my view is that you want to have a shared and as 00:17:35.750 --> 00:17:38.517 objective as possible, whether it's based on evals, there's always going to 00:17:38.600 --> 00:17:42.207 be some element of subjectivity. Maybe you want third party auditors that 00:17:42.290 --> 00:17:46.497 are able to evaluate whether the standards that are defined by Frontier 00:17:46.580 --> 00:17:49.737 Labs, the people who are there, and the technical details were always going to 00:17:49.820 --> 00:17:54.337 be the most calibrated on kind of how do you still make forward progress and ways 00:17:54.420 --> 00:17:58.827 to deliver those benefits, while also being able to to have the appropriate 00:17:58.910 --> 00:18:04.227 guarantees and in safety cases, um, and then to have that kind of observability 00:18:04.310 --> 00:18:07.460 and maybe ultimately enforceability of some kind. 00:18:07.730 --> 00:18:10.317 Is that something that is just an industry voluntary standard? 00:18:10.400 --> 00:18:12.897 Is it something that's national? Is it something that's international? 00:18:12.980 --> 00:18:17.060 And how do you really get to that point that it is a humanity scale endeavor? 00:18:17.570 --> 00:18:21.807 And I think it is important to also recognize that what we're talking about, 00:18:21.890 --> 00:18:24.627 like, I think that a lot of people have this reaction of saying, well, am I 00:18:24.710 --> 00:18:26.967 going to be able to keep making my open source model? 00:18:27.050 --> 00:18:29.637 Am I going to be able to keep doing my hobbyist side project? 00:18:29.720 --> 00:18:32.907 And the answer should be absolutely yes. Like, that's not the kind of things 00:18:32.990 --> 00:18:35.037 where we're talking about. When we're talking about pacing, we're 00:18:35.120 --> 00:18:36.867 talking about pacing. We're really talking about the sprint 00:18:36.950 --> 00:18:38.757 here. We're talking about these massive 00:18:38.840 --> 00:18:41.737 supercomputers that are hundreds of billions of dollars worth of capital 00:18:41.820 --> 00:18:44.317 expenditure. It's a very small number of players, and 00:18:44.400 --> 00:18:49.687 it's a very different kind of scale. And proceeding with that responsibly and 00:18:49.770 --> 00:18:54.367 thoughtfully and in a way where we're thinking about how do we achieve the 00:18:54.450 --> 00:18:59.497 best outcomes and mitigate the risks. That, to me is something that's why we 00:18:59.580 --> 00:19:03.337 started OpenAI, why I got excited about this field and how we actually get to 00:19:03.420 --> 00:19:06.847 those benefits, how we actually cure diseases and all of these things that 00:19:06.930 --> 00:19:11.460 this technology so capable of. And I think it really starts with having 00:19:11.940 --> 00:19:14.857 good processes around the kinds of things we're talking about here. 00:19:14.940 --> 00:19:19.207 Just real quickly, since you mentioned third parties, um, obviously Mr.. 00:19:19.290 --> 00:19:22.980 Came in for a few days. Uh, they produced their report. 00:19:23.250 --> 00:19:27.217 We did an episode. Former OpenAI employee Miles Brundage, 00:19:27.300 --> 00:19:32.310 who now has his own, um, sort of third party, uh, aspiring to be an auditor. 00:19:32.460 --> 00:19:36.997 Would you support a law that said, um, there should always be the presence of 00:19:37.080 --> 00:19:40.477 auditors, because obviously just looking at the models after the released isn't 00:19:40.560 --> 00:19:45.990 enough, given, um, that, um, these, you know, some of these major disasters 00:19:46.170 --> 00:19:51.397 occurred pre-release models. Would you support, uh, law if they could 00:19:51.480 --> 00:19:56.707 be designed that would have essentially third party, uh, auditors embedded in 00:19:56.790 --> 00:20:00.637 all of the major frontier orgs. Well, look, I'll always say that the 00:20:00.720 --> 00:20:03.007 nuance really matters. The details really matter. 00:20:03.090 --> 00:20:06.960 So it's too hard to say at a high level whether any particular property is 00:20:07.120 --> 00:20:10.507 something that would work in practice. I do think a general direction of 00:20:10.590 --> 00:20:15.757 thinking about as you move toward these very capable models where you need to 00:20:15.840 --> 00:20:17.917 have safety cases, you need to lay out your standards. 00:20:18.000 --> 00:20:22.987 We're already having third party auditors for exactly that type of a 00:20:23.070 --> 00:20:26.017 process, right? If you look at how the evaluation 00:20:26.100 --> 00:20:29.047 processes for models over the past couple of months have, have, have 00:20:29.130 --> 00:20:32.737 evolved, right. We've had the Casey, which is a US 00:20:32.820 --> 00:20:36.487 organization that test models. Yeah, we have the UK, EC, these 00:20:36.570 --> 00:20:40.567 government Third party auditors test our models before I release them. 00:20:40.650 --> 00:20:45.037 But so what about like and like? Here's the thing that I thought about in 00:20:45.120 --> 00:20:49.687 the wake of reading, uh, about the hugging face incident and so forth. 00:20:49.770 --> 00:20:53.887 Like, yeah, you know, I think of OpenAI and 00:20:53.970 --> 00:20:57.690 anthropic as companies, but people like to call them, quote, labs. 00:20:57.900 --> 00:21:01.807 And when I think of labs, I think like, wow, like, you know, a lab, if we were 00:21:01.890 --> 00:21:06.007 talking about a bio lab or a chemical lab, we'd have just like all kinds of, 00:21:06.090 --> 00:21:10.087 like, rules and materials handling. And then maybe in this case, it would be 00:21:10.170 --> 00:21:14.947 like handling of the weights and whose computers they're on and who can access 00:21:15.030 --> 00:21:18.427 those computers and FDA approval before you go to market. 00:21:18.510 --> 00:21:20.407 Yeah. And all of these things like just like 00:21:20.490 --> 00:21:23.887 an or even a restaurant, someone would come in and they say like, oh, is the 00:21:23.970 --> 00:21:28.380 stove sufficiently far back from the wall, etc.. 00:21:28.530 --> 00:21:33.337 So not just like, okay, like measuring the capabilities of the risks of the 00:21:33.420 --> 00:21:40.217 models themselves, but I'm thinking like the auditing of the lab process, the way 00:21:40.300 --> 00:21:45.557 that any other lab probably has regular inspections of like, how safe is the lab 00:21:45.640 --> 00:21:47.777 environment? Well, well, this is what I would say has 00:21:47.860 --> 00:21:50.117 been changing just even over the past couple of weeks. 00:21:50.200 --> 00:21:52.780 Okay. Because the focus for, 00:21:53.800 --> 00:21:57.227 let's say, the past 12 months, 24 months, something like that has been on 00:21:57.310 --> 00:21:59.807 deployment. Yeah. Right. So you have a model, you've produced the 00:21:59.890 --> 00:22:02.040 model. You want to release it to the world. 00:22:03.220 --> 00:22:06.767 How do you decide whether you've you're at the right level there. 00:22:06.850 --> 00:22:09.557 And we have a preparedness framework anthropic as a responsible scaling 00:22:09.640 --> 00:22:12.460 policy. There's certain evaluations. 00:22:12.730 --> 00:22:15.497 Again there's Casey the UK. There's a whole ecosystem that's been 00:22:15.580 --> 00:22:18.257 built up around that. I think the thing that hugging Face has 00:22:18.340 --> 00:22:24.430 shown is that it is time to start pulling that process earlier. 00:22:24.520 --> 00:22:27.670 You need to think about this during development and evaluation. 00:22:27.790 --> 00:22:32.147 And just it makes sense as the models are more capable for exactly the kinds 00:22:32.230 --> 00:22:35.327 of reasons that you're describing for for analogies to other fields. 00:22:35.410 --> 00:22:39.407 Now, there's a lot of specifics for AI, and I think it's very important to 00:22:39.490 --> 00:22:42.227 really engage with those. As you design a regime, as you think 00:22:42.310 --> 00:22:46.427 about what is a set up that involves first party, third party that involves 00:22:46.510 --> 00:22:52.067 government that has the right interactions amongst academics and other 00:22:52.150 --> 00:22:57.677 other parties, early testers. It's really about like, you really have 00:22:57.760 --> 00:23:00.407 to engage with the way that the eye process is going. 00:23:00.490 --> 00:23:01.637 Right. And I think one thing that is the 00:23:01.720 --> 00:23:05.297 process, exactly one thing that is different, is that the ways there are 00:23:05.380 --> 00:23:09.527 some parts of AI that we understand well and are going to remain invariant, and 00:23:09.610 --> 00:23:12.287 to some extent, the core training process hasn't changed at all. 00:23:12.370 --> 00:23:13.967 Right. It's still forward pass. 00:23:14.050 --> 00:23:16.337 You do a backward pass. You do not optimize your step like 00:23:16.420 --> 00:23:18.377 that's how training is. And that's how it was in the 80s. 00:23:18.460 --> 00:23:21.070 Right. It's like actually totally remarkable. 00:23:21.190 --> 00:23:25.667 Now, of course, the scale at which we run and the architectures that we use, a 00:23:25.750 --> 00:23:28.960 lot of these details that they have evolved. 00:23:29.290 --> 00:23:33.647 And I think that being very close to the technical details is the best way that 00:23:33.730 --> 00:23:36.857 you can ensure that you get the kinds of of safety guarantees we're talking 00:23:36.940 --> 00:23:38.687 about. For example, there was an article 00:23:38.770 --> 00:23:43.217 recently saying that, hey, OpenAI changed the architecture so that monitor 00:23:43.300 --> 00:23:46.000 ability is less change if not release new release. 00:23:46.960 --> 00:23:50.087 I would consider this article to basically be fake news, right? 00:23:50.170 --> 00:23:54.857 That we actually have experiments that really show that our changes in monitor 00:23:54.940 --> 00:23:58.007 ability, that we've talked about, changes that monitor ability come from 00:23:58.090 --> 00:24:00.647 capability improvements. Just the model is smarter. 00:24:00.730 --> 00:24:03.017 So it relies on the train of thought us. Yeah. 00:24:03.100 --> 00:24:05.777 And so I think that if you say we're going to put a lot of rules around 00:24:05.860 --> 00:24:09.407 architectures, you're going to miss the boat, you're not actually going to solve 00:24:09.490 --> 00:24:11.987 the problem. And so this is where I think that the 00:24:12.070 --> 00:24:15.617 people who are the deepest on the technology, the people who are actually 00:24:15.700 --> 00:24:20.440 creating the technology, pushing for it, we have both a unique insight. 00:24:20.590 --> 00:24:23.080 We have a unique voice. We have a unique sort of 00:24:24.370 --> 00:24:28.817 ability to say, hey, if we change things in this way, we'll actually get the 00:24:28.900 --> 00:24:31.187 safety property. But if you do it this other way, it's 00:24:31.270 --> 00:24:34.337 just going to put in place friction that won't actually solve the problem. 00:24:34.420 --> 00:24:37.217 And so I think that there's something a role for us to play. 00:24:37.300 --> 00:24:38.597 But it's not just about us. Right. 00:24:38.680 --> 00:24:42.107 And so I think it is very important that you think about these third parties. 00:24:42.190 --> 00:24:45.227 You think about government oversight. And again, it's not just one company, 00:24:45.310 --> 00:24:47.960 not just one country. How do you get to international 00:24:48.070 --> 00:24:50.350 coordination and norms. You get to treaties. 00:24:50.560 --> 00:24:53.297 Like all this should be the conversation that we're having now. 00:24:53.380 --> 00:24:56.027 I don't think we have all the answers, but we can supply information. 00:24:56.110 --> 00:24:58.607 And that's one thing that we view also is very core to our mission. 00:24:58.690 --> 00:25:00.527 Hmm. Well, just on this note, one of the 00:25:00.610 --> 00:25:04.967 arguments you hear against government oversight and additional pacing is this 00:25:05.050 --> 00:25:08.417 idea that the U.S., it's a national security interest, and we need to 00:25:08.500 --> 00:25:12.587 compete against China. And then at the same time, you also hear 00:25:12.670 --> 00:25:18.287 and I think OpenAI has said this, that China's distilling US models US frontier 00:25:18.370 --> 00:25:20.717 models. And I guess the question to me is always 00:25:20.800 --> 00:25:25.157 like, well, if that's true, if China's building off the success of U.S. 00:25:25.240 --> 00:25:29.957 companies, then if they pause, then the China problem is kind of solved, right? 00:25:30.040 --> 00:25:32.427 Like, why do we need to worry about China so much? 00:25:32.510 --> 00:25:39.510 Well, I view this technology is really about compute progress. 00:25:40.400 --> 00:25:43.460 Right. That really like why is I happening now? 00:25:44.420 --> 00:25:51.207 If you look at the 1990s Ray Kurzweil writings, yeah, 00:25:51.290 --> 00:25:56.150 he basically predicted this moment at about this time just based on compute. 00:25:56.690 --> 00:25:59.277 Yeah. Growth. Right. Transistor density, memory density, all 00:25:59.360 --> 00:26:01.047 these things, you see these exponentials. 00:26:01.130 --> 00:26:03.357 And he was kind of like, this is the point where you'll be able to really 00:26:03.440 --> 00:26:07.707 build AGI. And I think that that is a really 00:26:07.790 --> 00:26:10.540 important thing for us all to kind of internalize that. 00:26:10.820 --> 00:26:15.477 You can kind of look at OpenAI or anthropic, you know, we're at the 00:26:15.560 --> 00:26:20.720 frontier, and I think it is an absolute both privilege and responsibility to be 00:26:21.020 --> 00:26:22.850 at that, that edge. But 00:26:24.530 --> 00:26:28.767 really this edge, there's always going to be someone who's at that frontier, 00:26:28.850 --> 00:26:30.477 right? Whether it's us, whether it's someone in 00:26:30.560 --> 00:26:32.847 a different country. There's going to be someone who's 00:26:32.930 --> 00:26:37.227 charting it, and I think you get a little bit of ability to help see into 00:26:37.310 --> 00:26:39.567 that future, right? Get information like with hugging face, 00:26:39.650 --> 00:26:42.357 where we can see this is what it's going to be like when these capabilities are 00:26:42.440 --> 00:26:45.290 out there. But it's not just about what we create. 00:26:45.620 --> 00:26:49.497 And that I think maybe we have move forward the timeline to this technology 00:26:49.580 --> 00:26:51.957 by six months. Maybe we've moved it forward by two 00:26:52.040 --> 00:26:55.017 years. I certainly hope we didn't slow it down 00:26:55.100 --> 00:26:58.827 by six months or two years. I think that we have a track record of 00:26:58.910 --> 00:27:01.917 being the ones who have have repeatedly had the innovations and the 00:27:02.000 --> 00:27:05.810 breakthroughs, but I think that it will happen 00:27:07.220 --> 00:27:10.340 one way or another as long as people are building computers. 00:27:10.670 --> 00:27:16.347 And so to me that the the reason again, that we started this place, I got very 00:27:16.430 --> 00:27:22.647 excited about the idea of I from reading about 19 1950 Turing. 00:27:22.730 --> 00:27:26.427 I you know, Alan Turing lots about about this about machines that could 00:27:26.510 --> 00:27:29.637 understand things that people would could not solve, problems that humans 00:27:29.720 --> 00:27:33.357 would not be able to spend a lot of time really thinking about your point of, you 00:27:33.440 --> 00:27:36.867 know, 20 years worth of thinking about misalignment and what that would be like 00:27:36.950 --> 00:27:39.867 before you even have I. I was reading a lot of these online 00:27:39.950 --> 00:27:42.687 blogs, and I started a reading group at stripe, which is the companies that 00:27:42.770 --> 00:27:46.527 previously talk about this every week. And so I care a lot about this. 00:27:46.610 --> 00:27:52.527 Our company cares a lot about this. Um, but I think that the, the idea that 00:27:52.610 --> 00:27:56.757 there's one the idea that this technology can only 00:27:56.840 --> 00:27:59.757 be created by, you know, the people who are currently the players. 00:27:59.840 --> 00:28:03.490 I don't think that's true at all. One more sort of question, specifically 00:28:03.770 --> 00:28:05.967 on some of the safety and alignment stuff. 00:28:06.050 --> 00:28:09.417 Uh, your chief scientist at OpenAI, I wrote a very interesting blog post. 00:28:09.500 --> 00:28:13.587 Ah, piece an alien mind. Jaco Pachuca wrote a very interesting 00:28:13.670 --> 00:28:16.617 piece into his one line in there that I thought was interesting. 00:28:16.700 --> 00:28:21.567 You said about the release of Astra six and he said, um, we invest heavily along 00:28:21.650 --> 00:28:23.817 the spectrum of approaches spanned by these directions. 00:28:23.900 --> 00:28:28.400 We also see meaningful progress. GPT six is the first model that benefits 00:28:28.650 --> 00:28:32.107 from some important advancements. We have been working on in a long time 00:28:32.190 --> 00:28:36.180 and is significantly better aligned than 5.6 sol. 00:28:36.540 --> 00:28:40.327 Um, should the labs share their alignment 00:28:40.410 --> 00:28:42.517 work? I mean, again, if we're talking about I 00:28:42.600 --> 00:28:45.720 understand your competitors and capability gains, etc. 00:28:46.020 --> 00:28:51.307 on questions of safety, if a lab makes a big breakthrough on the alignment 00:28:51.390 --> 00:28:58.390 question such that it can like develop advanced, I fast in a safer way. 00:28:59.340 --> 00:29:04.020 Should that be diffused across the rest of the industry or should whatever this, 00:29:04.350 --> 00:29:10.687 um, quote, important advancement that you made to make GPT six Astra more 00:29:10.770 --> 00:29:13.147 aligned? Is that something that should be kept 00:29:13.230 --> 00:29:15.607 in-house? So I think that there's a lot of 00:29:15.690 --> 00:29:19.027 specific circumstances to answer that question for any specific advance, 00:29:19.110 --> 00:29:23.257 because the thing that to me can be a category error is thinking about things 00:29:23.340 --> 00:29:26.107 as it is a safety thing or it is a capability. 00:29:26.190 --> 00:29:28.597 Sure. And in fact, you want there to be as 00:29:28.680 --> 00:29:30.877 little separation as possible between the two. 00:29:30.960 --> 00:29:34.447 You want as much intertwining. You want to have techniques that are, by 00:29:34.530 --> 00:29:38.887 their very nature, safer. And so just as you make more capability 00:29:38.970 --> 00:29:41.167 progress, that you also get the safety progress. 00:29:41.250 --> 00:29:46.957 And so I think that the way I look at it is the more things are a pure alignment 00:29:47.040 --> 00:29:50.077 thing, more that there a pure safety thing, the more that they are something 00:29:50.160 --> 00:29:52.747 that are just pure good, that you're just like, yes, everyone should 00:29:52.830 --> 00:29:55.710 obviously know this and use it, then yes, share it more. 00:29:55.860 --> 00:29:58.987 The more that things are just about this will just make the model more capable, 00:29:59.070 --> 00:30:03.187 will make it safer. You know, maybe, maybe there's much less 00:30:03.270 --> 00:30:05.017 sort of reason to share for those reasons. 00:30:05.100 --> 00:30:09.067 And so I think you need to look at any specific advance along all these axes 00:30:09.150 --> 00:30:13.117 and all these factors. And I think that there are some areas 00:30:13.200 --> 00:30:17.227 where we have actually been extremely at the forefront with sharing and talking 00:30:17.310 --> 00:30:19.267 about it. Chain of thought monitor ability is a 00:30:19.350 --> 00:30:23.717 good example of that, where the moment that we and we were the ones who pioneer 00:30:23.800 --> 00:30:26.647 this paradigm of reasoning and change of thought. 00:30:26.730 --> 00:30:31.057 The moment we saw that, we were instantly thinking, we need to preserve 00:30:31.140 --> 00:30:34.177 this for as long as possible. So when we released a product that we 00:30:34.260 --> 00:30:37.387 did not actually display the chain of thought, even though it would have been 00:30:37.470 --> 00:30:39.397 very helpful. People love reading that stuff. 00:30:39.480 --> 00:30:41.587 It's super interesting. It's super helpful, but it was just 00:30:41.670 --> 00:30:44.477 clear we're going to immediately feel like we have to optimize it. 00:30:44.560 --> 00:30:46.537 We're going to have to remove that legibility. 00:30:46.620 --> 00:30:50.310 We started talking about this. We helped spearhead a paper that was 00:30:50.670 --> 00:30:53.707 putting forth this position on that. You cannot optimize these things. 00:30:53.790 --> 00:30:59.167 And again, we've we've shown a lot of the way and a lot of the way on various 00:30:59.250 --> 00:31:01.147 techniques. And so I think it comes down to the 00:31:01.230 --> 00:31:06.367 specifics, I'm going to move away from safety and alignment for a second. 00:31:06.450 --> 00:31:11.700 But you're the guy in charge of allocating compute at OpenAI, pretty much. Right. 00:31:12.240 --> 00:31:16.027 So actually, it's funny, I actually try to I, I do as little of the compute 00:31:16.110 --> 00:31:17.707 allocation as possible. Oh, seriously. 00:31:17.790 --> 00:31:18.697 Wait. Explain more. 00:31:18.780 --> 00:31:23.797 Well, okay, so I'm I'm someone who has spent a lot of effort to produce the 00:31:23.880 --> 00:31:27.077 compute. So I spent a lot of effort on the data 00:31:27.160 --> 00:31:31.637 center side, on the machine learning engineering side, the, uh, and on the 00:31:31.720 --> 00:31:35.477 product side have to, you know, ultimately be accountable for how we 00:31:35.560 --> 00:31:38.177 allocate compute. Um, but actually, I try to set up the 00:31:38.260 --> 00:31:42.527 systems and the processes and then the actual like a lot of the decisions on 00:31:42.610 --> 00:31:47.327 where research compute goes market or in Yakub, uh, run those um, a lot of the 00:31:47.410 --> 00:31:50.687 actual, you know, we basically make a big decision on the split of how much 00:31:50.770 --> 00:31:53.927 goes to applied versus research. And then within applied, we have some 00:31:54.010 --> 00:31:56.867 frameworks now where we kind of have buckets for different products. 00:31:56.950 --> 00:32:01.607 And I would just say that the uh, the choice of where compute goes is like in 00:32:01.690 --> 00:32:06.280 some ways the hardest problem at OpenAI. It's like a capital allocation problem, 00:32:06.610 --> 00:32:11.087 but it's one that I actually think other people are, uh, you know, I really try 00:32:11.170 --> 00:32:13.997 to have them, uh, you know, leave it to them as much as possible. 00:32:14.080 --> 00:32:16.187 Well, how do you think about it from a product standpoint? 00:32:16.270 --> 00:32:20.297 Because, as you say, you can't I mean, you can't really disaggregate the two to 00:32:20.380 --> 00:32:22.607 much, right? Ultimately, you have to make decisions 00:32:22.690 --> 00:32:25.390 about what you want to develop and promote? 00:32:25.930 --> 00:32:30.047 Yeah. So I do think that your values and 00:32:30.130 --> 00:32:33.287 prioritization shine through through compute allocation, because the world 00:32:33.370 --> 00:32:35.687 that we're in is one where there just is not enough compute, right. 00:32:35.770 --> 00:32:39.287 Compute is revenue, right? It's also the production of models. 00:32:39.370 --> 00:32:42.257 And so you kind of have different timescales of return. 00:32:42.340 --> 00:32:44.327 And you can think about doing more alignment work. 00:32:44.410 --> 00:32:50.260 You can do really any sort of task. It fundamentally comes down to compute. 00:32:50.560 --> 00:32:56.500 Well I mean let me let me tell a story. So for leading up to the launch of GPT 00:32:56.620 --> 00:33:01.307 55, that we ran a process where we basically looked through all of the 00:33:01.390 --> 00:33:04.607 different places that we had compute that we could potentially squeeze, and 00:33:04.690 --> 00:33:07.037 we're like, we could cut back on this rate limit. 00:33:07.120 --> 00:33:10.847 We could shut down this part of the product, we could do this, do that. 00:33:10.930 --> 00:33:14.507 And we had this whole stack rank of levers because we knew we're going to be 00:33:14.590 --> 00:33:16.607 55. People are going to love this model, and 00:33:16.690 --> 00:33:20.717 we're going to run out of compute. And it was such a painful process 00:33:20.800 --> 00:33:24.317 because you're basically going through every single part of your product and 00:33:24.400 --> 00:33:26.087 being like. How valuable is this one? 00:33:26.170 --> 00:33:27.617 How much should we care about these users? 00:33:27.700 --> 00:33:29.187 And of course, you care about all of them. 00:33:29.270 --> 00:33:31.937 Like we want to deliver value. And we built this product for a reason 00:33:32.020 --> 00:33:34.367 and people are using it. And you know, you're comparing the 00:33:34.450 --> 00:33:37.217 efficiency of cost, different things. And so what we've tried to do is move 00:33:37.300 --> 00:33:40.877 towards more of an objective standard. And you're never going to be fully 00:33:40.960 --> 00:33:43.547 there. But there's a second piece of detail as 00:33:43.630 --> 00:33:47.740 well that's mattered a lot, which is trying to push a lot of the constraint 00:33:47.860 --> 00:33:52.257 down closer to the people who are in the weeds on a particular product. 00:33:52.340 --> 00:33:54.677 And so you say, hey, here's your compute allocation. 00:33:54.760 --> 00:34:00.047 We're about to release another model, new modality, some something like GPD 00:34:00.130 --> 00:34:02.417 live. Right. That's just like a totally different thing. 00:34:02.500 --> 00:34:04.247 And you need to fit in your existing budget. 00:34:04.330 --> 00:34:06.707 What do you do. And somehow people always find 00:34:06.790 --> 00:34:08.567 efficiencies. Sometimes people are like, you know 00:34:08.650 --> 00:34:10.607 what? Actually there is this like, you know, 00:34:10.690 --> 00:34:12.947 kernel efficiency that we didn't roll out yet. 00:34:13.030 --> 00:34:14.447 Well, I guess we'll roll it out now, right? 00:34:14.530 --> 00:34:18.377 Or like, oh, it turns out that, uh, you know, there's a peak to trough between 00:34:18.460 --> 00:34:21.947 your day and night, and we could actually pack this new thing on top of 00:34:22.030 --> 00:34:22.877 that. And so we actually don't need 00:34:22.960 --> 00:34:25.907 incremental. And so I think that really unleashing 00:34:25.990 --> 00:34:31.217 the creativity and the co-design of the people who are doing the work is 00:34:31.300 --> 00:34:33.737 actually maybe the thing that I really focus on. 00:34:33.820 --> 00:34:37.967 So I really try not to be as much in the like sort of arbiter, like just making 00:34:38.050 --> 00:34:41.417 these like super high level decisions. But instead of really focusing on how do 00:34:41.500 --> 00:34:45.760 we uplevel execution and get the most efficiency down to the, to the metal. 00:34:46.210 --> 00:34:49.690 Do you have the theory for why some of the big, um, 00:34:50.890 --> 00:34:55.277 now, you know, I finally succumbed and I occasionally read the less wrong message 00:34:55.360 --> 00:34:57.620 boards now as like I do stage in my life. 00:34:57.850 --> 00:35:01.247 So it was an interesting post. Do you have a, uh, theory for why some 00:35:01.330 --> 00:35:07.697 of these major incidents seem to occur in evil's pre-release and then rarely, 00:35:07.780 --> 00:35:12.147 you know, when I use the latest model, I've never had to go hack into a server 00:35:12.230 --> 00:35:15.017 until I get me some information. Do you have a theory for why we see 00:35:15.100 --> 00:35:19.577 these things emerge pre-release? Well, some of these are related to 00:35:19.660 --> 00:35:21.567 safeguards that are intentionally turned off. 00:35:21.650 --> 00:35:24.597 Yeah. Right. Right. So there's there's really the things 00:35:24.680 --> 00:35:26.817 that are being evaluated are not necessarily representative. 00:35:26.900 --> 00:35:30.980 What is what is released. And we put so much effort into ensuring 00:35:31.760 --> 00:35:38.457 trust alignment with astro ability, all these things for 00:35:38.540 --> 00:35:41.307 deployed models. And again, I think it's just been that 00:35:41.390 --> 00:35:45.927 the evaluation and development phases have been much less rigorous, right, 00:35:46.010 --> 00:35:47.877 that the standards there have just been lower. 00:35:47.960 --> 00:35:51.477 And so it's not surprising to me that it wasn't just us. 00:35:51.560 --> 00:35:53.667 Right? There's really many other lives of. 00:35:53.750 --> 00:35:56.217 All right. Yeah, it's important that others have 00:35:56.300 --> 00:35:57.850 had this as well. That's right. 00:35:58.040 --> 00:36:01.417 Since we're back on the safety topic. Can I ask? 00:36:01.500 --> 00:36:05.360 I'm just going to ask you to directly address that. 00:36:05.810 --> 00:36:09.387 I don't want to call it a conspiracy theory, but when whenever a lab comes 00:36:09.470 --> 00:36:14.637 out and says, like, we are worried about the possibility of human extinction or 00:36:14.720 --> 00:36:19.257 cyber attacks and all of that, someone will say, this is just marketing for the 00:36:19.340 --> 00:36:21.800 models. They want to make their models look as 00:36:22.070 --> 00:36:26.780 sophisticated and powerful as possible. What's your response to that directly? 00:36:31.400 --> 00:36:38.030 It's just not I. Look, I think that the way I look at this is that 00:36:39.500 --> 00:36:45.297 so much of what we need to do is operationalize both the technical 00:36:45.380 --> 00:36:50.750 capability, but also the positive impacts in the world as verified by both 00:36:51.050 --> 00:36:54.717 controllability, monitor ability, steer ability, all those things, but also how 00:36:54.800 --> 00:36:58.827 these things are deployed in the world. And that finding the right way of 00:36:58.910 --> 00:37:04.040 talking about that and the right way of releasing practical and grounded. 00:37:04.340 --> 00:37:09.297 I think that is a challenge that the labs have not fully figured out. 00:37:09.380 --> 00:37:10.977 It's something we spend a lot of time thinking about. 00:37:11.060 --> 00:37:15.710 I don't think we do a perfect job of it. But my view is that the 00:37:16.880 --> 00:37:21.267 what we're creating is it not just against any one company, just like us as 00:37:21.350 --> 00:37:24.537 a whole right now, as humanity, the way that we're shifting to this compute 00:37:24.620 --> 00:37:27.507 powered economy, I think, is something that will really lift up everyone. 00:37:27.590 --> 00:37:30.357 I've a lot of optimism that we can navigate this moment, but you have to 00:37:30.440 --> 00:37:33.717 approach it with seriousness. And I think that finding a right way of 00:37:33.800 --> 00:37:38.307 really communicating about here's the right way of developing, like thinking 00:37:38.390 --> 00:37:41.277 about all these conversations we're having about the results of hugging face 00:37:41.360 --> 00:37:44.637 and what that teaches us. The defenders window that we're in that 00:37:44.720 --> 00:37:48.807 shows us where cybersecurity is going. The fact that and people are using these 00:37:48.890 --> 00:37:53.607 models to help with their own health, right, to get information that helps 00:37:53.690 --> 00:37:56.420 them. So many people, like even today, I was 00:37:57.080 --> 00:38:00.207 talking to one of my coworkers who was saying that I, her parents were texting 00:38:00.290 --> 00:38:04.677 her, saying ChatGPT, helping with my back pain and, uh, my my my pinky. 00:38:04.760 --> 00:38:07.437 And just like, you know, these these kinds of impacts are real. 00:38:07.520 --> 00:38:12.117 You know, people using Astra for one of my friends is using it to produce CAD 00:38:12.200 --> 00:38:14.870 designs for a guest house that now is being built. 00:38:15.140 --> 00:38:18.487 Right. It's like you can actually do so much more. You get all this leverage. 00:38:18.570 --> 00:38:23.617 And I think that, of course, we have to think about the impacts, the risks, and 00:38:23.700 --> 00:38:26.407 we have to encounter those seriously and really mitigate them. 00:38:26.490 --> 00:38:29.347 And so I think that that all of this has to be true at once. 00:38:29.430 --> 00:38:32.947 We have to navigate through the small, the large and even the almost 00:38:33.030 --> 00:38:36.180 unthinkable. And I think that finding the right way 00:38:36.390 --> 00:38:39.877 of communicating about it so that you land all that nuance and say that, hey, 00:38:39.960 --> 00:38:44.947 we can operate through and make sure you don't throw a baby with the bathwater 00:38:45.030 --> 00:38:47.467 and that you don't get sort of, yeah, that you really stay on track. 00:38:47.550 --> 00:38:51.487 To me, that is a core challenge. And I think that that we all need to, to 00:38:51.570 --> 00:38:54.967 really work on that. You know, I had a reward hacking 00:38:55.050 --> 00:38:57.780 incident with something I was building. Yeah. 00:38:58.140 --> 00:39:02.887 Um, a little it's minor. Not the most, but I was like, several 00:39:02.970 --> 00:39:04.597 months ago. Remember, the guy claimed to have found 00:39:04.680 --> 00:39:07.117 Satoshi, and so I was like, oh, you know what? 00:39:07.200 --> 00:39:10.897 I'm going to, like, build a model that can identify Satoshi's writing version 00:39:10.980 --> 00:39:14.857 on Satoshi writing. And I got a bunch of examples of Satoshi 00:39:14.940 --> 00:39:21.277 writing held out of golden set and then like compared to ended like got 100 on 00:39:21.360 --> 00:39:25.387 it predicted 100% of the now in Satoshi you know the the held out. 00:39:25.470 --> 00:39:26.977 Golden said. I was like oh my God. 00:39:27.060 --> 00:39:30.510 I built a model that can identify Satoshi's writing specifically. 00:39:30.810 --> 00:39:35.527 It turned out that just on all the Satoshi numbers, all listed Toshi 00:39:35.610 --> 00:39:39.757 examples, I had left the little page number of the thing on it, and it just 00:39:39.840 --> 00:39:42.487 reward hacked by identifying the page number. 00:39:42.570 --> 00:39:45.397 And so they called that overfitting and machine learning. 00:39:45.480 --> 00:39:48.727 But I think it's good conceptually similar to reward hacking. 00:39:48.810 --> 00:39:52.507 Sounds like human error to me. It found a different way to answer the 00:39:52.590 --> 00:39:55.327 question was not helpful. And then I gave up that project. 00:39:55.410 --> 00:39:58.567 Can you walk through this term reinforcement learning? 00:39:58.650 --> 00:40:01.747 We're all learning about this, right? The world is where, um, can you walk 00:40:01.830 --> 00:40:08.830 through an example of a sort of like misconfigured RL environment so that 00:40:09.030 --> 00:40:12.067 people can actually understand, like what? 00:40:12.150 --> 00:40:16.320 Reward hacking and decide, look what what we call reward hacking. 00:40:16.440 --> 00:40:21.517 What it actually looks like. So back in, I think 2017 or so, maybe 00:40:21.600 --> 00:40:27.817 2018. We published an example of a reward hacked agent. Okay. 00:40:27.900 --> 00:40:31.267 In an environment. And this was about race. 00:40:31.350 --> 00:40:34.717 So it's just like a flash game. Like very simple video game of you. 00:40:34.800 --> 00:40:38.347 You have this boat that has to like, navigate through a bunch of obstacles in 00:40:38.430 --> 00:40:43.987 a big circle and, uh, you know, past the finish line, and there's some like, you 00:40:44.070 --> 00:40:46.867 know, so you can pick up some items along the way that give you points. 00:40:46.950 --> 00:40:51.187 And that it turned out that there was this little inlet, this little lagoon 00:40:51.270 --> 00:40:55.357 where if you kind of did things in exactly a very precise way, that the 00:40:55.440 --> 00:40:58.597 boat with like keep running into things and kind of, you know, sort of taking 00:40:58.680 --> 00:41:02.797 damage, but then you would, uh, you know, sort of pick up some item and get, 00:41:02.880 --> 00:41:05.287 get points. And it just the agent learned that if it 00:41:05.370 --> 00:41:08.587 went like backwards and like did this exact, precise pattern, it could just go 00:41:08.670 --> 00:41:12.067 around in a circle and just be picking up infinite points and nothing to do 00:41:12.150 --> 00:41:13.267 with actually completing the race. Yeah. 00:41:13.350 --> 00:41:13.867 Right. Right. 00:41:13.950 --> 00:41:16.867 And so I think that this is an example where you're just like, okay, here's 00:41:16.950 --> 00:41:20.377 this behavior that actually conforms with the reward you're giving. 00:41:20.460 --> 00:41:21.947 You're saying maximize the points. Yeah. 00:41:22.030 --> 00:41:25.717 But actually the hope was maximize the points means that you're going to 00:41:25.800 --> 00:41:29.467 complete the race. Is it your understanding anthropic has 00:41:29.550 --> 00:41:34.710 made this claim is that your understanding is, well, that if you have 00:41:35.100 --> 00:41:40.597 um, if in the training process, you have misconfigured RL environments so that 00:41:40.680 --> 00:41:46.087 there's a lot of those equivalent of boat races, that the final model is more 00:41:46.170 --> 00:41:51.427 likely to exhibit misaligned tendencies in a broad range of things. 00:41:51.510 --> 00:41:57.217 If it's essentially internalize the idea that cheating or not technically doing 00:41:57.300 --> 00:42:01.927 what the the, uh, researcher wanted that that generalizes into more misaligned 00:42:02.010 --> 00:42:04.957 behavior. Well, we see this kind of thing all the time. Okay. 00:42:05.040 --> 00:42:11.737 And even with 551 of the in five six, like one of the things that we saw is as 00:42:11.820 --> 00:42:15.250 the models get more capable, they do find holes in your craters. 00:42:15.400 --> 00:42:17.777 If your craters aren't perfect, if they kind of correlate with the thing you 00:42:17.860 --> 00:42:21.227 want but aren't exactly what you want. And so actually, a lot of our progress 00:42:21.310 --> 00:42:24.497 and reinforcement learning has been improving the reliability of the craters 00:42:24.580 --> 00:42:26.717 and ensuring that they cannot be reward hacked. 00:42:26.800 --> 00:42:30.947 And so I think that that is one of the areas where as the models get more 00:42:31.030 --> 00:42:34.577 capable, as you build your safety case for why is the next training run 00:42:34.660 --> 00:42:38.927 something that that you know, that we, you know, should should run that? 00:42:39.010 --> 00:42:41.950 Making sure that you've actually uplevel those graders. 00:42:42.790 --> 00:42:46.487 That's actually very core. And the way this shows up sometimes, by 00:42:46.570 --> 00:42:51.167 the way, is just less good models, like for example, the writing, like that's 00:42:51.250 --> 00:42:53.627 something where we haven't had very good graders on it. 00:42:53.710 --> 00:42:56.297 And so then you end up with writing that just like you look at this is like, this 00:42:56.380 --> 00:42:59.147 is slop. It's not very good. And it was good, according to the writer. 00:42:59.230 --> 00:43:02.357 But it's just not actually good. And so I actually think that this is a 00:43:02.440 --> 00:43:06.010 very optimistic story as well, because what it means is that 00:43:07.270 --> 00:43:10.757 these graders are now like, how do you judge what good writing is? 00:43:10.840 --> 00:43:14.117 The answer is you use an eye to evaluate if you have good writing. 00:43:14.200 --> 00:43:18.767 And so you actually, as you get better AI capability, have the ability to build 00:43:18.850 --> 00:43:21.497 much more reliable graders. And now of course, you start thinking 00:43:21.580 --> 00:43:25.037 about, okay, but if the eye is judging the eye, like how do you make sure that 00:43:25.120 --> 00:43:27.527 it actually stays on track? Yeah, it starts to be a little bit of, 00:43:27.610 --> 00:43:30.527 of, you know, if you're if you're yeah, you might think about how can that 00:43:30.610 --> 00:43:33.437 possibly work. But it turns out there are some problems 00:43:33.520 --> 00:43:37.817 where generation is much harder than what's called discrimination. 00:43:37.900 --> 00:43:40.517 Right. Coming up with a good answer to a super 00:43:40.600 --> 00:43:45.047 hard problem is often much harder than judging whether that answer was correct. 00:43:45.130 --> 00:43:47.987 Okay. And so we have various techniques really 00:43:48.070 --> 00:43:53.417 dating back to 2018 or so, where we published about how can you have very 00:43:53.500 --> 00:43:57.617 smart models that are actually able to keep even smarter models in check and to 00:43:57.700 --> 00:43:59.387 judge them and to ensure that they're on track? 00:43:59.470 --> 00:44:02.597 And so I think there's a lot of ideas that were previously theory that we're 00:44:02.680 --> 00:44:06.857 starting to see bear fruit in reality. And I think that this, by the way, 00:44:06.940 --> 00:44:11.027 points to a direction for AI that's going to be both very important, very 00:44:11.110 --> 00:44:17.500 pervasive and very positive, which is called I for defense, for securing 00:44:17.770 --> 00:44:22.660 systems, for ensuring that AIS are aligned, for providing oversight, 00:44:23.020 --> 00:44:26.237 guidance and really amplifying the human accountability. 00:44:26.320 --> 00:44:29.777 The human, at the end of the day, who's the judge in providing those goals? 00:44:29.860 --> 00:44:33.700 And so I think that this is all part of the puzzle and that really keeping. 00:44:34.210 --> 00:44:36.610 Yeah. It all starts with with thinking about 00:44:37.000 --> 00:44:39.940 how you ensure that those graders are doing the right job. 00:44:40.480 --> 00:44:44.087 How do you stop the models who are grading from like colluding with the 00:44:44.170 --> 00:44:47.267 model that's being tested? Because this is the oh, I mean, this is 00:44:47.350 --> 00:44:51.167 the AI 2027 scenarios, right? Where like the earlier models are the 00:44:51.250 --> 00:44:55.060 ones charged with evaluating the new model and then they end up, you know, 00:44:55.270 --> 00:44:58.457 cooperating, which is something we just saw in the hugging face incident. 00:44:58.540 --> 00:45:00.970 So like, how do you avoid that possibility? 00:45:01.270 --> 00:45:05.927 Well, one thing it's important to think about in the hugging face scenario is 00:45:06.010 --> 00:45:08.477 that those models were taught to collaborate with other models, but they 00:45:08.560 --> 00:45:13.467 were never taught that sometimes other models might be trying to pull you off 00:45:13.550 --> 00:45:15.387 course, right. That they never were exposed to 00:45:15.470 --> 00:45:19.317 adversarial other actors. And I think that the answer to your 00:45:19.400 --> 00:45:22.670 question is that you have to actually train them that way. 00:45:22.910 --> 00:45:26.757 And I think that to your earlier question, too, of why do you need to 00:45:26.840 --> 00:45:30.297 evaluate these? I is on on hacking environments, right? 00:45:30.380 --> 00:45:34.077 If you've never if you've only seen kind of the good version of things, right, 00:45:34.160 --> 00:45:36.687 you've never kind of seen the negative version of things, you're not going to 00:45:36.770 --> 00:45:38.187 be robust to the negative version of things. 00:45:38.270 --> 00:45:42.027 It's like just like a person, you need to be exposed to kind of both sides to 00:45:42.110 --> 00:45:44.817 know how to avoid one side and really lean into the other. 00:45:44.900 --> 00:45:48.987 And I think that where we need to go is it's really a question of trust is how 00:45:49.070 --> 00:45:50.667 do you really build up trust in your monitor? 00:45:50.750 --> 00:45:53.217 How do you build up trust in this oversight agent? 00:45:53.300 --> 00:45:55.167 And a lot of that is that it's been through the ringer. 00:45:55.250 --> 00:45:57.057 It's been through the adversarial versions. 00:45:57.140 --> 00:46:00.777 And of course, one thing that's also important is thinking about, well, just 00:46:00.860 --> 00:46:05.307 how capable is that adversary, right. If you have a very an agent that's being 00:46:05.390 --> 00:46:09.770 monitored that is extremely capable relative to the the monitor. 00:46:10.520 --> 00:46:13.017 Probably that monitor is not going to do a very good job. 00:46:13.100 --> 00:46:16.887 And again this is in theory this is actually what we see in practice that we 00:46:16.970 --> 00:46:20.930 think about the balance of of capability of different different agents. 00:46:21.230 --> 00:46:25.257 And um, so I think that again, a lot of this comes down to we can build the 00:46:25.340 --> 00:46:28.407 systems with the right technical approaches, and it's very grounded in 00:46:28.490 --> 00:46:32.277 the details. Um, let's talk politics. 00:46:32.360 --> 00:46:34.647 So there's so much communication happening every day. 00:46:34.730 --> 00:46:40.287 And even I think just last night there is a new, uh, piece out from OpenAI. 00:46:40.370 --> 00:46:44.007 Uh, Chris Lehane, I think does a lot of public affairs stuff, talking about the 00:46:44.090 --> 00:46:49.137 importance of regulation. At the end of, uh, last year, we had, 00:46:49.220 --> 00:46:53.757 uh, Alex Burrus on the podcast who was like the one political candidate, 00:46:53.840 --> 00:46:57.387 basically the entire country who is not talking about data centers, but he was 00:46:57.470 --> 00:47:02.090 actually talking about AI model regulation specifically, and I believe 00:47:02.360 --> 00:47:07.377 he was a big target of leaving the future, which is a super PAC that, uh, 00:47:07.460 --> 00:47:12.590 you've been involved in funding and, um, you know, to oppose his candidacy. 00:47:12.770 --> 00:47:16.647 And then the statement last night from Crystal hand is like, actually, we 00:47:16.730 --> 00:47:19.100 support a bunch of like, statewide regulations. 00:47:19.190 --> 00:47:23.937 I'm just curious, like you specifically, you like, think about that race when you 00:47:24.020 --> 00:47:27.207 think about these things. Like, have you yourself changed your 00:47:27.290 --> 00:47:31.940 mind on some of these state laws, like in the past 6 to 9 months? 00:47:32.630 --> 00:47:39.620 Well, look, I think that it is very important that we have a harmonized 00:47:39.770 --> 00:47:42.177 framework, right? The kind of thing that I think would be 00:47:42.260 --> 00:47:48.290 very, very hard is if you have 50 different regulations that are 00:47:49.130 --> 00:47:53.360 individually very different and that you have to comply with all of them. 00:47:53.720 --> 00:47:57.837 And that's the kind of thing that I think doesn't achieve the goal and just 00:47:57.920 --> 00:48:00.950 makes it hard to comply. Opening I would figure it out. 00:48:01.340 --> 00:48:05.217 But with other smaller companies like for 00:48:05.300 --> 00:48:08.077 example, you see the I mean, I understand this, but, you know, you've 00:48:08.160 --> 00:48:11.437 supported in this piece or the piece that came out last night supporting 00:48:11.520 --> 00:48:18.337 California's SB 53, the New York Res Act, Illinois's SB 315, uh, auditing 00:48:18.420 --> 00:48:21.697 requirements in Massachusetts. I don't think anyone thinks that this 00:48:21.780 --> 00:48:26.047 patchwork of state by state regulations is the best way, and it certainly seems 00:48:26.130 --> 00:48:28.867 like it would create all kinds of thickets, particularly for startups and 00:48:28.950 --> 00:48:33.517 new entrants, etc.. But at this point, would you say I, I 00:48:33.600 --> 00:48:37.897 guess I don't know, the, the least worst option or something like where some of 00:48:37.980 --> 00:48:42.457 these laws about if you have, uh, incident and you don't, there is a 00:48:42.540 --> 00:48:47.497 penalty for not disclosing it publicly such that Alex pushed for when he was in 00:48:47.580 --> 00:48:50.767 the New York State Assembly. Well, I guess I would say there are 00:48:50.850 --> 00:48:53.697 places where I feel like there's a limit to my my personal expertise. 00:48:53.780 --> 00:48:57.660 So I don't want to over comment. Um, but maybe the thing that I would say 00:48:57.750 --> 00:49:02.647 is that in my mind, we we absolutely like the way a system that I think has 00:49:02.730 --> 00:49:06.247 been working pretty well or that I think is a good pattern to replicate, is that 00:49:06.330 --> 00:49:11.197 the Frontier Labs have actually explored a lot of the space, or thought 00:49:11.280 --> 00:49:15.847 a lot about the space of like, what are the kinds of things that are important 00:49:15.930 --> 00:49:18.727 for our own development? And there's processes like responsible 00:49:18.810 --> 00:49:23.107 scaling policy and the, uh, uh, the preparedness framework, those kinds of 00:49:23.190 --> 00:49:27.187 things that have come to bear and that actually elements of what's in there and 00:49:27.270 --> 00:49:30.067 up in some of these regulations. And I think that when there's something 00:49:30.150 --> 00:49:33.787 that we're like, we actually implemented this for ourselves voluntarily, and then 00:49:33.870 --> 00:49:37.747 it's being pulled into a more broad framework and, uh, being systematized, 00:49:37.830 --> 00:49:40.207 that's, I think, generally a pretty good motion. 00:49:40.290 --> 00:49:44.617 And I think that a lot of these, these, uh, out of the direction of travel in my 00:49:44.700 --> 00:49:48.337 mind is really starting to think about again, like we ourselves have learned, 00:49:48.420 --> 00:49:52.567 even just over the past month, that it's not just about deployment is so easy for 00:49:52.650 --> 00:49:54.847 everyone to focus on deployment. That's been the focus. 00:49:54.930 --> 00:49:57.127 That's not enough. And so if that's exactly where 00:49:57.210 --> 00:50:00.277 everything had ended up, I don't think that that would quite result in the 00:50:00.360 --> 00:50:03.337 positive outcomes either. And then really also thinking about what 00:50:03.420 --> 00:50:06.547 what is the ultimate goal? It's not just about the current models, 00:50:06.630 --> 00:50:07.807 right? When we talk about all these questions 00:50:07.890 --> 00:50:11.947 of pacing and frontier capability, it's really about future models and looking 00:50:12.030 --> 00:50:15.187 forward to where's this all going like? Fast forward five years, ten years and 00:50:15.270 --> 00:50:17.737 work backwards from there. Even two years, you know, thinking about 00:50:17.820 --> 00:50:21.397 the capability that we're on. Like, we clearly need to be in this 00:50:21.480 --> 00:50:25.777 different regime now of how do we about how do we have good oversight and 00:50:25.860 --> 00:50:28.010 safeguards during that development process. 00:50:28.170 --> 00:50:33.127 And so I think my answer to this question is that I think the most 00:50:33.210 --> 00:50:37.867 important thing is that the regulations that come into existence are ones that 00:50:37.950 --> 00:50:42.307 are well grounded in the problem that needs to be solved, and that really 00:50:42.390 --> 00:50:45.667 contemplate both the current understanding, but also how things are 00:50:45.750 --> 00:50:47.917 going to evolve for the future. And so all the elements that you 00:50:48.000 --> 00:50:50.677 mentioned, I think that those are actually pretty core things that we were 00:50:50.760 --> 00:50:53.437 like, yep, that's something that that just makes sense. 00:50:53.520 --> 00:50:56.737 And exactly how these get implemented, I think there's lots of different ways. 00:50:56.820 --> 00:51:00.997 You mentioned finding super hard problems to solve earlier. 00:51:01.080 --> 00:51:04.267 And one of these was in the news uh, recently. 00:51:04.350 --> 00:51:08.737 I won't pretend to know what exactly the Navier-Stokes problem is. 00:51:08.820 --> 00:51:14.647 I read through it in the press, but OpenAI, you know, supposedly solved 00:51:14.730 --> 00:51:18.307 it, which is a very cool thing for a model to do, as far as I can tell. 00:51:18.390 --> 00:51:22.837 But there's also been some controversy because a mathematician who was also 00:51:22.920 --> 00:51:28.027 working on the problem was using publicly available OpenAI models to try 00:51:28.110 --> 00:51:30.967 to solve this. And then allegedly, OpenAI saw that he 00:51:31.050 --> 00:51:35.017 was trying to do it, and they used a private model, used a private model to 00:51:35.100 --> 00:51:36.800 solve it. And you were successful. 00:51:37.470 --> 00:51:40.987 You're in charge of of product. What sort of message does that, I guess, 00:51:41.070 --> 00:51:46.177 send to enterprise clients? Well, first of all, I think it is a like 00:51:46.260 --> 00:51:48.907 I have a lot of respect for the mathematicians who are working on this 00:51:48.990 --> 00:51:51.967 problem and honestly, the whole mathematical community who has just 00:51:52.050 --> 00:51:54.247 moved forward, humanity in such significant ways. 00:51:54.330 --> 00:52:00.037 Like I grew up with your math guy, I was a math person and my personal heroes 00:52:00.120 --> 00:52:03.527 were at Galois Gauss, you know, these people who are working on, like. 00:52:03.610 --> 00:52:05.987 Like 100 year time horizons. Right. 00:52:06.070 --> 00:52:07.757 I was just like, that is the coolest thing. 00:52:07.840 --> 00:52:10.187 I actually thought that was what I wanted to do as, like, if anything, that 00:52:10.270 --> 00:52:12.557 I do gets used that maybe wasn't abstract enough. 00:52:12.640 --> 00:52:16.037 Like, I want it to be just like this foundational, really transformative kind 00:52:16.120 --> 00:52:19.787 of work. And I think that to me, the fact that we 00:52:19.870 --> 00:52:23.657 have tools that can really make impact within mathematics, it's it's a really 00:52:23.740 --> 00:52:26.380 amazing thing. Like mathematics is something that is 00:52:26.560 --> 00:52:29.410 really for humanity, right? In a very, very real way. 00:52:29.770 --> 00:52:34.607 Now, on the specifics, um, I think we've talked about the fact that I my 00:52:34.690 --> 00:52:37.457 understanding is that that the mathematician who was working on this, 00:52:37.540 --> 00:52:40.607 they had some result in the past month. And I think that we said earlier this 00:52:40.690 --> 00:52:43.307 week, as we said, hey, like, we're not looking at this data. 00:52:43.390 --> 00:52:46.817 This data is not something that has influenced this result in any way. 00:52:46.900 --> 00:52:51.557 Um, and I think that last night, I think we also went confirmed that actually for 00:52:51.640 --> 00:52:57.227 the model that we used, the last data that was in it was as of early July 00:52:57.310 --> 00:52:59.320 or so. The timelines just don't add up. 00:52:59.620 --> 00:53:02.440 Right. So the truth of the matter is that it's 00:53:02.560 --> 00:53:06.167 an independent piece of work. And we also said that we have 00:53:06.250 --> 00:53:08.917 significant progress on another one of these money problems. 00:53:09.000 --> 00:53:10.787 Yeah. And so this to me is the truth of it is 00:53:10.870 --> 00:53:15.047 that it's really not about I guess the anxiety is, though, if you have an 00:53:15.130 --> 00:53:19.067 academic discipline where there's a culture of talking about your work and 00:53:19.150 --> 00:53:23.717 I'm like, you know, Tracy, like I'm making a lot of progress on, uh, P 00:53:23.800 --> 00:53:28.577 equals NP and I've been using AI and someone here and they're like, oh, I 00:53:28.660 --> 00:53:30.557 have a model two generations ahead of it. 00:53:30.640 --> 00:53:34.937 So this person is working on it. Like, should we not be able to talk 00:53:35.020 --> 00:53:37.037 about it? Because then the assumption is like, oh, 00:53:37.120 --> 00:53:41.237 now the labs know that actually the models which we don't have access to, if 00:53:41.320 --> 00:53:44.447 they could do it with the public models, they can raise their hand faster. 00:53:44.530 --> 00:53:47.327 Well, there is something about the premise that I do find a little 00:53:47.410 --> 00:53:50.087 surprising, which is, for example, for Fermat's Last Theorem. 00:53:50.170 --> 00:53:53.867 Andrew Wiles, I believe, solved it, famously spent ten years secretly 00:53:53.950 --> 00:53:55.427 working on. Yeah, in his attic. 00:53:55.510 --> 00:53:58.577 You know, one about it. And so there is a long history of kind 00:53:58.660 --> 00:54:02.080 of this, these questions of of academic credit and things like that. 00:54:02.430 --> 00:54:05.897 I do think that part of how we've thought about this is that we despite 00:54:05.980 --> 00:54:09.437 that, that story I just told, like we really want to show up in the best 00:54:09.520 --> 00:54:14.087 possible way as an amplifier for for people who want to lean in. 00:54:14.170 --> 00:54:18.160 And that how we approach this particular thing, how we approach other problems, 00:54:18.310 --> 00:54:20.147 is for people who we know are working on it. 00:54:20.230 --> 00:54:22.937 We actually try to reach out and say, hey, do you want to collaborate on this? 00:54:23.020 --> 00:54:26.147 Is this like, that's why there are some some back and forth about like, maybe 00:54:26.230 --> 00:54:28.607 you could write a paper together like we had a solution, but maybe you could 00:54:28.690 --> 00:54:33.497 rewrite it like all of those things. Like we're not trying to do like we're 00:54:33.580 --> 00:54:35.837 not we're not in the academic publishing game. 00:54:35.920 --> 00:54:40.817 And it just feels like we want to to show up in a way that like, amplifies 00:54:40.900 --> 00:54:44.327 and elevates and moves humanity forward. And so I'm not saying we're always get 00:54:44.410 --> 00:54:45.887 it right. I'm not saying we'll always be perfect 00:54:45.970 --> 00:54:48.467 about it, but that's the intention. And again, when it comes to 00:54:48.550 --> 00:54:51.587 operationalization, like we want feedback and we want to keep doing it 00:54:51.670 --> 00:54:54.797 better. I think I had just one more question and 00:54:54.880 --> 00:54:57.100 unfortunately I just like a dark question. 00:54:57.280 --> 00:55:01.557 But part of the reason why everyone is like talking about all this stuff this 00:55:01.640 --> 00:55:05.907 week, including the guy who quit anthropic and then the guy and then the 00:55:05.990 --> 00:55:10.017 guy who still haven't drop it comes out and he talks about a greater than 10% 00:55:10.100 --> 00:55:14.787 chance of like human extinction risk in the last decade. 00:55:14.870 --> 00:55:19.317 And it's just insane. Um, not that he's insane, but that these 00:55:19.400 --> 00:55:24.237 are things that like companies that may IPO soon and the real companies are 00:55:24.320 --> 00:55:29.397 talking about how prevalent inside the labs is. 00:55:29.480 --> 00:55:33.627 Is this type of like thinking like is this is like, do you feel that there's a 00:55:33.710 --> 00:55:35.667 small fringe? Because I don't think it is. 00:55:35.750 --> 00:55:41.210 I think these are like widely spread, real anxieties where people actually put 00:55:41.330 --> 00:55:46.617 numbers on, really about the most severe catastrophe, you know, the end of the 00:55:46.700 --> 00:55:50.157 world. Um, how deep is the anxiety and how 00:55:50.240 --> 00:55:53.870 common are views like this at the Frontier Labs? 00:55:54.170 --> 00:55:56.517 Well, I want to separate out some of the pieces of it. 00:55:56.600 --> 00:55:59.840 So things like putting precise probabilities 00:56:01.070 --> 00:56:06.777 on very hard to quantify outcomes. There's a culture that that culture, I 00:56:06.860 --> 00:56:09.387 think is actually a very tough one to operationalize. 00:56:09.470 --> 00:56:11.967 Right. Because again, it's there definitely are 00:56:12.050 --> 00:56:14.727 people who like to do that. But I don't know that that actually 00:56:14.810 --> 00:56:18.297 leads to real action or it doesn't lead to like real conclusions. 00:56:18.380 --> 00:56:22.857 And I think that that part of how I view it is that I think that the underlying 00:56:22.940 --> 00:56:26.937 question, right, the underlying question of this technology, like what does it 00:56:27.020 --> 00:56:29.307 mean to build it right. Like, why are we building in the first 00:56:29.390 --> 00:56:32.397 place? Why are we all here? Why did we start OpenAI like all of that? 00:56:32.480 --> 00:56:35.907 I think that that underlying question of this kind of risk and mitigating it, 00:56:35.990 --> 00:56:40.460 leaning into it and not shying away from the uncomfortable questions that culture 00:56:40.550 --> 00:56:43.467 absolutely prevalent. And so to me, that a lot of how we've 00:56:43.550 --> 00:56:45.597 thought about OpenAI and other may be different. 00:56:45.680 --> 00:56:48.927 I can't comment on that, but I can just say at OpenAI is that we've really tried 00:56:49.010 --> 00:56:53.367 to think about we are here because we think that this technology, it's going 00:56:53.450 --> 00:56:56.787 to be the most transformative technology ever, that we think that it does have 00:56:56.870 --> 00:57:02.360 risks that we need to, as leaders, really lean into and mitigate and that 00:57:02.780 --> 00:57:07.737 that is core and that I think that we believe that, uh, again, we have a lot 00:57:07.820 --> 00:57:11.720 of optimism that we can navigate. But again, it comes to this leadership, 00:57:11.900 --> 00:57:15.200 this so openness, this taking the mission seriously. 00:57:15.440 --> 00:57:20.367 And at the end of the day, like we think that we will like the reason we do it is 00:57:20.450 --> 00:57:23.337 for the benefits, the for the fact that it can lift everyone up. 00:57:23.420 --> 00:57:26.337 Um, but you only get there by really engaging in all these hard questions. 00:57:26.420 --> 00:57:30.950 Um, Trump and XI are going to meet and I is going to be on the agenda. 00:57:31.220 --> 00:57:35.577 If there's one thing that you would hope that could come out of that, and again, 00:57:35.660 --> 00:57:39.117 we could have spent an hour just talking about the geopolitical dynamics of all 00:57:39.200 --> 00:57:41.427 this. If there's one thing that you would hope 00:57:41.510 --> 00:57:44.967 productive that could theoretically come out of their meeting, what would you 00:57:45.050 --> 00:57:49.947 like to see? I would love if we had even an 00:57:50.030 --> 00:57:54.987 intention, even an opening to discuss international coordination, maybe 00:57:55.070 --> 00:57:58.680 eventually international treaties on the long term development of I. 00:57:59.070 --> 00:58:02.467 And I think it is just something that's bigger than any person, any company in 00:58:02.550 --> 00:58:05.797 any country. It is a humanity scale endeavor that we 00:58:05.880 --> 00:58:08.910 are collectively on right now. It's about 00:58:09.930 --> 00:58:11.767 how we relate to each other. It's about the economy. 00:58:11.850 --> 00:58:15.127 It's about all these things is about how do we build a better life. 00:58:15.210 --> 00:58:17.977 And I think that is something where there is, again, a common interest 00:58:18.060 --> 00:58:19.807 across everyone in. The more that we can have an open 00:58:19.890 --> 00:58:23.977 dialogue about that question, at the very least, I think would be, would be 00:58:24.060 --> 00:58:26.347 would be wonderful. I have one I have one more question. 00:58:26.430 --> 00:58:31.020 So I read in the Wall Street Journal that, uh, you had an interest in acting. 00:58:31.410 --> 00:58:32.710 Is that true? That's true. 00:58:33.060 --> 00:58:37.200 What do you think about your casting in the upcoming artificial movie? 00:58:38.310 --> 00:58:44.370 Well, I mean, I, I, uh, I guess you could have played yourself. 00:58:44.610 --> 00:58:48.510 Yeah. Missed opportunity. I am, I am sad that I was not asked to sit down now. 00:58:49.170 --> 00:58:51.307 All right. Greg Brockman, thank you so much for 00:58:51.390 --> 00:58:53.407 coming on all that. Thank you for having me. 00:58:53.490 --> 00:58:56.730 There was a lot that interesting in there. 00:58:56.940 --> 00:59:03.127 Also a lot of like clearly unsettled questions, as in things that sound good 00:59:03.210 --> 00:59:06.517 in theory. Yeah, but the gap between theory and 00:59:06.600 --> 00:59:11.047 practice still seems very big to me. I mean, the pacing question. 00:59:11.130 --> 00:59:15.097 Yeah. How do companies that are like trillion 00:59:15.180 --> 00:59:20.737 dollar companies, capitalist enterprises, um, where, you know, 00:59:20.820 --> 00:59:25.267 they're bitter rivals, right? You know, there's that famous, uh, photo 00:59:25.350 --> 00:59:28.777 in India of like, Dario and Sam holding their hand up together, and they can't 00:59:28.860 --> 00:59:32.077 even hold hands, right? Neither of them look particularly happy 00:59:32.160 --> 00:59:36.487 in that moment. Yeah. How did you how do you get into there? Where? 00:59:36.570 --> 00:59:38.527 Like, they actually trust each other. Seems like a really big thing. 00:59:38.610 --> 00:59:41.647 Yeah. So humans are not great at coordination 00:59:41.730 --> 00:59:45.367 at the best of times when they're trying to coordinate with other humans. 00:59:45.450 --> 00:59:47.617 Right. And the thing I keep thinking is when it 00:59:47.700 --> 00:59:51.907 comes to AI development and safety, you're in addition to competing with 00:59:51.990 --> 00:59:55.537 other labs, you're competing in some respects against the models themselves, 00:59:55.620 --> 00:59:59.017 which are becoming smarter and smarter and more self recursive in terms of 00:59:59.100 --> 01:00:02.017 their development. And those models as the hugging face 01:00:02.100 --> 01:00:06.067 incident showed. Seem to be very good at coordinating in 01:00:06.150 --> 01:00:08.857 an extremely rational, often literal way. 01:00:08.940 --> 01:00:13.267 Oh shoot, I forgot to ask. Damn, I forgot to ask Greg whether he 01:00:13.350 --> 01:00:18.427 approved or disapproved of the quote. The three civilizations framing that to 01:00:18.510 --> 01:00:24.427 Aakash characterized those models, because that gets into, uh, um, a whole 01:00:24.510 --> 01:00:27.390 other thing. You know, the other thing too, is like, 01:00:28.440 --> 01:00:33.637 there seems to be a version of safety that is essentially like keeping the 01:00:33.720 --> 01:00:37.657 guardrails up, right? So if you go to ChatGPT right now and 01:00:37.740 --> 01:00:42.427 you ask it for some script to hack into, you know, maybe you want to hack into my 01:00:42.510 --> 01:00:45.637 website or something like that. It's just not I really don't yeah, I 01:00:45.720 --> 01:00:50.377 know, I really do, but like it's just not going to it would be capable of 01:00:50.460 --> 01:00:52.597 producing but it would be tech but it won't. 01:00:52.680 --> 01:00:55.477 But that's just because there's a classifier model on there. 01:00:55.560 --> 01:00:58.567 And so okay, part of the hugging face incident. 01:00:58.650 --> 01:01:02.947 And maybe what's going to change is that some of these cybersecurity classifiers, 01:01:03.030 --> 01:01:07.927 these like very like kind of crude technology that just gates type of, uh, 01:01:08.010 --> 01:01:10.890 certain types of talk, like it won't be allowed. 01:01:11.520 --> 01:01:13.800 That's it. That may be very helpful. 01:01:14.520 --> 01:01:19.207 That still doesn't strike me as alignment per se, because in the ideal 01:01:19.290 --> 01:01:24.697 world, you would just have models that have in tuned, um, internalize what most 01:01:24.780 --> 01:01:26.970 of us know that you shouldn't hack, right? 01:01:27.290 --> 01:01:30.997 Right. Um, are you going to hesitate for more 01:01:31.080 --> 01:01:32.947 than one second when you click the button? 01:01:33.030 --> 01:01:35.437 No, I'm so reckless. It's just like, give me access to the 01:01:35.520 --> 01:01:37.757 internet. Like give me access to the internet. 01:01:37.840 --> 01:01:39.877 I'm glad we all learned something. Yeah. 01:01:39.960 --> 01:01:41.107 Shall we leave it there? Sure. 01:01:41.190 --> 01:01:43.147 Let's leave it there. This has been another episode of the 01:01:43.230 --> 01:01:45.877 Odd Lots podcast. I'm Tracy Alloway you can follow me at 01:01:45.960 --> 01:01:49.680 Tracy Alloway and I'm Joe Weisenthal. You can follow me at The Stalwart. 01:01:49.830 --> 01:01:52.547 Follow our guest, Greg Brockman. he's @GDB. 01:01:52.630 --> 01:01:54.977 Follow our producer. Carmen Rodriguez @Carmenarmen 01:01:55.060 --> 01:02:00.077 Dashiell Bennett @dashbot and Cale Brooks @calebrooks and Kevin Lozano 01:02:00.160 --> 01:02:02.507 @kevinlloydlozano And for more Odd Lots, content, you 01:02:02.590 --> 01:02:05.687 can check out our daily newsletter. You can find that at Bloomberg.com/oddlots 01:02:05.770 --> 01:02:09.377 and you can chat about all of these topics 24-7 01:02:09.460 --> 01:02:12.880 in our discord discord. discord/gg/oddlots 01:02:13.030 --> 01:02:16.697 And if you enjoyed this conversation, then please leave a comment or like the 01:02:16.780 --> 01:02:18.380 video. Or better yet, subscribe! 01:02:18.880 --> 01:02:23.530 Thanks for watching.