WEBVTT 00:00:00.000 --> 00:00:04.750 At the end of last year, we're like, all right, we are quite 00:00:05.130 --> 00:00:06.880 We haven't seen the models yet. 00:00:07.010 --> 00:00:08.927 You know, we're going to write this check for like, I don't know, 00:00:09.010 --> 00:00:11.507 like $100 billion, $200 billion. 00:00:11.590 --> 00:00:14.640 I'm like, Alex, are you sure this is going to work? 00:00:15.140 --> 00:00:16.640 It's like—Cheers. 00:00:17.600 --> 00:00:21.810 Cheers. Yeah. Cheers. Cheers. Cheers. 00:00:24.520 --> 00:00:27.360 So, give me the update on Muse. 00:00:27.480 --> 00:00:31.700 If you were looking at the third-party reports, 00:00:32.070 --> 00:00:38.580 you would see that this might be the fastest-growing agent or AI app 00:00:39.290 --> 00:00:40.790 that I think has existed yet. 00:00:41.540 --> 00:00:46.210 What are your guys' personal favorite Muse use cases, either yours or other people's? 00:00:47.510 --> 00:00:49.257 I was actually talking about this at dinner yesterday. 00:00:49.340 --> 00:00:52.720 My favorite thing is I have my Muse, whose name is Oiler, 00:00:53.760 --> 00:00:57.270 constantly source increasingly obscure teas 00:00:57.770 --> 00:01:01.980 from increasingly obscure vendors of tea. 00:01:02.100 --> 00:01:06.980 So at this point he's, on a weekly basis, my Muse is buying 00:01:07.400 --> 00:01:14.070 some random tea from some farm in Taiwan or some farm in Japan 00:01:14.360 --> 00:01:16.460 and sourcing it over and bringing it back. 00:01:16.660 --> 00:01:18.410 What's your favorite Muse use case? 00:01:18.830 --> 00:01:21.330 I don't know. I mean, I use it for a ton of stuff. 00:01:21.500 --> 00:01:22.333 I still like... 00:01:23.000 --> 00:01:26.170 My favorite both use case and probably eval for the product 00:01:26.590 --> 00:01:28.340 is having it coach me in MMA. 00:01:28.710 --> 00:01:34.340 So I put cameras in my gym, and I basically... 00:01:35.220 --> 00:01:38.727 It watches the cameras when I'm in there, and it gives me feedback in between 00:01:38.810 --> 00:01:42.147 the rounds, which is a very hard problem, it turns out, because not 00:01:42.230 --> 00:01:43.600 only does it have to... 00:01:43.940 --> 00:01:50.817 It basically has to process all the video quickly enough because you only 00:01:50.900 --> 00:01:52.607 get 1 minute in between the rounds. 00:01:52.690 --> 00:01:54.197 Oh, it coaches you between the rounds? Yeah. 00:01:54.280 --> 00:01:56.537 I want feedback. I want feedback on how to win. 00:01:56.620 --> 00:01:57.617 You know what I mean? It's like, come on. 00:01:57.700 --> 00:01:58.907 Did you have to instruct it... 00:01:58.990 --> 00:02:02.120 I've wondered about this too, because looking at videos is 00:02:02.620 --> 00:02:03.920 a really good AI use case. 00:02:04.580 --> 00:02:07.837 It feels like you can't just give it a multi-gigabyte video file. 00:02:07.920 --> 00:02:10.320 So do you tell it how to slice stills out of it? 00:02:10.630 --> 00:02:12.730 No, it basically... Yeah, it pulls stills. 00:02:12.920 --> 00:02:15.220 I think it kind of figured that out by itself. 00:02:15.340 --> 00:02:20.850 Um, it turns out that Muse has pretty good native understanding 00:02:21.100 --> 00:02:25.230 of MMA, so that's good. So like the multi-modal— The pre-training. 00:02:25.600 --> 00:02:30.857 Yeah, someone on the team knew. Now... 00:02:30.940 --> 00:02:32.657 Just kind of out there in the corpus. 00:02:32.740 --> 00:02:38.740 I think it's a good test of the kind of coding ability that... 00:02:39.070 --> 00:02:41.867 Because really basically what it's doing, it's like, you know, 00:02:41.950 --> 00:02:45.707 within the agent, I'm not seeing any code, but it basically is writing this whole 00:02:45.790 --> 00:02:47.917 video processing, image processing pipeline. 00:02:48.000 --> 00:02:50.547 Yes. And I'm constantly telling it like, all right... 00:02:50.630 --> 00:02:53.427 Like, look you took 30 seconds to give me that feedback. 00:02:53.510 --> 00:02:55.597 I only have a 60-second break. Like, you gotta go faster. 00:02:55.680 --> 00:02:57.780 Just optimize yourself, optimize yourself. 00:02:58.090 --> 00:03:02.057 And it's, I think, a good example of how stuff is 00:03:02.140 --> 00:03:05.267 going to go in the future where basically everyone is actually 00:03:05.350 --> 00:03:06.607 going to be writing tons of code. 00:03:06.690 --> 00:03:08.977 They're just not going to know that they're writing code. 00:03:09.060 --> 00:03:12.020 What kind of MMA steers does a model give you? 00:03:12.900 --> 00:03:16.567 Um, you want to see? Yeah. Basically, let me see if I can pull some stuff out. 00:03:16.650 --> 00:03:22.617 But basically it gives me a lot of stuff about balance and, like, hip position. 00:03:22.700 --> 00:03:26.327 Or it's like when you're kind of in side control here. 00:03:26.410 --> 00:03:28.997 It's just like you didn't get this underhook 00:03:29.080 --> 00:03:30.207 in this place so you're going to... 00:03:30.290 --> 00:03:33.250 You're vulnerable to this getting rolled over here. 00:03:33.340 --> 00:03:35.677 I'll pull up some stuff. You guys should try this. 00:03:35.760 --> 00:03:38.597 It was definitely one of his first use cases actually of even 00:03:38.680 --> 00:03:40.387 the initial prototypes. Yeah. Yeah. 00:03:40.470 --> 00:03:43.720 I mean, I did this with OpenClaw before we even had Muse working. 00:03:43.810 --> 00:03:45.140 It was just like... 00:03:45.310 --> 00:03:49.770 Then that's why I was saying it's kind of like become an eval for this too. 00:03:50.060 --> 00:03:52.527 It's good. I actually didn't know that it was an eval for the product. 00:03:52.610 --> 00:03:55.570 So noted. Yeah. There you go. 00:03:57.280 --> 00:03:58.180 How is all this... 00:03:59.110 --> 00:04:02.870 In the model's training data, like the fine-grained, 00:04:03.370 --> 00:04:07.700 this is what an MMA fighter looks like when they're appropriately 00:04:07.790 --> 00:04:10.497 balanced, and this is when they're like too much on the back foot. 00:04:10.580 --> 00:04:13.750 Just, I'm wondering, how has it consumed so many images of 00:04:14.460 --> 00:04:17.210 appropriate versus inappropriate MMA posture? 00:04:17.380 --> 00:04:17.837 Here you can see it. 00:04:17.920 --> 00:04:22.090 It just pulls out frames from it and it's like too much on my back leg there. 00:04:22.600 --> 00:04:24.810 Yeah, but is this the emergent intelligence? 00:04:24.930 --> 00:04:26.680 Like it does actually... 00:04:27.480 --> 00:04:30.140 Again, does it know from seeing lots of images of MMA 00:04:30.560 --> 00:04:33.027 or does it know because it understands the concept of where you're 00:04:33.110 --> 00:04:36.317 meant to be balancing? It's read books about MMA, and then it sees that you're 00:04:36.400 --> 00:04:37.407 too much on the back foot, or— 00:04:37.490 --> 00:04:43.870 Well, we did train the models with a lot of visual perception data. 00:04:44.280 --> 00:04:47.957 Yeah. So, I mean, I think that that's like kind of making good multimodal models. 00:04:48.040 --> 00:04:52.290 I mean, this gets into like, I think, a core approach thing that we have, 00:04:52.380 --> 00:04:55.180 which is we're training the models end-to-end to be good 00:04:55.590 --> 00:04:56.797 personal agents, right? 00:04:56.880 --> 00:05:00.760 So like the question is why would Meta want to build these models? 00:05:01.130 --> 00:05:04.260 First of all, I think just superintelligence overall is going 00:05:04.350 --> 00:05:07.307 to allow us to run the company better and build all kinds of different things. 00:05:07.390 --> 00:05:11.767 But for the products that we're building, there's certain capabilities and certain 00:05:11.850 --> 00:05:17.397 mixes of skills that basically are just much more applicable to these cases. 00:05:17.480 --> 00:05:20.947 So I mean, this is one of the things that Alex focused on from the beginning 00:05:21.030 --> 00:05:23.827 was like, all right, we really want to make sure that our 00:05:23.910 --> 00:05:27.407 multimodal program is good because that's disproportionately important for the types 00:05:27.490 --> 00:05:29.667 of things that people use all these services for. 00:05:29.750 --> 00:05:31.800 How do you train a good multimodal model? 00:05:32.420 --> 00:05:34.707 Is this where Meta just has an advantage because you guys have so 00:05:34.790 --> 00:05:36.707 much image and video data? It's very helpful. 00:05:36.790 --> 00:05:42.677 Yeah. And I think ultimately, a lot of this, it comes down to how are 00:05:42.760 --> 00:05:43.597 you focused on it, right? 00:05:43.680 --> 00:05:47.927 I think because of the fact that we wanted to build personal superintelligence. 00:05:48.010 --> 00:05:51.270 And fundamentally, I think most people in the world really 00:05:51.600 --> 00:05:53.687 experience the world in a highly multimodal way. 00:05:53.770 --> 00:05:58.940 They like talking to other people and listening to things and seeing videos 00:05:59.480 --> 00:06:01.107 and filming things of their own. 00:06:01.190 --> 00:06:04.780 I think most consumers have this highly multimodal experience. 00:06:04.910 --> 00:06:09.907 And especially earlier this year, there was actually a point of tension 00:06:09.990 --> 00:06:14.830 almost in the industry because I think so much of the consensus view within the AI 00:06:14.920 --> 00:06:19.130 industry was that it was just all about coding and coding was all that mattered. 00:06:19.800 --> 00:06:23.130 And I think that for the success of a product like 00:06:23.220 --> 00:06:28.010 Muse and the success of a lot of these consumer use cases, it really depends on 00:06:28.180 --> 00:06:30.480 the ability to be strong and multimodal. Yeah. 00:06:30.970 --> 00:06:33.017 I think we're just kind of scratching the surface of that. 00:06:33.100 --> 00:06:35.767 I mean, a lot of the initial theory that we had was that, 00:06:35.850 --> 00:06:39.647 I mean, most people don't actually like reading that much, right? 00:06:39.730 --> 00:06:43.900 So even the text-based interaction model, I think it's something that people do. 00:06:43.990 --> 00:06:45.280 I mean, everyone texts. 00:06:46.320 --> 00:06:49.537 But when you think about stuff like the glasses that we're working on, 00:06:49.620 --> 00:06:52.020 or even our other products that are very visual, 00:06:52.750 --> 00:06:56.417 the glasses basically will eventually just be a pure multimodal product, right? 00:06:56.500 --> 00:07:01.420 It's like you can let it hear what you are listening to, and it can talk to you, 00:07:01.630 --> 00:07:03.730 and it can see what you see if you let it. 00:07:03.880 --> 00:07:06.680 And then eventually, I don't even know if we're going to 00:07:07.260 --> 00:07:08.347 have an app store on the glasses. 00:07:08.430 --> 00:07:12.310 It'll just be like a multimodal generation that kind of creates whatever 00:07:13.270 --> 00:07:14.810 you need right in your vision. 00:07:15.480 --> 00:07:17.727 And I imagine something like that will probably end up powering a lot 00:07:17.810 --> 00:07:20.360 of the apps that we build too. 00:07:20.980 --> 00:07:23.780 But yeah, I mean, we're still in the early days of that. 00:07:24.900 --> 00:07:27.900 But for now, the perception work has been really impressive. 00:07:28.700 --> 00:07:31.700 I want to ask you guys about the product space here in AI 00:07:32.160 --> 00:07:33.700 and try something out on you. 00:07:33.830 --> 00:07:37.707 If I was to describe to you, say, with Stripe, what I think a winning 00:07:37.790 --> 00:07:41.207 product looks like, it would take me pages and pages to talk about because we have 00:07:41.290 --> 00:07:43.377 the developer experience and we have to talk about all the international 00:07:43.460 --> 00:07:46.257 heterogeneity of payment methods. Then we have to talk about stablecoins. 00:07:46.340 --> 00:07:47.887 Like, it would take me a long time. 00:07:47.970 --> 00:07:51.800 Whereas I notice a phenomenon in AI where it feels like the winning products 00:07:52.050 --> 00:07:56.057 collapse very quickly onto an agreed-upon product space 00:07:56.140 --> 00:07:59.517 that everyone is pursuing, and so transformers worked, and then 00:07:59.600 --> 00:08:02.770 like the chat agent UI really worked. 00:08:03.230 --> 00:08:04.780 And now we are seeing... 00:08:04.980 --> 00:08:09.860 You mentioned OpenClaw earlier, the kind of OpenClaw persistent agent, 00:08:09.950 --> 00:08:11.990 has a computer, able to do things. 00:08:12.120 --> 00:08:13.867 You know, computer use out there in the world view. 00:08:13.950 --> 00:08:15.787 We're collapsing on that product space. 00:08:15.870 --> 00:08:18.720 But there's something about AI where the winning products 00:08:19.000 --> 00:08:23.880 tend to be very simple, pursued by all the companies that are taking it seriously. 00:08:24.300 --> 00:08:26.297 You could describe on the back of a postcard... 00:08:26.380 --> 00:08:27.927 Like you could describe very concisely. 00:08:28.010 --> 00:08:30.927 And obviously, implementing them is massively complicated. 00:08:31.010 --> 00:08:32.800 Like, we'll get into the new model. 00:08:32.890 --> 00:08:35.927 There's, you know, huge amounts of frontier work 00:08:36.010 --> 00:08:37.477 that goes into making a good version. 00:08:37.560 --> 00:08:42.650 But the winning brand—now you're talking about the glasses or the future versions. 00:08:42.940 --> 00:08:48.190 But somehow there's a very small number of clearly correct products in AI 00:08:48.780 --> 00:08:53.570 as opposed to it being this, like, very broad and complicated product space. 00:08:54.490 --> 00:08:56.290 I kind of disagree. Sure. 00:08:57.870 --> 00:08:59.957 Well, first of all, I think part of the theory 00:09:00.040 --> 00:09:03.790 from the beginning was that this kind of agent or whatever form it took was 00:09:03.880 --> 00:09:09.630 going to be quite a different thing from this text-based assistant, right? 00:09:09.720 --> 00:09:12.427 Every company has one of those. I mean, ours is Meta AI. 00:09:12.510 --> 00:09:14.807 It reaches a ton of people, but I mean, it's like the same thing 00:09:14.890 --> 00:09:16.347 that all these companies have. 00:09:16.430 --> 00:09:18.597 And we, I think, 00:09:18.680 --> 00:09:23.230 specifically think that that is used for very informational purposes, 00:09:23.350 --> 00:09:26.817 but people don't think about it as like a thing that gets stuff done for them. 00:09:26.900 --> 00:09:28.647 It's retrieval, not delegation or something. 00:09:28.730 --> 00:09:31.700 Yeah, and even like when you say it has a computer... 00:09:31.950 --> 00:09:33.947 Yeah, but there's lots of different ways to do that, right? 00:09:34.030 --> 00:09:39.370 You know, I think that it's only kind of this most recent wave that 00:09:39.870 --> 00:09:44.037 gives the agent a dedicated VM, which I think is actually incredibly 00:09:44.120 --> 00:09:49.667 important for making you feel confident that the security and privacy of your 00:09:49.750 --> 00:09:53.967 information is—it's not just like, okay, it's not connecting to some elastic 00:09:54.050 --> 00:09:56.680 compute that just everyone uses. 00:09:57.050 --> 00:10:00.730 But my claim isn't that it's easy to implement, nor that it's obvious— 00:10:00.850 --> 00:10:02.017 I think there are going to be a lot of things. 00:10:02.100 --> 00:10:04.437 Yeah. Okay. You think there's going to be a lot of different things? 00:10:04.520 --> 00:10:07.187 Because again, my claim is that now that people see Muse, 00:10:07.270 --> 00:10:12.567 everyone looks at that and says, "Oh yeah, that's clearly how you want the AI agent 00:10:12.650 --> 00:10:16.100 product to work, and that the future AI agent products that we'll see 00:10:16.370 --> 00:10:19.080 will be fairly Muse-shaped." So I think... 00:10:19.410 --> 00:10:23.160 My take on this is, I think we're just so early in the realm 00:10:23.290 --> 00:10:27.080 of what AI products will look like that I think a lot of the interesting 00:10:27.210 --> 00:10:30.547 differentiation about the products kind of gets collapsed in how 00:10:30.630 --> 00:10:31.927 we view the products today. 00:10:32.010 --> 00:10:35.507 I think the traditional way in which we wrote down and think about 00:10:35.590 --> 00:10:36.840 product specs, that is... 00:10:38.010 --> 00:10:41.720 Those aren't the relevant dimensions and levers and axes 00:10:42.140 --> 00:10:46.560 to consider the important design decisions on AI-native products. 00:10:47.100 --> 00:10:52.110 And so, Muse, for example, I think so much attention was put in the design 00:10:52.320 --> 00:10:54.070 of the model for the product. 00:10:54.360 --> 00:10:57.867 And there were a lot of very intentional decisions around, in all these situations, 00:10:57.950 --> 00:11:00.200 how exactly should the model behave? 00:11:00.370 --> 00:11:02.787 And what's the exact personality of the model in these situations? 00:11:02.870 --> 00:11:05.787 And how should it handle these sorts of tool-use scenarios? 00:11:05.870 --> 00:11:07.920 And these are very intentional decisions. 00:11:08.420 --> 00:11:14.297 And probably in the same way that the full spec of how we wanted the model to behave 00:11:14.380 --> 00:11:17.257 in all these scenarios would be an extremely long document. 00:11:17.340 --> 00:11:23.060 But because of just how we think about products today, it's just AI. 00:11:23.810 --> 00:11:26.097 It's kind of like how when mobile first started happening, 00:11:26.180 --> 00:11:28.357 it was like, "Oh, it's just an app and everything is just 00:11:28.440 --> 00:11:32.520 an app." Well, it turns out there's a lot of nuance in what exactly an app is. 00:11:33.360 --> 00:11:37.117 And we're kind of in this phase where, oh, it's just AI, or it's just an agent, 00:11:37.200 --> 00:11:39.487 or it's just an agent with a VM or something like that. 00:11:39.570 --> 00:11:44.410 But the interesting differentiations we have to tease out. 00:11:44.700 --> 00:11:46.917 Okay. So you guys think I'm making the classic mistake, which is like, 00:11:47.000 --> 00:11:49.337 my domain is complex and this domain is, again, from a product 00:11:49.420 --> 00:11:51.337 point-of-view, it's simple. All payment processors seem the same. 00:11:51.420 --> 00:11:53.880 Yeah, no, I really agree with what you're saying. 00:11:54.050 --> 00:11:57.260 One thing that we talked about from early on, is that basically 00:11:58.010 --> 00:12:00.967 there is this pattern where I think the products are getting thinner 00:12:01.050 --> 00:12:03.307 and the model is getting to be more of the product, right? 00:12:03.390 --> 00:12:04.677 So one of the things that we say internally is like, 00:12:04.760 --> 00:12:05.897 the model is the product. Yes. 00:12:05.980 --> 00:12:08.187 Right? So, and when you're talking about an agent, 00:12:08.270 --> 00:12:10.920 a lot of the complex product work that went into Muse 00:12:11.310 --> 00:12:14.527 is basically around the privacy and security model for handling your data 00:12:14.610 --> 00:12:15.857 and the things that get connected to it. 00:12:15.940 --> 00:12:18.737 And that, I think, is something that we're just very proud of that we 00:12:18.820 --> 00:12:19.657 really focused on. 00:12:19.740 --> 00:12:22.867 But that's a lot of the product surface that we need to innovate on. 00:12:22.950 --> 00:12:26.580 But the capabilities, the personality, a lot of the stuff is like 00:12:26.950 --> 00:12:28.247 you're training into the model. 00:12:28.330 --> 00:12:32.290 And whether they're the people in the lab who are working on that or the people 00:12:32.540 --> 00:12:34.790 on Nat's team who are working on the product, 00:12:34.960 --> 00:12:38.717 I think when Nat is thinking about what is the key stuff that we need to make 00:12:38.800 --> 00:12:42.307 the product feel good, he's talking about post-training the model, right? 00:12:42.390 --> 00:12:43.177 Not like... 00:12:43.260 --> 00:12:46.137 I mean, of course the UI of the product matters too, but the most important 00:12:46.220 --> 00:12:48.810 part of the UI is the product. Yeah, exactly. 00:12:48.930 --> 00:12:53.610 I think there's like, there's still very viscerally, I think from a user, 00:12:53.860 --> 00:12:56.480 there's a feeling of using Muse over time. 00:12:57.070 --> 00:13:00.567 And maybe some meaningful portion of that is just the reliability 00:13:00.650 --> 00:13:02.157 of Muse to handle certain tasks. 00:13:02.240 --> 00:13:05.330 But then another meaningful portion is how is it proactive? 00:13:05.490 --> 00:13:07.287 And in what context is it proactive? 00:13:07.370 --> 00:13:10.537 And when it talks to you, how do you feel about how it talks to you? 00:13:10.620 --> 00:13:13.080 And so there's clearly a lot of... 00:13:14.250 --> 00:13:17.717 And I'm starting to see more and more of this actually show up on the X feed, 00:13:17.800 --> 00:13:20.837 which are people are starting to tease out these, like... 00:13:20.920 --> 00:13:24.260 They're all proactive agents, but what are the actual meaningful 00:13:24.350 --> 00:13:25.810 differences between them? 00:13:26.180 --> 00:13:28.687 And these, I think, are the interesting product decisions, 00:13:28.770 --> 00:13:32.477 which, you know, the funny thing is how do you specify these product decisions? 00:13:32.560 --> 00:13:36.460 Well, it's ultimately in model design decisions, which ultimately boil down to 00:13:36.820 --> 00:13:40.067 training methods or data or evals or, you know, all these things. 00:13:40.150 --> 00:13:44.280 Yes. As we talk about product space, one notable Muse detail is 00:13:44.370 --> 00:13:45.947 that there is no model selector. 00:13:46.030 --> 00:13:48.410 and no effort selector. 00:13:49.120 --> 00:13:52.577 Do you guys think that's just obviously correct in a consumer AI product, or— 00:13:52.660 --> 00:13:54.160 I just want to make it simple. 00:13:54.250 --> 00:13:55.850 Yeah. I think that we've been... 00:13:57.960 --> 00:14:01.110 I think somewhat infamously, a lot of the AI products have been 00:14:01.630 --> 00:14:07.927 designed by a highly technical group that do not have these sort of randomly 00:14:08.010 --> 00:14:10.980 sampled American in their mental model. 00:14:11.520 --> 00:14:12.640 And so the products... 00:14:13.060 --> 00:14:18.900 I mean, the naming of some of the models is like so egregious. 00:14:19.020 --> 00:14:19.817 The numbers go backwards? 00:14:19.900 --> 00:14:22.567 Some of the times the numbers go backwards and sometimes it's just like a totally 00:14:22.650 --> 00:14:24.657 different word and you have another set of words. 00:14:24.740 --> 00:14:27.140 And like, I just like, how is somebody who's not 00:14:27.740 --> 00:14:32.827 carefully reading API design specs supposed to know 00:14:32.910 --> 00:14:35.497 what they're supposed to pick? Like, it's just far too confusing. 00:14:35.580 --> 00:14:37.917 Okay, we're talking about models a lot. Tell me about the new model. 00:14:38.000 --> 00:14:38.960 It sounds awesome. 00:14:39.670 --> 00:14:41.587 Yeah, I mean, I think we're quite proud of this. 00:14:41.670 --> 00:14:45.720 Um, yeah, the new model—Muse Wonder—it's a new class of model 00:14:45.930 --> 00:14:47.007 that is at the frontier. 00:14:47.090 --> 00:14:49.097 So we've basically been working on this since the beginning. 00:14:49.180 --> 00:14:53.597 Basically, when you're doing these pre-trains, you don't just go to whatever 00:14:53.680 --> 00:14:55.477 the biggest cluster is that you have. 00:14:55.560 --> 00:14:59.110 You want to work your way up the ladder training models that are 00:14:59.650 --> 00:15:04.610 maybe 4 or 5 times bigger, and then you just keep on repeating that process. 00:15:05.450 --> 00:15:11.367 So the first set of models that we've had Muse Spark was all based on this initial 00:15:11.450 --> 00:15:14.000 pre-training that we did in, what was it, December? 00:15:14.120 --> 00:15:15.497 Yeah, the end of last year. Yeah. 00:15:15.580 --> 00:15:16.917 And, and it's actually... 00:15:17.000 --> 00:15:21.457 I think we've all just been quite surprised and impressed with how 00:15:21.540 --> 00:15:25.967 much intelligence and capability the team has been able to build 00:15:26.050 --> 00:15:30.050 into a model which is like frankly not that big, right? 00:15:30.140 --> 00:15:33.680 I mean, and then after that we were working on this next model, 00:15:33.970 --> 00:15:36.897 and the pre-training has been done for a bit, and we've been working 00:15:36.980 --> 00:15:39.400 on the post-training, and now it is here. 00:15:40.230 --> 00:15:42.360 So it's the next class of model. 00:15:42.480 --> 00:15:44.277 It is significantly bigger. 00:15:44.360 --> 00:15:48.740 So I think you you will feel that in terms of the nuance in which it can think. 00:15:50.030 --> 00:15:50.657 And it really. 00:15:50.740 --> 00:15:55.077 A lot of the areas that we focused on were making it the best personal 00:15:55.160 --> 00:15:57.080 and professional agentic model. 00:15:57.210 --> 00:16:01.710 Some of the different things that you can optimize for that's it for us, and then 00:16:02.380 --> 00:16:06.757 yeah, a lot of the basic capabilities that we focused on building in. 00:16:06.840 --> 00:16:11.800 Very good memory, very good ability to persist through complicated tasks. 00:16:11.890 --> 00:16:15.557 But a ton of work went into this across the lab and the product 00:16:15.640 --> 00:16:16.980 teams and all of this. 00:16:17.310 --> 00:16:21.310 And I think it's also just sort of a testament to the the team being able 00:16:21.400 --> 00:16:24.857 to methodically figure out the process to scale these models. 00:16:24.940 --> 00:16:28.320 So I think this is also just, you know, 00:16:30.150 --> 00:16:33.157 a milestone on the way towards the time 00:16:33.240 --> 00:16:36.367 between each subsequently bigger class of models, I think will 00:16:36.450 --> 00:16:37.950 now start to compress. 00:16:38.080 --> 00:16:40.497 And that's very exciting for the type of stuff that we can build. 00:16:40.580 --> 00:16:42.337 At Meta or in the industry generally? 00:16:42.420 --> 00:16:46.710 At Meta. I mean, I think you're seeing something like this everywhere else, but— 00:16:46.800 --> 00:16:48.800 Is this an RSI effect or something else? 00:16:49.050 --> 00:16:51.590 Well, I think at Meta, part of this is just... 00:16:52.220 --> 00:16:53.053 We needed to... 00:16:53.510 --> 00:16:56.427 Well, I think quite famously, we were not happy with the trajectory 00:16:56.510 --> 00:16:59.847 that we were on with Llama 4. We kind of like rebuilt the whole team. 00:16:59.930 --> 00:17:02.977 That's been this kind of crazy thing over the last year and a half that we should 00:17:03.060 --> 00:17:04.487 probably spend a bunch of time on. 00:17:04.570 --> 00:17:04.817 Oh, we'll get to that. Yeah, yeah. 00:17:04.900 --> 00:17:11.067 Some war stories. But the team basically had to go build up like different tiers 00:17:11.150 --> 00:17:16.910 of models to get to this one. And now that we are kind of at the frontier, 00:17:17.490 --> 00:17:21.497 now everything that we do going forward will just be pushing the frontier, 00:17:21.580 --> 00:17:25.087 as opposed to there's a lot of background work that we needed to do before that. 00:17:25.170 --> 00:17:27.500 I don't know, what would you say? Yeah. 00:17:27.590 --> 00:17:29.547 Well, I think we're really excited about it. 00:17:29.630 --> 00:17:35.467 We will have 2 models in this series that are going to be available. 00:17:35.550 --> 00:17:39.140 So there's Muse Wonder and Muse Wonder Full. 00:17:39.850 --> 00:17:41.307 Laugh? Yeah, all right, there you go. 00:17:41.390 --> 00:17:45.310 Yeah, very good. Uh, which is basically a much higher— Studio band version. 00:17:45.400 --> 00:17:47.437 Yeah, no, a much higher-end model. 00:17:47.520 --> 00:17:49.777 It's the full version of Muse 1. 00:17:49.860 --> 00:17:53.740 I know, I got it. Yeah, yeah, it's not— It's 2 words. Wonder full version. 00:17:56.410 --> 00:17:58.577 You're clearly a father. What's up? 00:17:58.660 --> 00:18:00.787 This is an excellent dad joke. Yeah, there you go. 00:18:00.870 --> 00:18:01.827 Um, and, and the performance... 00:18:01.910 --> 00:18:05.830 I mean, we've seen like incredible performance across a slate of evaluations 00:18:05.920 --> 00:18:08.210 on personal, professional agent tasks. 00:18:08.300 --> 00:18:10.467 It's state-of-the-art based on our measurement. 00:18:10.550 --> 00:18:14.217 And then there's a number of, I think, specific highlights which we're 00:18:14.300 --> 00:18:17.700 really excited about. Which we've designed into the model over time. 00:18:17.890 --> 00:18:21.890 So active memory, like Mark mentioned, is one area where 00:18:22.600 --> 00:18:26.687 we think that to build personal agents, personal professional agents, 00:18:26.770 --> 00:18:31.480 like memory is just one of the clear requirements to build reliable agents 00:18:31.690 --> 00:18:33.657 and is one of the areas where we've invested, I think, 00:18:33.740 --> 00:18:36.987 disproportionately actually compared to many of the other efforts. 00:18:37.070 --> 00:18:40.407 And we have our own internal evals of Memory Bench which demonstrate 00:18:40.490 --> 00:18:42.090 a lot of the capabilities there. 00:18:42.910 --> 00:18:45.310 What does memory mean in a model to a layperson? 00:18:46.250 --> 00:18:47.497 Is it just how much stuff it can... 00:18:47.580 --> 00:18:50.337 Like the working memory of if you're working on a task, how much stuff 00:18:50.420 --> 00:18:52.170 the model can kind of keep in its— 00:18:52.840 --> 00:18:54.387 Yeah. How's it different from context? 00:18:54.470 --> 00:18:57.090 I think it's in the experience of using the model. 00:18:57.390 --> 00:19:01.597 How well does it remember things that you mentioned to it over days, weeks, 00:19:01.680 --> 00:19:05.017 or months, and then leverage those in the process of doing future tasks? 00:19:05.100 --> 00:19:07.860 I see. This is the stuff that's buggy AF in OpenClaw. 00:19:08.150 --> 00:19:10.700 Like, I think we're all extremely fond of OpenClaw. 00:19:11.150 --> 00:19:15.067 Very happy OpenClaw user, but it ultimately does this compaction 00:19:15.150 --> 00:19:18.327 where it has like the long-term memory and the working short-term memory or whatever. 00:19:18.410 --> 00:19:22.207 And I think once people use OpenClaw for a while, they find the memory pretty buggy. 00:19:22.290 --> 00:19:27.540 Yeah. So we've designed basically both the model and a system for the model to use 00:19:27.710 --> 00:19:31.110 to develop a much stronger form of memory. We call it active memory. 00:19:31.250 --> 00:19:34.880 Yeah. In addition, as Mark mentioned on the persistence— 00:19:35.010 --> 00:19:37.507 But so this is outside the model, like the model exists and it's— 00:19:37.590 --> 00:19:39.927 No, no, but some of the capabilities that you're talking about, 00:19:40.010 --> 00:19:43.137 like the compaction, are things that the model does, right? 00:19:43.220 --> 00:19:45.687 So basically being able to do this well, both with— 00:19:45.770 --> 00:19:47.807 It's not a harness, it's in the model itself. 00:19:47.890 --> 00:19:50.230 Yes, it's in the model itself. Exactly, yeah. 00:19:50.810 --> 00:19:55.490 So it's aware of its memory as a core system. 00:19:55.820 --> 00:19:59.870 And it's about how it's able to leverage various sources to be able to 00:20:00.950 --> 00:20:03.957 conduct tasks in the future, as well as understand its ability to do 00:20:04.040 --> 00:20:06.997 compaction, how it should do that compaction most effectively, et cetera. 00:20:07.080 --> 00:20:09.957 Okay. But this is a big deal because I think a lot of people's early experience 00:20:10.040 --> 00:20:12.840 of AI agents has been buggy memory. 00:20:13.130 --> 00:20:14.750 And so, yeah, improving that is— 00:20:14.920 --> 00:20:17.257 And to be clear, there's still plenty of room to improve. 00:20:17.340 --> 00:20:20.217 Our evaluations show this, but this is something that we're quite 00:20:20.300 --> 00:20:22.557 proud of, and we think this is a meaningful contribution. 00:20:22.640 --> 00:20:23.847 Okay. So active memory is one thing. 00:20:23.930 --> 00:20:24.637 Yeah. Active memory. 00:20:24.720 --> 00:20:28.807 Active focus is another, which is effectively just the ability for the model 00:20:28.890 --> 00:20:31.520 to, when conducting tasks, 00:20:31.610 --> 00:20:34.360 be able to carry them through, especially as we've seen 00:20:34.730 --> 00:20:39.030 Muse users as well as Muse Code users, give higher and higher complexity 00:20:39.240 --> 00:20:41.947 and more and more ambiguous and more and more complex tasks, 00:20:42.030 --> 00:20:45.117 we want the model to be able to conduct those and do those very effectively. 00:20:45.200 --> 00:20:48.700 And this is one of the things actually in early dogfooding and testing 00:20:48.790 --> 00:20:52.297 that a lot of people have noticed about the model that is sort of special 00:20:52.380 --> 00:20:55.930 and stands out with respect to other models that they've tried or used. 00:20:57.170 --> 00:20:59.630 The third one is domain expertise. 00:20:59.720 --> 00:21:04.510 What is focus in AI terms? Is it like putting a lot of compute on the task? 00:21:04.600 --> 00:21:07.310 Is it not getting distracted with other— 00:21:08.180 --> 00:21:09.937 Kind of, persistence? Yeah, like persistence. 00:21:10.020 --> 00:21:14.027 It's basically like the ability to conduct complete tasks over very long horizons. 00:21:14.110 --> 00:21:19.820 Okay. And then when faced with various setbacks along the way, 00:21:20.240 --> 00:21:24.070 the ability to identify other solutions to those problems in an aligned way. 00:21:24.830 --> 00:21:25.740 Okay. 00:21:26.410 --> 00:21:31.247 Then there's... It's actually a big step up in coding capabilities, 00:21:31.330 --> 00:21:32.377 so frontier coding capabilities. 00:21:32.460 --> 00:21:37.420 So it is actually state-of-the-art on DeepSWE and, SWE-Marathon 00:21:38.420 --> 00:21:41.887 and does very well on other benchmarks like Terminal-Bench 4 and many others. 00:21:41.970 --> 00:21:45.430 So we think that users who use it for coding, 00:21:45.640 --> 00:21:49.557 which will be a minority of all of our users, but the ones who do will see 00:21:49.640 --> 00:21:51.440 a big improvement and a big step up. 00:21:51.680 --> 00:21:55.940 Okay, so we should also just be using Muse Wonder Full for coding at Stripe? 00:21:56.440 --> 00:21:59.277 Potentially, yeah. I mean, let's go out there and— 00:21:59.360 --> 00:22:01.567 But like if it's state of the art on SWE-bench, yeah. 00:22:01.650 --> 00:22:04.277 We'd be missing out. Yeah, no, it's a big improvement on coding. 00:22:04.360 --> 00:22:07.907 I mean, obviously, we want people to try it and see how they 00:22:07.990 --> 00:22:11.540 experience it, but we've been very pleased in our own internal testing. 00:22:12.160 --> 00:22:14.627 Okay, so this is— Yeah, I think people will like it for a lot of things. 00:22:14.710 --> 00:22:19.417 Okay, so this is both just your latest frontier model, like people can come, 00:22:19.500 --> 00:22:22.217 you know, developers can come directly to you, they can use it on OpenRouter, 00:22:22.300 --> 00:22:26.850 like they can just consume it as a model, and it's also a big upgrade 00:22:27.680 --> 00:22:33.020 inside of Muse that people will just feel as active users of Muse. 00:22:33.560 --> 00:22:33.727 Yeah. 00:22:33.810 --> 00:22:36.647 Well, what we're going to do in Muse, because it's a lot bigger and it's more 00:22:36.730 --> 00:22:43.530 expensive to serve, we're going to start off primarily in paid subscriptions. 00:22:43.740 --> 00:22:45.827 So what we want to do with Muse as a product, we want it to be 00:22:45.910 --> 00:22:47.240 free for everyone to use. 00:22:47.740 --> 00:22:51.090 So we have this huge amount of free tokens that people can use with 00:22:51.290 --> 00:22:52.740 with Muse Spark as the model. 00:22:53.200 --> 00:22:58.500 But if you want to use Wonder for most of the hard tasks, then 00:23:00.250 --> 00:23:02.710 at least for now, you'll use the paid version. 00:23:03.170 --> 00:23:06.757 But, you know, I mean, eventually we'll, you know how quickly the field moves. 00:23:06.840 --> 00:23:09.807 Yes. And we hope to, you know, at some point not too far from now, 00:23:09.890 --> 00:23:13.770 have a model that is as good as Wonder, but in the Spark form factor. 00:23:14.270 --> 00:23:17.607 And that will be a great thing to be able to serve hundreds of millions 00:23:17.690 --> 00:23:20.687 of people around the world. Okay, so you go to Muse, you subscribe to the—what's it 00:23:20.770 --> 00:23:22.737 called—like the premium or pro plan, or... 00:23:22.820 --> 00:23:26.447 Pay money. Yeah. And just don't stop until you've taken out your credit card. 00:23:26.530 --> 00:23:28.487 Yeah. And then you'll be able to... 00:23:28.570 --> 00:23:30.827 You'll select it, or it'll just magically— Oh, it just gets better. 00:23:30.910 --> 00:23:32.537 It just gets better for you. Yeah. 00:23:32.620 --> 00:23:39.540 And then it'll also be available via API, on the Meta Model API and OpenRouter. 00:23:40.210 --> 00:23:45.010 And then it'll also be available for Muse Code users. 00:23:46.170 --> 00:23:50.430 And then Muse Wonder Full will be available to subscribers of Muse Code. 00:23:51.560 --> 00:23:53.720 So it's pretty exciting. 00:23:53.810 --> 00:23:56.897 I think we want to, as Mark mentioned, it is a much larger model. 00:23:56.980 --> 00:23:59.607 And so we're working through how to serve that most effectively 00:23:59.690 --> 00:24:00.977 to as many people as possible. 00:24:01.060 --> 00:24:03.897 But we're excited to get it in the hands of as many people as possible. 00:24:03.980 --> 00:24:07.697 It's a big step forward. I mean, this is like, and we really with Muse Spark, 00:24:07.780 --> 00:24:10.947 we were very happy with the performance that we were able to deliver as 00:24:11.030 --> 00:24:11.947 we were getting up to this point. 00:24:12.030 --> 00:24:17.000 But like, this is, the beginning of, like, MSL kind of being at the frontier. 00:24:18.080 --> 00:24:20.707 We feel like the models are quite competitive with the best 00:24:20.790 --> 00:24:24.090 models that are out there. And I think that this is— 00:24:24.630 --> 00:24:30.720 Okay, so you think it's like Astra, Fable, and Wonder altogether at the frontier? 00:24:31.220 --> 00:24:33.387 I mean, you can look at it empirically by the benchmarks. 00:24:33.470 --> 00:24:35.807 Empirically by the benchmarks, one would conclude that. 00:24:35.890 --> 00:24:36.723 Yes. I see. Yes. 00:24:37.140 --> 00:24:41.310 And I think actually one of the things that we're especially proud of is 00:24:41.770 --> 00:24:47.030 we have taken the time, as we've mentioned, to optimize models 00:24:47.190 --> 00:24:49.150 for the product form factor of Muse. 00:24:49.240 --> 00:24:51.697 So this is the state-of-the-art model. 00:24:51.780 --> 00:24:55.077 We think it really is the best model in the world for the use case 00:24:55.160 --> 00:24:56.760 of personal professional agents. 00:24:56.990 --> 00:24:59.877 Yeah, people inside companies like complaining about Goodhart's Law. 00:24:59.960 --> 00:25:03.297 You know, a metric that becomes a target ceases to become a good metric. 00:25:03.380 --> 00:25:07.297 You hear complaints that models are benchmaxed and that at a certain point, 00:25:07.380 --> 00:25:11.640 kind of overfitting to the benchmarks actually makes the models slightly worse. 00:25:12.550 --> 00:25:15.427 As people who actually have to train AI models, is that true? 00:25:15.510 --> 00:25:17.887 Are the benchmarks pretty good at grading models? 00:25:17.970 --> 00:25:20.937 Do you start to see weird edge failure cases from the— 00:25:21.020 --> 00:25:22.687 The benchmarks have to keep moving, right? 00:25:22.770 --> 00:25:26.980 Because it happens very quickly, but they just get saturated so quickly. 00:25:27.070 --> 00:25:31.280 So, I mean, I do think that— But you think non-saturated benchmarks are fine? 00:25:32.110 --> 00:25:37.660 I mean, as we've gone on, we steer the models more towards our internal 00:25:37.870 --> 00:25:40.287 benchmarks for the use cases that we're focused on. 00:25:40.370 --> 00:25:42.750 So, I mean, early on when we were kind of just 00:25:43.380 --> 00:25:45.780 doing the science of trying to climb the ladder, 00:25:46.090 --> 00:25:48.967 we cared more about what the external benchmarks were just to make 00:25:49.050 --> 00:25:50.467 sure we were on the right track. 00:25:50.550 --> 00:25:53.380 And then I'd say, you know, each release that goes by, 00:25:54.090 --> 00:25:57.850 the larger share of what we're looking at are internal ones. 00:25:58.430 --> 00:26:02.600 And one of the things that we actually quantified in our research blog post as 00:26:02.690 --> 00:26:07.147 part of Muse Wonder is this metric for big model smell, 00:26:07.230 --> 00:26:10.197 which is kind of, you know, big model smell and feeling of benchmaxing 00:26:10.280 --> 00:26:11.197 are kind of like inverses of one another. 00:26:11.280 --> 00:26:13.567 This idea that you could just like tell when you're using a really 00:26:13.650 --> 00:26:15.157 big model, it just... You can tell. 00:26:15.240 --> 00:26:16.947 It just has nuance. It's got that je ne sais quoi. 00:26:17.030 --> 00:26:20.367 Yeah. And so we, the way we quantify that is actually almost explicitly what you're 00:26:20.450 --> 00:26:23.497 mentioning, which is the gap between in-distribution performance 00:26:23.580 --> 00:26:26.087 and out-of-distribution performance. So what is the... 00:26:26.170 --> 00:26:29.710 We kind of measure how well does the model perform on tasks 00:26:29.800 --> 00:26:33.670 that are in distribution of our RL and post-training versus tasks 00:26:33.760 --> 00:26:37.210 that are out of distribution. And what does that gap look like for... 00:26:37.850 --> 00:26:40.887 What does that gap look like between the Muse Spark line and the Muse Wonder line? 00:26:40.970 --> 00:26:44.477 And we actually see this huge step up from the Muse Spark to the Muse Wonder 00:26:44.560 --> 00:26:47.727 line because of, as Mark mentioned, like, it's just a much better base 00:26:47.810 --> 00:26:52.277 model powering this new line. And there's an entire je ne sais quoi to that, 00:26:52.360 --> 00:26:55.610 but I think we've been able to actually quantify this effect. 00:26:56.030 --> 00:26:59.830 And it's an obviously dimension that we're excited to continue improving on. 00:27:00.200 --> 00:27:03.867 But if you're saying that external benchmarks have ceased to become 00:27:03.950 --> 00:27:07.350 as important, and you care much more about your internal benchmarks. 00:27:07.460 --> 00:27:10.860 Is that kind of an argument against benchmaxing, that you think that 00:27:11.420 --> 00:27:14.087 just trying to get the top score on the external benchmarks that everyone 00:27:14.170 --> 00:27:17.570 talks about is maybe not going to lead you to develop a great model? 00:27:17.840 --> 00:27:20.540 Well, yeah, but also we're focused on specific things. 00:27:20.680 --> 00:27:24.647 I mean, part of the whole strategy is personal superintelligence and just trying 00:27:24.730 --> 00:27:27.767 to build different types of products than other people are trying to build. 00:27:27.850 --> 00:27:30.980 So even if we did have the exact same evals as everyone else. 00:27:31.360 --> 00:27:32.193 It just— 00:27:32.820 --> 00:27:35.697 Yes. And though I would say a personal superintelligence is not like a super 00:27:35.780 --> 00:27:38.570 niche interest, like it's probably something that other 00:27:38.660 --> 00:27:40.120 models want to be able to do. 00:27:40.990 --> 00:27:44.037 I think that there are things that are very different about it. 00:27:44.120 --> 00:27:47.120 I mean, I think like the personality probably matters more. 00:27:47.250 --> 00:27:51.207 I mean, one of the things that we talk about is this idea of discretion, right? 00:27:51.290 --> 00:27:54.240 So if you want to like use the model to get something done, 00:27:55.050 --> 00:27:59.010 It needs to be able to know a lot about you and get things done in the world 00:27:59.510 --> 00:28:01.887 while disclosing as little as possible. 00:28:01.970 --> 00:28:04.847 So, you know, if you're— you have an allergy or you're 00:28:04.930 --> 00:28:06.847 pregnant or something, you're trying to, like, get a, 00:28:06.930 --> 00:28:09.517 you know, like, a table at a restaurant and you don't want to, like, 00:28:09.600 --> 00:28:11.937 have to tell them your medical condition. Yes, yeah. 00:28:12.020 --> 00:28:14.897 You know, to make sure that it's going to be a good place for you. 00:28:14.980 --> 00:28:16.157 Like, okay, how do you handle that? 00:28:16.240 --> 00:28:20.320 And I mean, humans have common sense and the models so far 00:28:20.740 --> 00:28:24.747 If you're building a coding agent that works in an environment where everyone has 00:28:24.830 --> 00:28:28.577 access to the same context, discretion isn't particularly important. 00:28:28.660 --> 00:28:32.920 So there are all these things that I think are just kind of more important for us. 00:28:33.290 --> 00:28:34.123 Yes. 00:28:34.550 --> 00:28:37.467 Aspects of coding where the code needs to work, 00:28:37.550 --> 00:28:41.637 but you don't— the consumers who are using it may not even ever see the code. 00:28:41.720 --> 00:28:45.387 I mean, that probably matters more in a personal superintelligence context 00:28:45.470 --> 00:28:47.517 rather than serving software developers. 00:28:47.600 --> 00:28:50.060 So I think these things are just a bit different. 00:28:50.480 --> 00:28:52.857 Yeah. Ultimately, we care about making useful models. 00:28:52.940 --> 00:28:56.070 So we want models when you put them into people's hands, 00:28:56.440 --> 00:28:58.400 they have a great experience using. 00:28:58.650 --> 00:29:02.657 And I think that is the most important metric for any model that we 00:29:02.740 --> 00:29:03.940 or other people produce. 00:29:04.120 --> 00:29:08.910 I think that really focusing on the external benchmarks, 00:29:09.410 --> 00:29:12.560 if you really dig into the supply chain a little bit, it's some 00:29:12.880 --> 00:29:17.800 PhD student who made these benchmarks, a benchmark as part of their, 00:29:18.460 --> 00:29:20.137 as the course of them doing their PhD. 00:29:20.220 --> 00:29:24.427 And so you're kind of outsourcing all of your product thinking to that PhD 00:29:24.510 --> 00:29:26.427 student who happened to make a viral benchmark. 00:29:26.510 --> 00:29:28.060 Right. So clearly that's wrong. 00:29:28.180 --> 00:29:30.080 Yeah. And so that's why we care a lot. 00:29:30.350 --> 00:29:34.060 A huge part of what a lot of people at Meta and MSL are focused 00:29:34.150 --> 00:29:38.690 on is crafting amazing evals. And now that we have them, we can work on our own. 00:29:38.820 --> 00:29:41.900 But if you're building a you're ramping up your lab, right? 00:29:41.990 --> 00:29:45.447 It's like those public ones are not a crazy place to start, but it's 00:29:45.530 --> 00:29:46.617 just not where you want to end up. 00:29:46.700 --> 00:29:48.827 You're just at that point, you're just looking for signs of life. 00:29:48.910 --> 00:29:52.810 But then when you're like really in this mode, as we are now, where it's like, 00:29:53.080 --> 00:29:55.627 you know, we want to be very intentional about how 00:29:55.710 --> 00:29:59.797 we're improving our model over time to build better and better, better products. 00:29:59.880 --> 00:30:03.047 Like a lot of the cases that Mark just mentioned, like these are 00:30:03.130 --> 00:30:07.470 actually detailed very rigorously in like the spreadsheet of all the little 00:30:07.560 --> 00:30:11.807 capabilities and nuances of the product that we think really 00:30:11.890 --> 00:30:13.100 matter to the end user. 00:30:13.190 --> 00:30:16.437 And maybe each of them matters like 2% to the user, but in aggregate, 00:30:16.520 --> 00:30:21.070 that's the obsession that will ultimately result in these models that feel magical. 00:30:21.940 --> 00:30:24.737 Can you guys describe for me just what goes into training an RL model? 00:30:24.820 --> 00:30:27.080 Or sorry, a leading-edge frontier model? 00:30:27.330 --> 00:30:30.367 I'm just curious in that I think a lot of people listening 00:30:30.450 --> 00:30:34.037 to this have a rough idea where there's there's a pre-training 00:30:34.120 --> 00:30:34.917 and a post-training step. 00:30:35.000 --> 00:30:38.150 So pre-training is you kind of ingest this huge amount of data. 00:30:38.250 --> 00:30:40.377 You look at a lot of pictures and these are cats and these are dogs. 00:30:40.460 --> 00:30:44.507 And after a while you start to develop a sense for what is catness and what is 00:30:44.590 --> 00:30:47.300 dogness and what is fur and everything like that. 00:30:47.720 --> 00:30:50.477 And so that's kind of the pre-training step where you just hoover up data 00:30:50.560 --> 00:30:52.687 and maybe you can talk about the data acquisition and just kind 00:30:52.770 --> 00:30:53.930 of what goes into that. 00:30:54.100 --> 00:31:00.940 And then there's this RL stage, which is the newish advance 00:31:01.030 --> 00:31:02.930 over kind of the past few years of AI. 00:31:03.320 --> 00:31:07.737 And again, it sounds like that is where a lot of the specific nuance of this model 00:31:07.820 --> 00:31:12.910 has come in, where you guys have chosen better domains, 00:31:15.120 --> 00:31:17.960 better hills to climb for the model. 00:31:18.330 --> 00:31:21.750 But just at a high level, describe how many people, 00:31:21.840 --> 00:31:25.240 how long it takes, what the steps are, how you think about those in— 00:31:26.300 --> 00:31:27.887 How much time do you have? Yeah, at a high level. 00:31:27.970 --> 00:31:32.057 This happens to be What we spend all of our time on. 00:31:32.140 --> 00:31:35.680 Exactly. And so summarizing— YouTube courses. 00:31:36.270 --> 00:31:39.977 Yeah, I think at a very high level, these are the stages. 00:31:40.060 --> 00:31:41.860 It's pre-training and post-training. 00:31:42.530 --> 00:31:46.400 Ironically, training is the superset of all of it. 00:31:46.780 --> 00:31:49.487 But that's— so how many people are on this whole effort, like when you 00:31:49.570 --> 00:31:50.247 develop these models? 00:31:50.330 --> 00:31:54.330 So this, I think you're getting into what is interesting about MSL 00:31:54.830 --> 00:31:58.627 and the approach that we've taken, which is, I mean, From the very beginning, 00:31:58.710 --> 00:32:01.297 I think when I first reached out to you, I think we were just looking at this 00:32:01.380 --> 00:32:04.297 because we were like, just like, wow, okay, how far we've come. 00:32:04.380 --> 00:32:06.177 And I was just like, I pulled up the screenshot of like 00:32:06.260 --> 00:32:07.927 when I first reached out to Alex. 00:32:08.010 --> 00:32:10.767 My first thing was like, hey, do you have time to catch up and talk? 00:32:10.850 --> 00:32:12.057 And then it's like, all right, we talked for— 00:32:12.140 --> 00:32:14.477 That's how they get you. 44 minutes or whatever. 00:32:14.560 --> 00:32:19.350 And then after that call, 2 days later, Alex sends me this long thing, 00:32:19.440 --> 00:32:23.940 which is like, well, here's what I think I would do if I were in your position. 00:32:24.530 --> 00:32:29.447 And the very first thing is that you don't need a lot of people, and 00:32:29.530 --> 00:32:31.327 talent density is the thing that matters. 00:32:31.410 --> 00:32:34.990 So I'd say, like, from the beginning of the plan, this was, 00:32:35.080 --> 00:32:37.287 like, the core of your strategy, right? 00:32:37.370 --> 00:32:39.620 It's basically, like, you actually don't need 00:32:40.120 --> 00:32:44.210 a massive number of people, but what you do need is for, 00:32:44.880 --> 00:32:48.830 like, each of the people to be as good as possible, which kind of informed this 00:32:49.130 --> 00:32:51.380 kind of crazy recruiting push and, like, very 00:32:52.720 --> 00:32:56.227 intense filtering that we had to take, you know, some of the best people 00:32:56.310 --> 00:33:00.770 from inside Meta and a very intense process to get people from outside Meta. 00:33:01.350 --> 00:33:05.020 But I actually think like a lot of what has made MSL special 00:33:05.480 --> 00:33:09.317 and why a lot of people have wanted to join is because I think it's in some 00:33:09.400 --> 00:33:12.537 ways it's just like more fun to work on a talent and smaller effort. 00:33:12.620 --> 00:33:14.077 It feels like the startup. Yes. 00:33:14.160 --> 00:33:15.010 That's operating. 00:33:16.330 --> 00:33:18.950 In some ways it has all the benefits of a startup 00:33:19.040 --> 00:33:21.690 with the stability and resources of a larger company. 00:33:21.830 --> 00:33:25.417 And this was like this environment that I think this was like point one in your kind 00:33:25.500 --> 00:33:29.550 of plan for what, um, for what Alex was like, this is what we need to go do. 00:33:29.760 --> 00:33:33.007 So I don't know the answer of like how big is a pre-training team, you know, 00:33:33.090 --> 00:33:34.677 it's, it doesn't need to be that big. 00:33:34.760 --> 00:33:38.470 But I think if you look at, you know, some of the other labs, um, it's 00:33:38.560 --> 00:33:39.937 probably quite big in some other places. 00:33:40.020 --> 00:33:44.437 So, um, I think part of the reason why, uh, these models are doing well is because 00:33:44.520 --> 00:33:48.777 you You have people who we're betting on and they have a lot of resources 00:33:48.860 --> 00:33:50.487 because we're taking our compute. 00:33:50.570 --> 00:33:53.657 We're not peanut buttering it across a lot of competing efforts. 00:33:53.740 --> 00:33:59.450 We're basically just investing in the key people, a lot of compute per researcher, 00:33:59.870 --> 00:34:01.750 and then they can do their work. 00:34:02.250 --> 00:34:04.750 Was Wonder the one big training effort? 00:34:04.960 --> 00:34:06.627 That was where most of the training was? 00:34:06.710 --> 00:34:09.217 It was the next pre-train. It was climbing up this ladder. 00:34:09.300 --> 00:34:11.967 So, I mean, we have this tiering system and then kind of Spark. 00:34:12.050 --> 00:34:14.507 I mean, internally, like, Avocado was the code name 00:34:14.590 --> 00:34:18.470 for the Muse Spark line, and then Watermelon was the code name for this one. 00:34:18.640 --> 00:34:20.437 And we're, like, out of fruits now. We're out of fruits. 00:34:20.520 --> 00:34:24.390 Yeah, so we're going to have to switch to a, to a new convention. 00:34:24.730 --> 00:34:28.440 But, um, I think the only fruit that is technically larger is the durian, 00:34:28.860 --> 00:34:30.357 and you don't want to go to the durian. No, no, no, no. 00:34:30.440 --> 00:34:31.987 Yeah, yeah, yeah. No, you want to stay far away. 00:34:32.070 --> 00:34:34.320 A durian is bigger than a watermelon? 00:34:34.450 --> 00:34:36.117 I think it's definitely heavier. That's horrifying. 00:34:36.200 --> 00:34:38.700 Definitely heavier. Spikier. Spikier, yeah. 00:34:39.370 --> 00:34:44.410 But no, yeah, I think as Mark mentioned, like, you know, each of the— I mean, 00:34:44.500 --> 00:34:48.417 each of the stages I'll describe, like, it's tens of people working on it 00:34:48.500 --> 00:34:50.590 at Meta versus, you know, hundreds at— 00:34:50.710 --> 00:34:54.137 So Meta is a huge company, but this is like a really tiny effort inside of Meta. 00:34:54.220 --> 00:34:56.387 Yeah. Oh yeah, I know. The research team is less than 200 people. 00:34:56.470 --> 00:35:00.887 Yeah, it's very small. It's intentionally very, very tight. I think that the— 00:35:00.970 --> 00:35:04.520 It's closer to 100, right? And then there's a bunch of InfoFolks. Yeah. 00:35:04.850 --> 00:35:06.017 It's a lot of infra work. 00:35:06.100 --> 00:35:11.270 So I think at a very high level, to go back to the, to the, to the what is 00:35:11.400 --> 00:35:13.237 it question, you know, how does it happen? 00:35:13.320 --> 00:35:15.657 How does model happen? That's basically what I'm asking. 00:35:15.740 --> 00:35:17.110 Where do models come from? 00:35:19.570 --> 00:35:24.160 You take a huge amount of data for pre-training, and here you're— 00:35:24.750 --> 00:35:28.500 And it's here, like, roughly, it's like all the, 00:35:28.960 --> 00:35:31.797 you know, public internet and like all the books you can get your hands 00:35:31.880 --> 00:35:34.177 on and just like as much data as possible. But different types. 00:35:34.260 --> 00:35:37.800 Plus the proprietary But also different modalities, right? 00:35:37.890 --> 00:35:40.267 So if you want it to be good at perception, then you want to make sure 00:35:40.350 --> 00:35:44.310 it has a lot of images and audio and video and this kind of stuff. 00:35:44.850 --> 00:35:47.100 Um, you know, if you want it to 00:35:47.770 --> 00:35:51.027 be good at certain domains, you don't just want to RL that in at the end. 00:35:51.110 --> 00:35:53.737 You want to have it have exposure to that from the beginning. 00:35:53.820 --> 00:35:57.860 Yeah. Yep. Yeah. So, so, and in this stage, you're kind of, you're much less, 00:35:58.280 --> 00:36:01.617 you care a lot less exactly what the data looks like or how it's formatted or 00:36:01.700 --> 00:36:03.550 You know, all of everything about it. 00:36:03.790 --> 00:36:05.787 But it's just lots and lots of data. 00:36:05.870 --> 00:36:08.920 When you talk about this being the best personal agent model, 00:36:09.210 --> 00:36:12.337 was there data you had to get for the pre-train where it's like huge 00:36:12.420 --> 00:36:15.420 amounts of restaurant reservation, call booking? 00:36:15.510 --> 00:36:16.717 I don't know what it is, but like— 00:36:16.800 --> 00:36:20.140 You just kind of want to dump in large volumes of the modalities 00:36:20.260 --> 00:36:21.467 that people are going to care about. 00:36:21.550 --> 00:36:22.767 I mean, there were a bunch of things. 00:36:22.850 --> 00:36:25.187 I mean, I don't actually think we talked about this when we were talking 00:36:25.270 --> 00:36:27.600 about where Muse Wonder excels. 00:36:27.890 --> 00:36:31.227 But some of the areas that it's like, all right, what do people care about, right? 00:36:31.310 --> 00:36:34.357 It's, I mean, there are things like shopping that people care about a lot, 00:36:34.440 --> 00:36:36.690 personal advice that people care about a lot. 00:36:36.900 --> 00:36:39.777 People have personal legal questions, personal health questions, 00:36:39.860 --> 00:36:44.490 personal tax questions. Muse Wonder, I mean, is like quite good at those. 00:36:44.580 --> 00:36:47.077 I don't know if there's any quantification that we want to give, 00:36:47.160 --> 00:36:51.830 but it's, let's say it's— It's very good on those personal advice questions. 00:36:51.920 --> 00:36:55.337 Yeah, and so those types of things, I mean, you can RL them to some degree, 00:36:55.420 --> 00:36:58.420 but you're like, just want to put in a large amount of data. 00:36:58.800 --> 00:37:01.257 Like, you can imagine health is a very— there's a lot of data 00:37:01.340 --> 00:37:05.347 out there about health. And, um, in, in pre-training, you just wanted to like see 00:37:05.430 --> 00:37:06.597 all these different things. 00:37:06.680 --> 00:37:11.020 And, um, so you, you kind of just put these different corpora in. 00:37:11.650 --> 00:37:17.190 Yeah. So, so that— so as, um, as Mark was mentioning, like, the, uh, we, 00:37:18.360 --> 00:37:22.487 uh, one of the highlights of Muse Wonder actually is its ability on domain 00:37:22.570 --> 00:37:24.117 various domain expertises. Yes. 00:37:24.200 --> 00:37:29.117 And I think our philosophy here is that, you know, the best personal agent should 00:37:29.200 --> 00:37:31.127 feel like a team of experts in your corner. 00:37:31.210 --> 00:37:33.960 Like, it should feel like it is your legal team. 00:37:34.170 --> 00:37:38.050 Yes. And your finance team, and your medical team, and, and, and, and, and. 00:37:38.420 --> 00:37:42.047 And so, um, on a bunch of these domains, we, you know, Muse Wonder is actually 00:37:42.130 --> 00:37:45.557 state of the art on, I think, um, a lot of the finance evals, as well 00:37:45.640 --> 00:37:48.100 as a lot of the, um, uh, legal evals. 00:37:48.640 --> 00:37:50.020 the state-of-the-art model. 00:37:50.140 --> 00:37:53.567 And that, you know, these, these domain areas, that starts in pre-training. 00:37:53.650 --> 00:37:57.690 Those, you know, we look for a lot of domain data to put into pre-training. 00:37:57.820 --> 00:38:01.360 Yes. But a lot of the, a lot of the art of pre-training— I mean, 00:38:01.650 --> 00:38:05.487 pre-training is, um, there's this one Rune tweet where it's like pre-training is full 00:38:05.570 --> 00:38:08.867 of the researchers who like chew glass and stare out into the distance 00:38:08.950 --> 00:38:10.917 and into the, into the abyss or whatnot. 00:38:11.000 --> 00:38:14.200 But, but I think there's a truth to that because it is like, um, 00:38:14.380 --> 00:38:17.960 It is both the highest stakes of the research areas because you, 00:38:18.460 --> 00:38:22.797 you know, we invested a huge percentage of our compute into training Muse Wonder, 00:38:22.880 --> 00:38:24.307 into pre-training Muse Wonder. 00:38:24.390 --> 00:38:26.930 And it was very high stakes. 00:38:27.050 --> 00:38:30.680 I think probably both of us at various points in that process, you know— 00:38:30.770 --> 00:38:32.187 You went through troughs of sadness. 00:38:32.270 --> 00:38:36.150 But yeah, troughs of uncertainties, you know, all the 7 stages of grief, 00:38:36.360 --> 00:38:38.980 you know, happen while you're pre-training a model. 00:38:39.070 --> 00:38:44.320 And it is the most sort of like fundamentally scientific in an interesting 00:38:44.410 --> 00:38:47.510 way, because, you know, you just don't have very many samples. 00:38:47.910 --> 00:38:52.330 So, like, your ability to infer exactly what will happen scientifically 00:38:52.700 --> 00:38:54.747 really depends on your ability to sort of like, 00:38:54.830 --> 00:38:56.837 you know, big brain reason about everything that's 00:38:56.920 --> 00:38:58.297 happening in this process. Okay. 00:38:58.380 --> 00:39:01.830 So in the pre-training stage, everyone who's training frontier models 00:39:02.260 --> 00:39:05.297 basically is trying to get as much data as possible for the pre-train 00:39:05.380 --> 00:39:09.637 with maybe a bent towards If you want Muse Wonder to excel at legal 00:39:09.720 --> 00:39:12.097 and finance and tax questions and things like this, 00:39:12.180 --> 00:39:15.940 then you want to have as much of that specific corpus as 00:39:16.060 --> 00:39:17.277 possible in the pre-train. 00:39:17.360 --> 00:39:20.067 Yeah, you still want as much intentionality about where ultimately you 00:39:20.150 --> 00:39:23.277 want your model to be, but again, it just, you know, benefits from a lot more data. 00:39:23.360 --> 00:39:28.660 Yeah. And this is sort of the massive data, massive compute stage. 00:39:29.030 --> 00:39:32.577 Do you hit some saturation in the data you put in the pre-train, or ultimately 00:39:32.660 --> 00:39:34.290 is it just more data is better? 00:39:35.290 --> 00:39:37.707 Well, there's a relationship between if you want 00:39:37.790 --> 00:39:39.940 the model to be bigger, you need more data. 00:39:40.340 --> 00:39:44.257 Otherwise, it overfits. Yes, but my point is, like, is there data where it's like, 00:39:44.340 --> 00:39:47.347 we can't distract the model with that, we can't clutter up its brain attic 00:39:47.430 --> 00:39:50.430 with knowing that stuff, or is it just more data is better? 00:39:50.800 --> 00:39:52.970 There are some of these effects, I think. 00:39:53.060 --> 00:39:54.727 Yeah, because this is part of the science. 00:39:54.810 --> 00:39:56.147 Okay, we're getting into the secret sauce, sorry. 00:39:56.230 --> 00:39:58.817 No, it's not even that, it's just like, I mean, I think that there are all these 00:39:58.900 --> 00:40:01.107 questions about, like, okay, well, we put a lot of these tokens in, 00:40:01.190 --> 00:40:03.277 is there any zero-sumness in a bunch of places there isn't, 00:40:03.360 --> 00:40:08.407 or it's actually positive transfer, but then like depending on how it's set 00:40:08.490 --> 00:40:10.617 up, like you can have some negative transfer. 00:40:10.700 --> 00:40:12.997 Yes. And so I don't know, there are all these interesting questions. 00:40:13.080 --> 00:40:18.250 But in general, one of the bigger challenges of scaling bigger models 00:40:18.790 --> 00:40:21.740 is it's not just about how many flops and how much compute. 00:40:22.290 --> 00:40:24.750 You want to be able to scale data with that. 00:40:24.840 --> 00:40:28.590 I mean, this is like part of what, when the original kind of scaling laws 00:40:28.680 --> 00:40:31.557 for pre-training were written, I mean, this is sort of what came 00:40:31.640 --> 00:40:35.560 out of that is is that. So if the internet isn't growing that quickly 00:40:36.310 --> 00:40:39.307 and you want to scale bigger models that can be more intelligent, 00:40:39.390 --> 00:40:41.937 I think one of the interesting problems is, all right, where's 00:40:42.020 --> 00:40:42.987 the data coming from? Yes. 00:40:43.070 --> 00:40:48.110 So I think that probably gets to the end of what we're going to talk about on that. 00:40:48.280 --> 00:40:54.700 But we could go to the other— Excellent CEO energy over here. 00:40:55.200 --> 00:41:00.287 But so post-training, so then I think Post-training is kind of— is 00:41:00.370 --> 00:41:04.040 a very exciting field now, I think, and is an area where 00:41:05.250 --> 00:41:08.047 I think actually, I think almost on a continual basis, 00:41:08.130 --> 00:41:10.927 the entire field is learning about the optimal recipes to be able 00:41:11.010 --> 00:41:12.550 to post-train these models. 00:41:12.890 --> 00:41:16.307 And again, for people, like, post-training is where you give the model 00:41:16.390 --> 00:41:19.187 a task that you expect it to do, grade it on how it does. 00:41:19.270 --> 00:41:21.977 And so if it's a chess-playing model, it plays chess games, learns the— Well, 00:41:22.060 --> 00:41:25.647 there's actually 2 stages. There's the SFT, So there's supervised learning. 00:41:25.730 --> 00:41:30.400 So that's like a lot of when you take this kind of raw thing and you give it both 00:41:31.450 --> 00:41:33.697 the question and the answer that you would want it to have, right? 00:41:33.780 --> 00:41:35.697 If like, if you're having a chat conversation, how would you 00:41:35.780 --> 00:41:37.037 want it to respond in this case? 00:41:37.120 --> 00:41:41.120 So when you're training things like personality, the SFT is actually like 00:41:41.210 --> 00:41:43.210 a very important part for molding it. 00:41:43.670 --> 00:41:47.370 But then I think most of the advances in the recent years have been in RL. 00:41:47.760 --> 00:41:48.593 Yeah. Yeah. 00:41:48.710 --> 00:41:52.630 So I think one mental model that can be helpful is like, And SFT is where 00:41:52.720 --> 00:41:55.217 you kind of initialize a lot of— SFT is supervised fine-tuning. 00:41:55.300 --> 00:41:56.517 Supervised fine-tuning, yeah. 00:41:56.600 --> 00:41:59.557 Where you initialize a lot of the behaviors that you want the model 00:41:59.640 --> 00:42:01.390 to be able to exhibit later on. 00:42:01.730 --> 00:42:04.650 And then, you know, maybe at the end of SFT, 00:42:05.110 --> 00:42:08.697 it has those behaviors kind of baked in, but it'll only do it like 10% of the time. 00:42:08.780 --> 00:42:13.860 And then in reinforcement learning, you really sort of give the model 00:42:14.320 --> 00:42:18.490 reinforcement in various situations to bring the elicitation of those 00:42:18.580 --> 00:42:19.880 behaviors dramatically up. 00:42:21.120 --> 00:42:24.580 And I think that the— yeah, so I think there's a bunch 00:42:24.670 --> 00:42:28.917 of interesting things that are happening in RL and post-training. 00:42:29.000 --> 00:42:33.130 I think one is that just the scaling of RL and post-training has been 00:42:34.470 --> 00:42:37.887 one of the clear axes, just spending more compute and gathering more data. 00:42:37.970 --> 00:42:40.770 Just doing much more of it before you release a model. 00:42:40.970 --> 00:42:44.767 Yes, doing much more of it, especially as the pre-trains scale up, doing much more. 00:42:44.850 --> 00:42:47.987 It's like anything in scaling laws, like it has to go with the rest 00:42:48.070 --> 00:42:50.860 of the factors or it doesn't work. Yeah, in general. 00:42:51.280 --> 00:42:53.947 I mean, if I were to give, say, like one broad statement about AI, 00:42:54.030 --> 00:42:55.777 it's just scale everything, you know? 00:42:55.860 --> 00:42:58.907 But, um, uh, but in particular, you have to scale everything. 00:42:58.990 --> 00:43:01.660 Like scaling one thing in isolation does not work. 00:43:02.250 --> 00:43:05.960 You, you technically can, you can scale one dimension. 00:43:06.420 --> 00:43:08.627 You just will get much more optimal performance if 00:43:08.710 --> 00:43:10.167 you scale many things altogether. 00:43:10.250 --> 00:43:13.177 Well, I mean, so one way to think about like Muse Wonder, right? 00:43:13.260 --> 00:43:14.637 It's like, all right, so we did the pre-train. 00:43:14.720 --> 00:43:16.757 Yep. we've done some RL, right? 00:43:16.840 --> 00:43:18.177 When you look at it like Muse Spark, right? 00:43:18.260 --> 00:43:22.980 So version 1 to 1.1, 1.2, 1.3, we didn't change anything about pre-training, right? 00:43:23.220 --> 00:43:25.640 We just continued doing more RL on top of it. 00:43:25.810 --> 00:43:29.770 So, um, so you, when you think about the early versions 00:43:30.020 --> 00:43:33.900 of a model of a new class, they're actually relatively less, 00:43:34.360 --> 00:43:37.660 like the RL to pre-training ratio is going to be lower for RL 00:43:38.320 --> 00:43:42.207 than the later versions. That is kind of what the point releases in models are. 00:43:42.290 --> 00:43:44.417 Well, I think every company has its own convention. 00:43:44.500 --> 00:43:45.577 For us, that's what it is meant to be. 00:43:45.660 --> 00:43:49.167 But for a lot of companies, like the big number is the pre-train and then the— 00:43:49.250 --> 00:43:51.450 I think it kind of lost that semantic value. 00:43:53.090 --> 00:43:54.217 Yeah, I think now you actually can't do that. 00:43:54.300 --> 00:43:56.047 Okay, but that is true at Meta at least. 00:43:56.130 --> 00:43:57.217 But we haven't even done it. 00:43:57.300 --> 00:43:59.677 We kind of decided to not even do it. It's true at Muse Spark at least. 00:43:59.760 --> 00:44:01.347 It is true at Muse Spark and Muse Wonder. 00:44:01.430 --> 00:44:03.427 It's true that Muse Spark and Muse Wonder are different pre-trains. 00:44:03.510 --> 00:44:06.647 And we decided to kick the can down the road on this question for Muse Wonder 00:44:06.730 --> 00:44:08.600 by not giving it a number yet. 00:44:08.770 --> 00:44:12.170 But when we have our first dot release, we will have to decide that. 00:44:13.900 --> 00:44:17.280 Yeah. But yeah, no, I think the RL stage, part of what has been— 00:44:18.610 --> 00:44:23.117 part of what's very interesting is first is just producing the environments 00:44:23.200 --> 00:44:27.120 and producing the tasks. And I think that is one of the areas where there's 00:44:27.210 --> 00:44:29.337 a lot of intentionality in training these models. 00:44:29.420 --> 00:44:33.000 It's also one of the areas where you notice very funny things. 00:44:33.090 --> 00:44:38.800 Like if you— in a lot of cases, less is actually more when it comes to data in RL. 00:44:39.390 --> 00:44:44.180 And so I think I think the sort of, like, the feel and the science of it, like, 00:44:44.760 --> 00:44:48.477 is pretty flipped, generally speaking, from pre-training. 00:44:48.560 --> 00:44:54.187 Yeah. The other is that I think it can give very magnified kind of instruction 00:44:54.270 --> 00:44:55.147 to the model, if that makes sense. 00:44:55.230 --> 00:44:58.527 So I think that, like, a lot of safety cases or alignment cases 00:44:58.610 --> 00:45:01.027 that we care a lot about are, like, you know, and, you know, 00:45:01.110 --> 00:45:05.010 where you would see when you do large high-compute RL runs on frontier models, 00:45:05.280 --> 00:45:11.037 are where you apply a lot of RL or optimization pressure in scenarios where 00:45:11.120 --> 00:45:15.250 the only way for the model to succeed is to do something misaligned 00:45:15.380 --> 00:45:16.530 or do something unsafe. 00:45:16.760 --> 00:45:19.847 And so that's another thing that we had to be very careful about and thoughtful 00:45:19.930 --> 00:45:24.030 about in training Muse Wonder is ensuring that we curated the RL environments very 00:45:24.680 --> 00:45:28.347 thoughtfully to ensure that all of them were eliciting and driving 00:45:28.430 --> 00:45:30.557 towards more aligned behavior from the model. 00:45:30.640 --> 00:45:32.607 So the point that you're saying about quality, I think, 00:45:32.690 --> 00:45:36.147 is an important one because I think a lot of the previous questions back and forth 00:45:36.230 --> 00:45:38.567 were about the quantity and have the flavor of like, oh, 00:45:38.650 --> 00:45:40.067 you need to scale the quantity. Yes. 00:45:40.150 --> 00:45:43.150 But I mean, I do think like a huge part of this is just, is, 00:45:43.530 --> 00:45:47.660 is the point that you just made of like the even a smaller amount of high quality. 00:45:48.240 --> 00:45:52.080 Yeah, I think, I think in RL, like we, you know, as we've seen 00:45:52.670 --> 00:45:57.710 pretty big jumps in our new Spark line, I think these have been like accompanied 00:45:57.800 --> 00:46:02.300 with, you know, real meaningful improvements to the underlying recipe. 00:46:02.510 --> 00:46:05.510 And each of these improvements in the recipe have been like, 00:46:06.350 --> 00:46:11.477 I think the improvement between Muse Spark 1 to Muse Spark 1.3 feels night and day. 00:46:11.560 --> 00:46:14.900 I think probably to most people, if you ask them, 00:46:16.690 --> 00:46:18.197 do these feel like they came from different pretrains? 00:46:18.280 --> 00:46:21.617 I think they probably would say yes, because I think you can— Mm-hmm. 00:46:21.700 --> 00:46:24.197 I think it's happening industry-wide for what it's worth. 00:46:24.280 --> 00:46:29.700 It's just like we are learning to squeeze so much capability and so much 00:46:30.500 --> 00:46:35.210 and elicit so much agentic capability from these existing base models. 00:46:35.330 --> 00:46:38.837 Now, that's not to say, like, we believe scaling is very important. 00:46:38.920 --> 00:46:41.137 That's why we have to use Wonderland in the first place. 00:46:41.220 --> 00:46:44.557 Yes. But, you know, this sort of like, these two things are happening at the same 00:46:44.640 --> 00:46:48.847 time, which is also, by the way, for us, this is why we're seeing so 00:46:48.930 --> 00:46:49.977 much progress in general. 00:46:50.060 --> 00:46:53.517 Like, I think it's the fact that we're scaling both pre-training and scaling our 00:46:53.600 --> 00:46:56.860 recipes in RL kind of like concurrently. At a nice clip. 00:46:57.360 --> 00:46:59.737 Yeah. And then there's these other scaling axes too. 00:46:59.820 --> 00:47:03.027 I mean, like making sure that your agent harness is good and that you can 00:47:03.110 --> 00:47:06.200 co-optimize these things is probably just another aspect of, 00:47:06.660 --> 00:47:08.120 or dimension for scaling. 00:47:08.740 --> 00:47:12.117 But if I'm understanding you guys correctly, are you saying that alignment 00:47:12.200 --> 00:47:16.250 is not about coming up with some alignment-specific RL 00:47:16.380 --> 00:47:20.380 steps and instead about making sure the entire RL process 00:47:20.960 --> 00:47:23.380 is kind of steering The model— 00:47:23.970 --> 00:47:26.467 You have to be thoughtful about it in the entire process of RL. 00:47:26.550 --> 00:47:27.347 Yeah. 00:47:27.430 --> 00:47:30.057 So I think this is a holistic effort, I think. 00:47:30.140 --> 00:47:33.540 Yeah. But also, I think different labs have very different theories. 00:47:33.930 --> 00:47:36.767 I mean, alignment is a very philosophical concept, right? 00:47:36.850 --> 00:47:38.727 Of like, who do you want it to be aligned to? 00:47:38.810 --> 00:47:42.980 I mean, a lot of our theory is like, okay, we care more about it being aligned to, 00:47:43.070 --> 00:47:46.197 you know, we're trying to give people agents, right? 00:47:46.280 --> 00:47:48.630 And it needs to, if you ask it to do something, 00:47:49.030 --> 00:47:52.957 you need to have a high confidence that it's going to do what you ask it to do. 00:47:53.040 --> 00:47:55.827 Yes. You know, like, you want it to do that with high reliability. 00:47:55.910 --> 00:47:58.920 You don't want a moralizing lecture for why you're wrong 00:47:59.130 --> 00:48:00.457 for asking it to do the thing. 00:48:00.540 --> 00:48:02.877 Obviously, if you're trying to do something that's catastrophic, 00:48:02.960 --> 00:48:06.760 it needs to gracefully, without pissing you off, like, not do it. 00:48:08.090 --> 00:48:10.640 But in general, you know, we view alignment as more 00:48:11.180 --> 00:48:15.890 of a question of instruction and values alignment with the person 00:48:15.980 --> 00:48:17.380 who it's intending to serve. 00:48:18.980 --> 00:48:22.980 And I think that that's kind of a different mode of thinking about it. 00:48:23.070 --> 00:48:25.947 It's very different from, I think, how some of the others do. 00:48:26.030 --> 00:48:30.907 But one of the more interesting things I think that is maybe a question related 00:48:30.990 --> 00:48:34.580 to your original question is also just how long the process of training 00:48:34.740 --> 00:48:35.577 one of these models takes. 00:48:35.660 --> 00:48:41.330 Like Muse Wonder, I mean, in general, we've planned Muse Wonder since 00:48:42.170 --> 00:48:43.800 last You know, August. 00:48:44.380 --> 00:48:48.297 Yeah, so yeah, I remember the offsite when we were— so this is like when 00:48:48.380 --> 00:48:49.387 they first presented this. 00:48:49.470 --> 00:48:54.097 Yeah, so I think all of the stages that we just talked about are like, you know, you 00:48:54.180 --> 00:48:55.887 really work on them for quite a long time. 00:48:55.970 --> 00:48:59.767 And, and, um, it's not that we, we didn't start working on Muse Wonder 00:48:59.850 --> 00:49:01.187 when we finished Muse Spark. 00:49:01.270 --> 00:49:03.937 It's that we've actually just been planning and working on this 00:49:04.020 --> 00:49:05.650 for quite a long time. 00:49:06.150 --> 00:49:09.447 And we really view Muse Spark, if anything, as like I think we mentioned 00:49:09.530 --> 00:49:12.130 this in the first blog post where we wrote about it. 00:49:12.240 --> 00:49:15.580 That was a stepping stone to get to Muse Wonder. 00:49:16.370 --> 00:49:17.367 And that really was how we thought about it. 00:49:17.450 --> 00:49:18.667 Even at the time you thought about it. 00:49:18.750 --> 00:49:22.207 Yeah, and in terms of how long it takes, I mean, we're also even just 00:49:22.290 --> 00:49:23.507 talking about the research part. 00:49:23.590 --> 00:49:27.840 I mean, there's also the compute part where we've been investing in building up, 00:49:28.090 --> 00:49:29.757 first of all, a huge amount of compute overall. 00:49:29.840 --> 00:49:32.940 But a lot of, you know, part of our philosophy on this is that 00:49:33.720 --> 00:49:38.597 you can kind of measure the compute strength of a lab by how big 00:49:38.680 --> 00:49:43.730 its largest single computer is, supercomputer, for being able to do these 00:49:44.360 --> 00:49:45.067 large training runs. 00:49:45.150 --> 00:49:48.780 So, I mean, we built this Prometheus cluster, we call it, in Ohio. 00:49:49.030 --> 00:49:52.610 Mm-hmm. That's scaling to larger than a gigawatt. 00:49:53.110 --> 00:49:56.247 And now we're building this Hyperion cluster in Louisiana that's 00:49:56.330 --> 00:49:57.910 going to scale to 5 gigawatts. 00:49:58.870 --> 00:50:03.540 And those actually, predate MSL. 00:50:05.000 --> 00:50:08.550 Because like building a whole like data center site from scratch, 00:50:09.210 --> 00:50:11.387 at least the plans for those, they're coming online now. 00:50:11.470 --> 00:50:15.137 But that was— You guys thought it might be the feed, right? 00:50:15.220 --> 00:50:17.477 That you just have used for a lot of compute generally? 00:50:17.560 --> 00:50:18.393 No, no, no. 00:50:18.850 --> 00:50:23.100 This is kind of predating, even going back to the, to kind of the LLaMA program. 00:50:23.190 --> 00:50:27.440 I see. I mean, it was probably in 2023. 00:50:29.480 --> 00:50:35.407 or yeah, either 2023, 2024 was when we basically started making 00:50:35.490 --> 00:50:36.697 the plans to build those clusters. 00:50:36.780 --> 00:50:39.490 They're now kind of fully coming online. 00:50:39.790 --> 00:50:43.370 Well, I guess the Louisiana one will come online over a couple years. 00:50:44.000 --> 00:50:44.833 Yeah. 00:50:45.620 --> 00:50:49.547 But if you want to go really end-to-end on undoing this stuff, then, I mean, 00:50:49.630 --> 00:50:54.050 obviously the kind of full stack, and I mean, Meta is a full-stack tech company. 00:50:54.180 --> 00:50:58.427 I mean, we build our own data centers, we build our own chips, 00:50:58.510 --> 00:51:00.177 we build our own low-level infrastructure. 00:51:00.260 --> 00:51:03.017 And now this is kind of the next component, which, I mean, 00:51:03.100 --> 00:51:06.147 arguably is probably the most important component in all of this for the next wave 00:51:06.230 --> 00:51:09.607 of technology is kind of building really good models and then 00:51:09.690 --> 00:51:10.567 delivering them to people. 00:51:10.650 --> 00:51:13.700 But I mean, yeah, if you want to go fully end-to-end, I mean, 00:51:13.860 --> 00:51:16.530 this is a long process. 00:51:16.860 --> 00:51:20.950 I do think this is like a— I mean, maybe it's well appreciated by folks 00:51:21.040 --> 00:51:24.997 in the tech industry, but It really is, you know, it feels like models 00:51:25.080 --> 00:51:25.877 are coming out constantly. 00:51:25.960 --> 00:51:28.757 So maybe you'd have the mental model that like, oh, a company like launches 00:51:28.840 --> 00:51:31.627 a model and then they figure out how they're going to make the next model and, 00:51:31.710 --> 00:51:32.337 or something like that. 00:51:32.420 --> 00:51:35.800 But it's actually these like very long planning processes and these 00:51:36.220 --> 00:51:39.967 sort of like very long work streams that kind of all come together at the very 00:51:40.050 --> 00:51:41.977 end and then result in these beautiful models. 00:51:42.060 --> 00:51:46.850 Yeah. How do you build good products when the feedback cycle is so long? 00:51:47.060 --> 00:51:49.227 Like ultimately you start planning the model in August and you don't 00:51:49.310 --> 00:51:51.710 know until more than a year later. if it's good. 00:51:51.940 --> 00:51:55.567 And, you know, Silicon Valley famously likes to run on metrics, but, like, 00:51:55.650 --> 00:51:58.570 ultimately, do you just need someone who 00:51:59.570 --> 00:52:02.327 is the aesthete and says, I'm sorry, I don't like it? 00:52:02.410 --> 00:52:04.367 It's nice when you have metrics, but I think the, like, 00:52:04.450 --> 00:52:08.667 state of being an entrepreneur is, like, how well do you make decisions 00:52:08.750 --> 00:52:11.290 in an information-poor environment, right? So— 00:52:11.630 --> 00:52:13.257 But this is, like, that on steroids, right? 00:52:13.340 --> 00:52:14.337 That on mega steroids. 00:52:14.420 --> 00:52:17.590 Yeah, I think that this is— and I think Marc gets a huge amount 00:52:17.680 --> 00:52:21.637 of credit for this, like, Because of how long everything takes, 00:52:21.720 --> 00:52:27.097 like, there were many points along over the past year since I've been here, 00:52:27.180 --> 00:52:30.477 but also, you know, as you mentioned, like, long before that, where 00:52:30.560 --> 00:52:34.230 you have to make very large investments or make very big 00:52:34.320 --> 00:52:38.527 decisions about, you know, how you're going to invest or develop 00:52:38.610 --> 00:52:41.810 the technology or, you know, what decisions you're going to make 00:52:41.950 --> 00:52:47.247 long before there's proof points around whether or not even the thing is working. 00:52:47.330 --> 00:52:51.040 And so I think this is like, in some sense, it's like 00:52:51.130 --> 00:52:52.207 entrepreneurship on hard mode. 00:52:52.290 --> 00:52:53.587 It's like the opposite of Lean Startup. 00:52:53.670 --> 00:52:56.177 Like, Lean Startup is like, yeah, you like see if you get something working 00:52:56.260 --> 00:52:58.597 and then you double down and then you see if that works, then you double down. 00:52:58.680 --> 00:53:00.260 Like, you cannot do that in AI. 00:53:00.720 --> 00:53:05.307 You have to like weave— I mean, we've been investing into these data 00:53:05.390 --> 00:53:07.597 centers since, you know, for years and years and years. 00:53:07.680 --> 00:53:08.477 We've had to— 00:53:08.560 --> 00:53:11.777 And you'll be radically behind if you wait to see if things are working. 00:53:11.860 --> 00:53:15.857 So we have these conversations where it's like, At the end of last year, we're like, 00:53:15.940 --> 00:53:19.237 all right, we are quite excited about the progress. 00:53:19.320 --> 00:53:21.027 Yeah, we haven't seen the models yet. 00:53:21.110 --> 00:53:23.947 Like, it's like, I'm going to, like, you know, we're going to write this check 00:53:24.030 --> 00:53:27.407 for, like, I don't know, like $100 billion, $200 billion. 00:53:27.490 --> 00:53:30.330 I'm like, Alex, are you sure this is going to work? 00:53:31.000 --> 00:53:35.667 It's like, um, yeah, so it was, it was so extreme that, you know, 00:53:35.750 --> 00:53:37.507 we had, we had like the whole exec team. 00:53:37.590 --> 00:53:38.967 I think Susan scared you a bit. 00:53:39.050 --> 00:53:43.677 Yeah, well, yeah, no, well, it was like The joke every time we talked 00:53:43.760 --> 00:53:47.887 about it would just be like, yeah, and we hope it's all going to work and be good. 00:53:47.970 --> 00:53:50.807 But like, there were literally, like, at the end of last year, other than 00:53:50.890 --> 00:53:54.397 me giving a vibe report on how our research was going and, you know, 00:53:54.480 --> 00:53:58.820 I could sort of show some plots and whatnot, but it was like, I mean, 00:53:59.320 --> 00:54:01.617 I was feeling good about it, but it was certainly not clear 00:54:01.700 --> 00:54:03.237 at all that it was going to work. 00:54:03.320 --> 00:54:08.497 Yeah. You guys are saying that the essence of training a frontier model is you have 00:54:08.580 --> 00:54:12.120 to push your chips all in before you've been dealt the cards. 00:54:12.290 --> 00:54:16.380 You just have to decide that you're going to develop a frontier model 00:54:16.540 --> 00:54:19.340 and you don't get to meter the inputs. At this scale. 00:54:19.840 --> 00:54:22.547 I mean, if we, like, back a few years ago, the clusters didn't have to be 00:54:22.630 --> 00:54:28.140 this big, but now they do. So I think this is part of the excitement 00:54:28.970 --> 00:54:32.520 of booting up a frontier lab now, right? 00:54:33.060 --> 00:54:35.690 Is basically like the ante is extremely high. 00:54:36.110 --> 00:54:40.357 And I think this is— and like, I kind of like viscerally remember this. 00:54:40.440 --> 00:54:42.157 We were in Hawaii at the end of last year. 00:54:42.240 --> 00:54:44.827 It was kind of like the whole exec team got together. 00:54:44.910 --> 00:54:48.160 And yeah, Susan pulled me aside many times and was like, 00:54:48.620 --> 00:54:52.167 you know, I just want to tell you, if this doesn't work, like, you know, 00:54:52.250 --> 00:54:53.997 this is going to be really, really bad. 00:54:54.080 --> 00:54:57.380 It's going to be very, very bad in financial terms. 00:54:58.420 --> 00:55:01.420 I really need to impress on you that this model has to work. 00:55:01.720 --> 00:55:04.927 it has to be good. If it's good, it's all good. We're great. 00:55:05.010 --> 00:55:07.967 But if it's bad, it's really, really, really bad. 00:55:08.050 --> 00:55:10.017 And Susan, who you know— And that was useful motivation? 00:55:10.100 --> 00:55:14.057 It was. Well, it was like, it was one of these very funny moments 00:55:14.140 --> 00:55:16.817 where I was like, you know, obviously the amounts of money 00:55:16.900 --> 00:55:21.857 are like so obscenely large. What are the amounts of money in training? 00:55:21.940 --> 00:55:23.527 Order of magnitude, whatever you can share, but just 00:55:23.610 --> 00:55:25.277 like how much are they costing? Well, you can do the classic math. 00:55:25.360 --> 00:55:26.987 Yeah, no, I mean, a gigawatt cluster. 00:55:27.070 --> 00:55:31.200 I mean, it's Like, I don't know, I think what you'd see out there would be, 00:55:31.490 --> 00:55:35.370 you know, to operate a gigawatt a year, call it on the order of $15 billion. 00:55:36.170 --> 00:55:40.090 Okay, so it's roughly a $15 billion effort to train a bunch of models. 00:55:40.300 --> 00:55:42.350 To operate a gigawatt cluster for a year. 00:55:42.670 --> 00:55:45.427 If you're talking about building the CapEx, it's probably closer to, 00:55:45.510 --> 00:55:48.137 you know, I mean, you pay for it upfront, so it's probably, you know, 00:55:48.220 --> 00:55:50.550 $50 or $60 billion. 00:55:51.140 --> 00:55:57.900 So, but then that's just kind of, one cluster for one part of training. 00:55:58.190 --> 00:56:00.730 Yeah. Um, then you have all the other parts. 00:56:01.400 --> 00:56:04.650 Um, research isn't just doing the run, right? 00:56:04.740 --> 00:56:07.390 It's like, that's like probably an equal amount of it 00:56:07.860 --> 00:56:11.077 is because for each change that you make, you're testing all the ladders all 00:56:11.160 --> 00:56:13.330 the way up to the big, the big scale. 00:56:13.700 --> 00:56:15.790 Um, and then they're serving the stuff. 00:56:16.160 --> 00:56:18.707 So, I mean, part of the stuff that in some ways— 00:56:18.790 --> 00:56:21.840 So tens of billions of dollars to train a Frontier model now. 00:56:23.210 --> 00:56:26.137 Yeah, definitely the whole, I mean, I've seen the financial rollups, 00:56:26.220 --> 00:56:28.557 tens of billions of dollars is a good estimate for— Yeah. 00:56:28.640 --> 00:56:30.597 And then, but then, but then you want to do something with it, right? 00:56:30.680 --> 00:56:34.177 So I actually think like some of the more complicated questions 00:56:34.260 --> 00:56:39.767 that we had were like, all right, so we're training this model and we know, 00:56:39.850 --> 00:56:43.147 we know we want to train the model cuz we, this is like key, but like 00:56:43.230 --> 00:56:44.857 how much product demand do we think we're going to have? 00:56:44.940 --> 00:56:48.407 Right. Because so now it's like, all right, well the product doesn't exist. 00:56:48.490 --> 00:56:51.867 Um, like, and as in you're trying to build data centers to serve Muse 00:56:51.950 --> 00:56:53.407 for a product that doesn't exist yet. Yeah. 00:56:53.490 --> 00:56:55.657 And we have to make it years in advance, right? 00:56:55.740 --> 00:56:58.707 So this is probably where, like, when we were talking about it last year, 00:56:58.790 --> 00:57:01.790 we were just like, huh, there was a moment where I think 00:57:01.880 --> 00:57:05.627 a bunch of people were like, yeah, maybe, maybe we should just 00:57:05.710 --> 00:57:07.337 like be a little more conservative. 00:57:07.420 --> 00:57:09.257 And it's like, no, we're doing this. 00:57:09.340 --> 00:57:11.097 It's like, it's like, we're going to make it work. 00:57:11.180 --> 00:57:12.257 We're going to make it work. We're going to burn the boats. 00:57:12.340 --> 00:57:13.467 Yeah, this is going to work. 00:57:13.550 --> 00:57:19.097 And But it was like, I mean, even, I mean, I too am an entrepreneur 00:57:19.180 --> 00:57:22.780 and founder and have, you know, at various points in my life taken risk. 00:57:22.900 --> 00:57:26.950 I remember the visceral feeling at the time was like, this is a fuck ton of risk. 00:57:27.490 --> 00:57:30.190 Like, I don't know if I really— Oh, it's only riskier. 00:57:30.490 --> 00:57:31.527 Yeah, yeah, it's only— 00:57:31.610 --> 00:57:37.450 well, I do think this is like one of the like things that I think, Mark, 00:57:37.740 --> 00:57:40.077 is very impressive about you, which is I think you have really 00:57:40.160 --> 00:57:45.090 internalized this concept of You know, it's about how big it is when you win, 00:57:45.250 --> 00:57:46.927 not like, you know, failure is okay. 00:57:47.010 --> 00:57:50.930 It's just like how big is the win when you do win? 00:57:51.300 --> 00:57:52.137 And this was, I think— 00:57:52.220 --> 00:57:55.347 Mark has fully internalized power laws in a way that many people have not. 00:57:55.430 --> 00:57:56.387 I think quite deeply, yeah. 00:57:56.470 --> 00:57:58.767 Well, I think the other piece is, I don't actually think it's 00:57:58.850 --> 00:58:02.390 quite as risky as this. I mean, the numbers are big, but I think— 00:58:02.480 --> 00:58:04.477 Meta has a backstop in terms of how it can use compute. 00:58:04.560 --> 00:58:05.817 Yeah, so I think there's 2 things. 00:58:05.900 --> 00:58:09.500 One is if AI works overall, plenty of other people want all our compute. 00:58:09.650 --> 00:58:13.737 Right? So they're willing to pay, like, I mean, if like the cost is whatever, 00:58:13.820 --> 00:58:16.920 $15 billion a gigawatt, we could get like a lot more for that. 00:58:17.120 --> 00:58:19.707 So I think that you, in order for it not to work out, 00:58:19.790 --> 00:58:23.080 it's not just that we have to fail, AI overall has to fail. 00:58:23.250 --> 00:58:24.083 Yes. 00:58:24.670 --> 00:58:29.047 And then even if that happens, we still have all these other 00:58:29.130 --> 00:58:30.717 uses to productively use compute. 00:58:30.800 --> 00:58:34.590 So in the grand scheme of things, we were definitely building out 00:58:35.090 --> 00:58:39.970 this compute for inference ahead of knowing that the product 00:58:40.060 --> 00:58:41.520 was going to be ready. 00:58:41.930 --> 00:58:45.520 But you were AI-pilled enough that you said GPUs online and powered 00:58:45.610 --> 00:58:48.607 will be useful to someone. Yeah, but also like we believed in what we were building. 00:58:48.690 --> 00:58:54.070 I mean, it's like, I think, and you can look at the kind 00:58:54.610 --> 00:58:57.737 of the trajectory where it's like, all right, first there's ChatGPT 00:58:57.820 --> 00:58:59.240 as the text-based assistant. 00:58:59.870 --> 00:59:02.540 Then, you know, Anthropic got coding agents to work. 00:59:03.160 --> 00:59:07.130 And no one had built the kind of personal consumer version of that. 00:59:07.500 --> 00:59:11.127 And it just seemed like this is the thing that Meta should build in the world. 00:59:11.210 --> 00:59:14.467 And I think that that was just like pretty clear to you and me and Nat, 00:59:14.550 --> 00:59:15.967 like from the beginning, right? 00:59:16.050 --> 00:59:18.717 That that's like, that's kind of what we're here to do. 00:59:18.800 --> 00:59:20.470 And so I don't know. 00:59:20.680 --> 00:59:24.480 I mean, there's always like the real question for me on the Muse launch even 00:59:25.140 --> 00:59:28.607 was you never know when you launch something how it's going to be received. 00:59:28.690 --> 00:59:32.197 Right? So there's always the case where you get a home run, which in this 00:59:32.280 --> 00:59:33.570 case it was, right? Yes. 00:59:33.700 --> 00:59:36.237 And it's just like extremely well received right off the bat. 00:59:36.320 --> 00:59:42.450 But the normal case would be you kind of— it's fine, and you have to like iterate 00:59:42.540 --> 00:59:45.590 a few times to get product-market fit before you really grow. 00:59:46.040 --> 00:59:48.127 In some ways, like, I actually think the product, 00:59:48.210 --> 00:59:51.797 we maybe could have even grown it faster, but we didn't have all the growth stuff 00:59:51.880 --> 00:59:54.757 really ready to go when we shipped it because we are like— our median 00:59:54.840 --> 00:59:56.637 expectation was like, all right, we're probably going to have to like 00:59:56.720 --> 00:59:59.177 iterate on this a bit, and like, that'll be That'll be fine. 00:59:59.260 --> 01:00:03.267 But I don't know, I think like if it takes a few iterations, 01:00:03.350 --> 01:00:05.267 it's like not the end of the world. That's normal. 01:00:05.350 --> 01:00:08.187 Yes. Um, you can have launches where it's like you 01:00:08.270 --> 01:00:10.817 get like, it's kind of catastrophic and then it takes a while to recover, 01:00:10.900 --> 01:00:14.317 but that's not good. But, but it's like, but I mean, even if it took a few 01:00:14.400 --> 01:00:15.907 iterations, it would have been fine. 01:00:15.990 --> 01:00:19.827 So I don't know, I guess it just seemed to me like this was going to be— yeah, 01:00:19.910 --> 01:00:22.697 the numbers were big, but I— it seemed like it was probably— 01:00:22.780 --> 01:00:27.037 Was it bonding for the executive team to be Holding hands, 01:00:27.120 --> 01:00:28.620 taking this risk together. 01:00:28.710 --> 01:00:31.007 You're talking about Susan taking UXi? Bonding or bonding moment? 01:00:31.090 --> 01:00:33.757 Is that a bonding moment? It's like, I mean, it's stressful. 01:00:33.840 --> 01:00:35.960 I think it was—Both. Both. Yeah. 01:00:36.050 --> 01:00:38.967 Being in a stressful situation together is generally bonding. 01:00:39.050 --> 01:00:45.557 Well, one thing that I think is quite notable about Meta is the degree to which, 01:00:45.640 --> 01:00:48.687 like, first of all, most of the exec team has been there quite a long time. 01:00:48.770 --> 01:00:50.227 Like, Nat and I were really newbies. 01:00:50.310 --> 01:00:57.190 And And, you know, most people have been through crazy situations many 01:00:57.280 --> 01:00:59.110 times in the past as well. Mm-hmm. 01:00:59.240 --> 01:01:02.907 And I think Meta as a company is like an extremely resilient company that has 01:01:02.990 --> 01:01:07.617 been through like, you know, many situations that also at the time 01:01:07.700 --> 01:01:09.327 would've seemed incredibly daunting. 01:01:09.410 --> 01:01:13.337 And so everyone's kind of like knows the program, so to speak, 01:01:13.420 --> 01:01:18.797 in terms of, you know, this obviously on paper, and especially when like 01:01:18.880 --> 01:01:22.847 Susan Li, our CFO, is sort of like pulling me aside in the meetings, like, does, 01:01:22.930 --> 01:01:26.177 you know, does seem like a particularly scary situation. 01:01:26.260 --> 01:01:28.937 But I think most people were like, yeah, yeah, no, but we're Meta, 01:01:29.020 --> 01:01:29.767 we're going to get through it. 01:01:29.850 --> 01:01:34.230 Like, I think there was like a real camaraderie around that and sort of like 01:01:34.320 --> 01:01:40.197 quite a nice energy around that, which really I did appreciate because I 01:01:40.280 --> 01:01:45.410 think it would be very easy for, you know, it's the other executives at 01:01:45.530 --> 01:01:49.577 the company who are operating and running— Who are paying the bills. 01:01:49.660 --> 01:01:51.707 Yeah, one of the greatest businesses of all time. 01:01:51.790 --> 01:01:52.623 Yes. 01:01:53.170 --> 01:01:58.210 And it'd be very easy for them to be quite annoyed by this whole endeavor. 01:01:58.300 --> 01:02:01.340 But I think everyone is also very excited about— They are. 01:02:01.430 --> 01:02:04.550 What the opportunities for the company with AI are. 01:02:04.680 --> 01:02:11.640 And I think, like, you know, taking a step back, I think one of Mark's jobs as, like, 01:02:11.730 --> 01:02:16.610 the founder and CEO of the company is to ensure that the company is always 01:02:17.400 --> 01:02:21.900 pushing towards the most ambitious and expansive version of what it can be. 01:02:22.410 --> 01:02:28.450 And I think that one of the great things about this entire MSL and AI push is 01:02:28.540 --> 01:02:31.910 that I think this is one of the rawest expressions of that is 01:02:33.790 --> 01:02:39.337 we knew that AI certainly is a very important technology and it's one 01:02:39.420 --> 01:02:40.757 of the bigger technologies right now. 01:02:40.840 --> 01:02:45.177 But if we were able to build it effectively 01:02:45.260 --> 01:02:49.430 and properly integrate it into everything that we do, as well as like, you know, 01:02:49.600 --> 01:02:52.480 as a full-stack technology company, like build it into, 01:02:53.230 --> 01:02:55.857 you know, these incredible products that we know will exist in the future, 01:02:55.940 --> 01:02:58.340 this could be like immensely expansive for Meta. 01:02:58.520 --> 01:03:04.740 And I think this is true. Like, I think Muse is an incredibly exciting product. 01:03:05.030 --> 01:03:08.780 It has like, you know, it can go in 1,000 different directions. 01:03:09.370 --> 01:03:13.710 I think it's one of like probably the most exciting products to just like You know, 01:03:14.120 --> 01:03:17.587 when like late at night and you just think about all the ways it can go. 01:03:17.670 --> 01:03:22.217 Like, I think Nat, Mark, and I just like, you know, shoot across messages late 01:03:22.300 --> 01:03:25.007 at night of just like, oh, we could do this and we could do this and, 01:03:25.090 --> 01:03:27.467 you know, we can make this move and we can like build this product. 01:03:27.550 --> 01:03:30.010 And I think it's just like, you know, very— 01:03:30.140 --> 01:03:33.267 It's a very exciting time in technology right now. 01:03:33.350 --> 01:03:35.437 Yeah, it's very blue ocean in that way. Yeah. 01:03:35.520 --> 01:03:36.353 Yeah. 01:03:36.650 --> 01:03:38.200 I'm struck by, as we talk about 01:03:39.860 --> 01:03:44.530 Muse and just how it clearly feels like 01:03:44.900 --> 01:03:47.660 the next platform, an agent operating on your behalf. 01:03:48.160 --> 01:03:51.700 You've talked for a long time about not wanting to be 01:03:52.870 --> 01:03:56.710 reliant on other platforms and how kind of the experience of building on 01:03:57.500 --> 01:04:01.167 Android and iOS kind of burned you on being dependent on other. 01:04:01.250 --> 01:04:05.297 And often when you would talk about, say, Reality Labs, is the importance of owning. 01:04:05.380 --> 01:04:08.430 the platform of the future and kind of many years 01:04:08.890 --> 01:04:13.600 working on Reality Labs with the intent of owning the platform of the future. 01:04:14.180 --> 01:04:18.600 And now this new modality comes along that feels 01:04:18.690 --> 01:04:21.147 pretty clearly like how you do a lot of productive work and is 01:04:21.230 --> 01:04:24.380 the platform of the future. And now Meta is really well set up. 01:04:24.570 --> 01:04:28.860 Yeah, no, I mean, it's less about wanting to own a platform for its own sake. 01:04:28.950 --> 01:04:31.300 It's about being able to build the best things. 01:04:31.410 --> 01:04:36.117 And at some level, you know, if you look at like other social apps, 01:04:36.200 --> 01:04:39.120 for example, I think part of the reason why Meta is so 01:04:39.210 --> 01:04:42.087 much better is because we don't just build the app, right? 01:04:42.170 --> 01:04:44.007 A lot of people like think about it, we're like this app company. 01:04:44.090 --> 01:04:45.297 We're not an app company, right? 01:04:45.380 --> 01:04:48.980 It's like we're like a deep technology company and we do this because we 01:04:49.220 --> 01:04:52.307 are world-class at machine learning and like all the recommendation systems 01:04:52.390 --> 01:04:55.687 are operating at a scale that basically few others are in the world. 01:04:55.770 --> 01:04:58.670 I think it's basically like Just us and ByteDance on that, 01:04:58.980 --> 01:05:03.730 and and then like okay we just kind of go all the way to the metal. 01:05:04.150 --> 01:05:09.110 And to me, being able to also deliver go the last mile through the device, 01:05:09.780 --> 01:05:14.907 it does come from a while of having these experiences where we couldn't build 01:05:14.990 --> 01:05:17.040 the experiences that we wanted on phones. 01:05:18.200 --> 01:05:23.207 But I just think for the future, the the models are going to be by far 01:05:23.290 --> 01:05:25.837 the most important part of delivering the technology experience. 01:05:25.920 --> 01:05:31.010 And I mean, this is something that I think we kind of bonded on and agreed on, like, 01:05:31.260 --> 01:05:35.637 very early, is this notion that, like, in every generation of technology 01:05:35.720 --> 01:05:39.390 companies, there's, like, no, there's usually not one company that 01:05:39.480 --> 01:05:41.567 kind of is the— is, like, wins everything. 01:05:41.650 --> 01:05:43.437 I don't think that that's how this is going to go either. 01:05:43.520 --> 01:05:48.400 But with the internet, there were, like, okay, 5 or 6 very big companies and then 01:05:48.490 --> 01:05:51.090 a whole ecosystem of other companies that are great. 01:05:51.280 --> 01:05:56.287 The companies that did the best kind of had the core technology to be 01:05:56.370 --> 01:05:59.330 able to serve people the best there. 01:05:59.790 --> 01:06:03.167 And I think in this next era, there are going to probably be a small 01:06:03.250 --> 01:06:07.127 handful of companies that have the ability to build frontier superintelligence. 01:06:07.210 --> 01:06:11.047 And I think that those companies are going to be the ones that effectively drive 01:06:11.130 --> 01:06:14.630 the agenda and can build most of the interesting, the most interesting 01:06:15.050 --> 01:06:17.677 experiences for consumers, for enterprises, hardware, 01:06:17.760 --> 01:06:19.137 these different kinds of things. 01:06:19.220 --> 01:06:23.057 So, and I just look at it as, okay, like, is it even really a choice, right? 01:06:23.140 --> 01:06:26.817 It's like, yeah, okay, Meta could, will probably exist into the future 01:06:26.900 --> 01:06:28.447 regardless of whether we do this. 01:06:28.530 --> 01:06:31.700 But if we want to be kind of an interesting company that is 01:06:31.860 --> 01:06:35.370 like inventing new things that matter to people in the future, 01:06:35.740 --> 01:06:39.827 I don't see how you can do that if you don't have the ability to build end-to-end 01:06:39.910 --> 01:06:43.127 your own superintelligence and build them into the products that people use. 01:06:43.210 --> 01:06:45.667 And I think that this is something that you just like believed from very 01:06:45.750 --> 01:06:48.837 early on, Is basically, it's like, all right, there 01:06:48.920 --> 01:06:50.297 are going to be like a handful of these. 01:06:50.380 --> 01:06:54.430 Like, it's like there's a kind of personal and consumer-oriented version of this. 01:06:54.760 --> 01:06:58.217 And like, this lane is open because everyone else seems to just be obsessed 01:06:58.300 --> 01:07:01.727 so much with just automating knowledge work, which, I mean, isn't even that 01:07:01.810 --> 01:07:03.767 charismatic version of the future, right? 01:07:03.850 --> 01:07:06.437 Because it's like, all right, like, a big part of our belief too is 01:07:06.520 --> 01:07:10.480 that the kind of highest use of superintelligence is invention 01:07:10.650 --> 01:07:12.190 and creation, not automation. 01:07:12.650 --> 01:07:15.117 And so, I don't know, it just seemed like there's like a big 01:07:15.200 --> 01:07:17.900 place in there, but you kind of need to own the stack. 01:07:21.290 --> 01:07:25.457 How is the Muse-mediated future going to shake 01:07:25.540 --> 01:07:30.050 out with businesses need to— okay, 01:07:30.550 --> 01:07:33.510 businesses variously need to build support 01:07:34.130 --> 01:07:34.963 for 01:07:38.640 --> 01:07:42.680 for Muse, where it might try to computer 01:07:42.770 --> 01:07:45.640 use its way in, but that might be janky in certain ways. 01:07:45.730 --> 01:07:48.437 So you kind of want everyone to have first-class support 01:07:48.520 --> 01:07:50.067 for agents talking to things. Yeah. 01:07:50.150 --> 01:07:54.777 Then the bigger question is they might actively try to block agents 01:07:54.860 --> 01:07:56.237 from interacting with them. 01:07:56.320 --> 01:07:58.787 And this might be because they have a misguided sense where, like, 01:07:58.870 --> 01:08:01.747 it's actually in their interest to support agents, but they don't want to. 01:08:01.830 --> 01:08:04.457 It might actually not be in their interest to support agents. 01:08:04.540 --> 01:08:08.340 You can imagine where maybe they have a very thin moat and it all comes from 01:08:09.090 --> 01:08:13.007 a business that only succeeds by consumers spending time on the site or only succeeds 01:08:13.090 --> 01:08:15.927 by overcharging consumers, then they should rationally not want 01:08:16.010 --> 01:08:17.637 to allow agents because the agents are like, why? 01:08:17.720 --> 01:08:19.870 But those are going to get disrupted. Yeah. 01:08:20.180 --> 01:08:22.137 So talk to me just through the second-order 01:08:22.220 --> 01:08:24.060 effects of how the economy works. 01:08:24.640 --> 01:08:27.437 Well, I think that it starts off with— I mean, we trained into the models 01:08:27.520 --> 01:08:29.060 very good browser use, right? 01:08:29.520 --> 01:08:32.897 So that way we want to make it so that no one No one has to do anything 01:08:32.980 --> 01:08:34.360 to integrate with Muse. 01:08:35.070 --> 01:08:38.360 And then, but then of course, for— we're kind of just going down 01:08:38.450 --> 01:08:42.620 the list of the most important partners, and for people who want to work with us, 01:08:43.080 --> 01:08:45.997 we'll just get direct API access, right? 01:08:46.080 --> 01:08:48.537 And then it's like, okay, then it ends up being way more efficient 01:08:48.620 --> 01:08:50.457 and faster for people to use Muse. 01:08:50.540 --> 01:08:53.417 It's more efficient because now you don't have an agent crawling your website. 01:08:53.500 --> 01:08:57.427 It's just going. So anyone who wants to do that, we eventually want to get there. 01:08:57.510 --> 01:09:01.260 Um, and then I think the fullness of time is 01:09:01.430 --> 01:09:04.720 that, you know, just like today every business has, 01:09:05.390 --> 01:09:10.230 um, a website and social media accounts and email addresses and all these things, 01:09:10.690 --> 01:09:12.227 every business is going to have agents too. 01:09:12.310 --> 01:09:15.527 Mm-hmm. So, and we're also building that on the other side of it. 01:09:15.610 --> 01:09:20.320 And, um, and so of course Muse agents will talk to other types of agents, but they'll 01:09:20.410 --> 01:09:22.117 work really well with our agents, right? 01:09:22.200 --> 01:09:25.077 Because I mean, we'll, we'll be able to integrate that Just, 01:09:25.160 --> 01:09:28.077 I mean, so Muse using websites is just a moment in time thing. 01:09:28.160 --> 01:09:29.247 Just to make it easy for everyone. 01:09:29.330 --> 01:09:32.587 You guys want to have it as a backstop so it's reliable for people using it. 01:09:32.670 --> 01:09:35.210 Yes, and easy for businesses to get bootstrapped. 01:09:35.500 --> 01:09:40.340 But we have, we have way more businesses that are calling us asking to integrate 01:09:40.430 --> 01:09:46.017 deeply, and very few really who have tried to block it. 01:09:46.100 --> 01:09:49.227 And my guess is that the ones that try to block it, it will end up not being 01:09:49.310 --> 01:09:50.817 the right strategic thing for them. 01:09:50.900 --> 01:09:52.567 But we'll see. This will kind of play out over time. 01:09:52.650 --> 01:09:54.727 It saves them a lot of time. Yeah. 01:09:54.810 --> 01:09:55.937 It saves them a lot of money. 01:09:56.020 --> 01:09:59.237 Like, I think these are, these are, like, clearly good things for the consumer. 01:09:59.320 --> 01:10:03.487 Yeah. So in the long arc, you'd clearly expect that demand will 01:10:03.570 --> 01:10:08.450 shift from current avenues more and more towards agentic demand. 01:10:09.330 --> 01:10:12.457 And, you know, there always will be demand that lives elsewhere, 01:10:12.540 --> 01:10:15.837 just in the same way that, like, there's still brick-and-mortar demand in commerce. 01:10:15.920 --> 01:10:17.467 But there's a lot of ecommerce demand. 01:10:17.550 --> 01:10:21.380 And kind of in the same way, like, I think agentic commerce demand 01:10:21.470 --> 01:10:22.967 will grow as a portion of the pie. 01:10:23.050 --> 01:10:25.467 The others will still exist, but will grow as a portion of the pie. 01:10:25.550 --> 01:10:28.887 And I think there's sort of these questions for businesses in the long run, 01:10:28.970 --> 01:10:32.230 like, do they want to participate in that slice 01:10:32.480 --> 01:10:35.060 or that portion of the economy over time? 01:10:35.690 --> 01:10:37.737 And I think it's an incredibly exciting area. 01:10:37.820 --> 01:10:40.777 I think it's, like, very, very exciting to imagine what will 01:10:40.860 --> 01:10:42.447 happen in this agentic commerce future. Yes. 01:10:42.530 --> 01:10:45.160 I actually think Patrick wrote a great tweeticle. 01:10:47.030 --> 01:10:48.367 Is that what we call it these days? I don't know. 01:10:48.450 --> 01:10:50.167 Patrick has his own thing going on with tweets. 01:10:50.250 --> 01:10:51.747 It's like they're buried amongst— 01:10:51.830 --> 01:10:57.797 I noticed that Brother Karamazov was quite a good literary analysis. 01:10:57.880 --> 01:11:01.470 But yeah, no, I think clearly 01:11:02.050 --> 01:11:04.930 this is a discontinuous event for how 01:11:06.100 --> 01:11:09.057 commerce will happen, broadly speaking, or just generally speaking, how 01:11:09.140 --> 01:11:12.767 the entire how all the business pipes will work, maybe even in a broader sense. 01:11:12.850 --> 01:11:16.310 But I think that, you know, you just go back to, this is clearly 01:11:16.400 --> 01:11:17.817 a lot better for the consumers. 01:11:17.900 --> 01:11:18.987 It's a lot better for the customers. 01:11:19.070 --> 01:11:22.070 So there's a natural gravity towards that reality. 01:11:22.400 --> 01:11:25.067 One of the exciting things, I'm curious what you guys see too, 01:11:25.150 --> 01:11:29.617 but I think Muse is one of the first products where you're actually people are 01:11:29.700 --> 01:11:32.370 fully delegating purchases to the agents. 01:11:32.750 --> 01:11:35.837 And this is like a— this is kind of like a, you know, a step change, right? 01:11:35.920 --> 01:11:39.667 Like, I think, you know, we've been talking a lot about agentic 01:11:39.750 --> 01:11:42.637 shopping and like the possibility of it, but now it's like really happening. 01:11:42.720 --> 01:11:45.047 And a lot of people are trusting the agents to shop for them. 01:11:45.130 --> 01:11:48.767 And I think the agents are finding them, you know, in my case, like great teas, or, 01:11:48.850 --> 01:11:52.437 you know, in whomever— whoever's case, like great products for them to purchase, 01:11:52.520 --> 01:11:53.770 which is quite exciting. 01:11:53.930 --> 01:11:58.267 And so I think that there's There's clearly an exciting thing happening here. 01:11:58.350 --> 01:12:01.317 I think, I think part of the value prop— well, one, one part of the value prop 01:12:01.400 --> 01:12:02.947 for the user is just reduced friction. 01:12:03.030 --> 01:12:06.527 Like, hey, I don't feel like click on all these websites and do all this research 01:12:06.610 --> 01:12:08.157 to try to find the exact products. 01:12:08.240 --> 01:12:10.077 Like, that step is taken care of for me. 01:12:10.160 --> 01:12:13.200 And also, like, criteria can just be automatically applied. 01:12:13.330 --> 01:12:15.380 Like, hey, I need something by Wednesday. 01:12:15.500 --> 01:12:18.087 So, you know, the agent will only look for things that could get to me by Wednesday. 01:12:18.170 --> 01:12:19.167 But that's, I mean, keyword search. 01:12:19.250 --> 01:12:20.667 It was insane that we ever relied on it because that's a very 01:12:20.750 --> 01:12:22.337 reasonable constraint. You just want to shop by. 01:12:22.420 --> 01:12:23.253 Yeah, exactly. 01:12:24.340 --> 01:12:28.340 And so I think the like old way of like searching for stuff just feels so 01:12:28.640 --> 01:12:31.140 obviously primitive now, now that you can do this. 01:12:31.350 --> 01:12:34.727 Then there's this sort of like back part of the funnel, so to speak, 01:12:34.810 --> 01:12:38.100 but just like the ease of friction reduction of just, yeah, 01:12:38.230 --> 01:12:41.520 filling out the form, just like getting it ordered is quite nice. 01:12:42.110 --> 01:12:45.570 And I think the long-term thing that we're quite excited about 01:12:46.190 --> 01:12:52.197 is greater personalization. Ultimately, I think one of the exciting things about 01:12:52.280 --> 01:12:58.747 Muse is that it has sort of like this holistic context into you as a person 01:12:58.830 --> 01:13:00.417 and all of the aspects of your life. 01:13:00.500 --> 01:13:04.337 So it sort of like can holistically understand things about your family or 01:13:04.420 --> 01:13:06.637 things about your coworkers or things about your friends. 01:13:06.720 --> 01:13:11.347 And then sort of like help you co-optimize across all of those to get you the exact 01:13:11.430 --> 01:13:16.017 right gift for your, you know, best friend in the context of your friend 01:13:16.100 --> 01:13:17.267 group based on what it knows about your friend group. 01:13:17.350 --> 01:13:20.500 Like, I think there's just like a lot of exciting opportunities 01:13:20.810 --> 01:13:24.647 that are somewhat new and novel when it comes to like these 01:13:24.730 --> 01:13:25.817 personalizations, these recommendations. 01:13:25.900 --> 01:13:29.050 Because historically, a lot of these recommendation systems are 01:13:29.240 --> 01:13:31.700 quite siloed and quite isolated. Mm-hmm. 01:13:32.070 --> 01:13:35.270 And so they're sort of like, they don't see like a holistic view 01:13:35.700 --> 01:13:37.160 of you as a person. 01:13:37.660 --> 01:13:41.417 And Muse has an opportunity to offer much greater levels of personalization. 01:13:41.500 --> 01:13:46.460 Yeah, I think people are already seeing that, where if they ask, like with Muse, 01:13:46.550 --> 01:13:49.887 if you ask it to buy something for you, quite frequently it'll add in some bit 01:13:49.970 --> 01:13:52.677 of context that it pulled in from a different conversation. 01:13:52.760 --> 01:13:54.610 You had a good example of this. Yeah. 01:13:54.850 --> 01:13:59.980 Well, so my 3-year-old daughter loves baking. 01:14:00.190 --> 01:14:03.397 She loves cooking. And so I'm like, all right, I'll start baking stuff. 01:14:03.480 --> 01:14:08.440 It's like I'm normally into smoking meats, but I'll do some baking in order 01:14:08.530 --> 01:14:10.700 to To make, to make Aurelia happy. 01:14:11.530 --> 01:14:14.660 And so we've gotten into this really cute routine where, 01:14:15.950 --> 01:14:19.330 um, you know, we, we, I've been playing with Muse voice mode, 01:14:19.540 --> 01:14:22.890 which isn't fully rolled out yet, but we're, we're playing with it. 01:14:23.000 --> 01:14:25.087 And, um, it does this great thing. 01:14:25.170 --> 01:14:26.797 It segments the, the voices. 01:14:26.880 --> 01:14:30.880 So it knows which one is me and it knows which one is not me based on that. 01:14:31.510 --> 01:14:35.387 And we just like, you know, throughout the week we're talking about 01:14:35.470 --> 01:14:37.097 what we'll want to bake that weekend. Mm-hmm. 01:14:37.180 --> 01:14:42.267 And she'll like give it ideas or like she'll kind of come 01:14:42.350 --> 01:14:48.230 up with crazy things that are not real recipes, but like bounce it off with Muse. 01:14:48.360 --> 01:14:51.900 And then Muse will just make sure that we, 01:14:52.610 --> 01:14:55.910 every Saturday when it's time to bake, 01:14:56.700 --> 01:15:01.037 I get the ingredients from Instacart and they kind of show up and they're 01:15:01.120 --> 01:15:04.087 on time, which I mean, the biggest thing is like if I were just 01:15:04.170 --> 01:15:07.257 starting the activity on Saturday, you'd have to get everything 01:15:07.340 --> 01:15:08.417 and it would take some time. 01:15:08.500 --> 01:15:12.137 But now everything's ready whenever we're going to go sit down and do it. 01:15:12.220 --> 01:15:15.057 This weekend it knew that watermelon was launching, so it threw 01:15:15.140 --> 01:15:18.260 a watermelon in the bag. It was very, very sweet. 01:15:18.350 --> 01:15:19.930 It was very heartwarming. 01:15:21.060 --> 01:15:25.310 It deduced that the recipe did not have 01:15:25.400 --> 01:15:29.777 enough— there was not enough, I guess, 01:15:29.860 --> 01:15:33.277 expense in it to get the delivery, so it needed something else. 01:15:33.360 --> 01:15:34.907 And it was like, you're going to— And it topped it up with a watermelon. 01:15:34.990 --> 01:15:36.867 It was like, do you want this watermelon? I was like, yes. 01:15:36.950 --> 01:15:38.780 I'm like, yes, that's very cute. 01:15:38.870 --> 01:15:41.037 Which I think you, uh, did you— is that the watermelon you ate? 01:15:41.120 --> 01:15:43.470 Yeah, that was the one I ate on the livestream. 01:15:43.620 --> 01:15:45.167 Okay, but the watermelon on the livestream was— 01:15:45.250 --> 01:15:47.037 That was just like, Muse just got it for me. 01:15:47.120 --> 01:15:50.547 That was a Muse—yeah, those are just— You just got it for me. 01:15:50.630 --> 01:15:52.420 But I think there's this, um— 01:15:52.510 --> 01:15:56.047 That was a very memeable moment of that very memeable situation. 01:15:56.130 --> 01:15:58.260 So we have Muse to thank for it. 01:15:58.350 --> 01:16:01.937 So, but here's another interesting thing about the whole thing is I didn't 01:16:02.020 --> 01:16:04.097 really use Instacart that much before. 01:16:04.180 --> 01:16:08.230 You know, it's like every once in a while, maybe twice a year, I would like like, 01:16:08.480 --> 01:16:10.567 use Priscilla's account or something. Yes. 01:16:10.650 --> 01:16:15.900 I went and got an Instacart account to give Muse, and now I use it like once 01:16:15.990 --> 01:16:19.320 or twice a week because like I'm talking with Aurelia with Muse, 01:16:19.410 --> 01:16:22.457 and we're coming up with these recipes, and it's delivering them, and now 01:16:22.540 --> 01:16:24.077 it's like a core part of my workflow. 01:16:24.160 --> 01:16:28.497 So I actually think that the opportunity here isn't just, okay, it's going to be 01:16:28.580 --> 01:16:30.790 20% more efficient on the conversion rate. 01:16:31.040 --> 01:16:34.170 Like, I am a new Instacart like subscriber, and I just 01:16:34.260 --> 01:16:35.467 wouldn't have done this before. Yes. 01:16:35.550 --> 01:16:38.097 So, I do think that there's this question, and then there's this question, 01:16:38.180 --> 01:16:42.220 which I think is these sort of interesting— like, 01:16:42.760 --> 01:16:45.410 this is maybe the more interesting economic question, 01:16:45.680 --> 01:16:48.850 is that Instacart account really for me, or is it for the Muse? 01:16:48.940 --> 01:16:51.937 Because I think, like, over time, I think that the question is, 01:16:52.020 --> 01:16:56.070 is the total addressable market for all of commerce just increasing dramatically? 01:16:56.280 --> 01:16:58.700 Yes. Because it's not just the people anymore. 01:16:58.820 --> 01:17:01.157 Like, now agents are effectively going to have accounts. 01:17:01.240 --> 01:17:05.490 Yes. And I think that's where this starts to, like, become a really very exciting 01:17:05.580 --> 01:17:08.537 opportunity where anyone who sits this out, I think, is going to end 01:17:08.620 --> 01:17:12.880 up feeling sad over time. And why did you have to sign up for an Instacart account? 01:17:13.340 --> 01:17:16.087 Like, we're solving this at Stripe Projects for developer 01:17:16.170 --> 01:17:20.637 stuff where your coding agent— The Stripe integration, by the way, is great. 01:17:20.720 --> 01:17:23.510 You guys did a great job with us on Muse. 01:17:23.640 --> 01:17:25.470 I think we're very happy. 01:17:25.640 --> 01:17:28.517 We have been very excited about agent-to-commerce for a long time, 01:17:28.600 --> 01:17:31.147 obviously, because again, felt to us like clearly how people 01:17:31.230 --> 01:17:33.320 are going to buy things in the future. 01:17:34.610 --> 01:17:38.447 And that was both apparent because no one needs all that friction in their life 01:17:38.530 --> 01:17:43.280 from the actual buying stage, and how early product research with AI had 01:17:43.370 --> 01:17:46.450 product-market fit, even from kind of the earliest AIs. 01:17:47.040 --> 01:17:50.917 It really feels like computer use was the last big unlock that we needed to make 01:17:51.000 --> 01:17:54.750 it happen, because now you can ask your muse to buy anything, more or less. 01:17:54.920 --> 01:17:57.217 And it'll just figure it out with computer use, right? 01:17:57.300 --> 01:17:59.047 And so it's like the ultimate backwards compatibility layer. 01:17:59.130 --> 01:18:01.557 Yeah, it's pretty— this is a set of things that Nat really obsessed over. 01:18:01.640 --> 01:18:02.257 Yeah, yeah. 01:18:02.340 --> 01:18:05.887 I think, I think there was a lot of, like, I think a lot of magic of Muse is 01:18:05.970 --> 01:18:08.470 the fact that like a lot of this stuff just works. 01:18:08.850 --> 01:18:12.437 And so like clearly if everything was just like a little bit 01:18:12.520 --> 01:18:15.527 less reliable, it would feel like a total pain in the ass and nobody would use 01:18:15.610 --> 01:18:16.897 it and people just throw away the app. 01:18:16.980 --> 01:18:20.157 And so I think the fact that so much of this has gotten to the point 01:18:20.240 --> 01:18:24.280 of reliability where You know, it just works is like a huge part of the magic. 01:18:24.410 --> 01:18:26.287 Yeah. What were you saying Matt was obsessed with? 01:18:26.370 --> 01:18:29.537 Oh, well, I mean, well, Matt's kind of obsessed with everything. 01:18:29.620 --> 01:18:30.917 Yeah, well, that's generally a sexy character. 01:18:31.000 --> 01:18:32.377 Yeah, no, it's great, it's great. 01:18:32.460 --> 01:18:35.297 But this is, I mean, but yeah, I mean, this getting this experience to be good, 01:18:35.380 --> 01:18:38.297 I think is something that he and a core group of people on that 01:18:38.380 --> 01:18:41.130 team really focused on. It's kind of funny. 01:18:41.220 --> 01:18:44.890 I think I had a dinner with Patrick, like, I want to say it was 2023, 01:18:46.510 --> 01:18:48.477 where I was, he was like, what do you think we should be doing now? 01:18:48.560 --> 01:18:51.600 I was Oh, you're like, you're already way behind the 8-ball 01:18:51.690 --> 01:18:52.897 on this agentic commerce stuff. 01:18:52.980 --> 01:18:54.517 Like, you got to figure this stuff out. 01:18:54.600 --> 01:18:57.737 Like, agents are going to be buying things and like, you're not prepared for it. 01:18:57.820 --> 01:18:59.277 So you were saying this to Patrick or Patrick saying this to you? 01:18:59.360 --> 01:19:00.487 I was saying this to Patrick. 01:19:00.570 --> 01:19:02.657 Well, because Patrick was very politely asking, what do you 01:19:02.740 --> 01:19:06.327 think we should be doing? And I sort of had this like very like urgent spiel. 01:19:06.410 --> 01:19:09.287 Yeah, and I think it's very, uh, it's funny how that was like— 01:19:09.370 --> 01:19:10.247 Well, thank you for your provocation. 01:19:10.330 --> 01:19:14.250 Well, it was directionally correct, but, uh, but like 3 years early. 01:19:14.830 --> 01:19:18.547 2023, I think. And so, um, no, it's very interesting to look 01:19:18.630 --> 01:19:22.590 at the agentic commerce stats at Stripe because 01:19:23.050 --> 01:19:25.597 we started developing all this stuff and we had a whole bunch of announcements 01:19:25.680 --> 01:19:29.720 at Sessions back in May, uh, and then it only as of September 01:19:29.810 --> 01:19:31.470 basically started going vertical. 01:19:34.310 --> 01:19:37.710 Meta is building increasingly capable models, including Muse Wonder, 01:19:38.060 --> 01:19:41.810 and their models are available on OpenRouter, Alongside hundreds of others. 01:19:42.070 --> 01:19:46.070 Stripe has agreed to acquire OpenRouter, who's the leading model gateway. 01:19:46.780 --> 01:19:50.617 OpenRouter already routes more than 10 trillion tokens a day and handles much 01:19:50.700 --> 01:19:53.167 of the complexity of working with different models through a single 01:19:53.250 --> 01:19:57.540 API, making it easier to experiment, find competitive prices, and automatically 01:19:57.630 --> 01:19:59.130 route around provider outages. 01:19:59.630 --> 01:20:01.960 Try it for yourself at openrouter. ai. 01:20:06.640 --> 01:20:11.970 Hey, could you order a watermelon to Stripe's office in South San Francisco? 01:20:12.270 --> 01:20:14.570 I'm doing a podcast here. This would be funny. 01:20:16.690 --> 01:20:17.850 Funny podcast idea. 01:20:18.480 --> 01:20:20.480 I'll check that watermelon delivery now. 01:20:22.360 --> 01:20:24.030 Great. We'll see how we get on. 01:20:26.570 --> 01:20:29.420 Well, okay. This actually raises an interesting question. 01:20:29.820 --> 01:20:34.450 People have been trying to make The HAL 9000. 01:20:34.580 --> 01:20:35.413 It's working. 01:20:36.750 --> 01:20:37.587 I really like this. 01:20:37.670 --> 01:20:40.507 I think this is an awesome touch where it's like clacking away at its laptop. 01:20:40.590 --> 01:20:42.300 I really like that in the app. 01:20:43.420 --> 01:20:46.510 We'll just leave them there and we'll see what we get. 01:20:46.720 --> 01:20:50.430 Yeah, it's tough to prop up. Yeah, there we go. 01:20:52.220 --> 01:20:55.767 People have been trying to make the speak to a computer AI vision 01:20:55.850 --> 01:20:57.390 happen for a really long time. 01:20:57.980 --> 01:21:03.230 Where Apple launched Siri in 2014, 2015, something like that. 01:21:03.770 --> 01:21:08.280 And obviously there's been Voice Mode and various AI products for a few years now. 01:21:08.400 --> 01:21:10.197 Why is it so hard? Like, it's still not the case. 01:21:10.280 --> 01:21:14.157 If you were driving your car and you wanted to check on the baking or move 01:21:14.240 --> 01:21:16.117 a meeting or something like that, you actually probably 01:21:16.200 --> 01:21:17.787 wouldn't do it with the voice. 01:21:17.870 --> 01:21:19.837 I mean, maybe almost now with the latest things. 01:21:19.920 --> 01:21:20.707 Yeah, well, there's a lot. 01:21:20.790 --> 01:21:22.297 I mean, so— But you wouldn't develop Siri like you're doing. 01:21:22.380 --> 01:21:24.797 Siri right now is actually a good example. As we're talking over each other. 01:21:24.880 --> 01:21:26.177 And I mean, I don't know, when you're using different voices— 01:21:26.260 --> 01:21:29.007 So is it just technical limitations? That's why we haven't done it? 01:21:29.090 --> 01:21:30.847 Well, I think that there's a few things. 01:21:30.930 --> 01:21:32.927 And one is you actually want it to go do stuff. 01:21:33.010 --> 01:21:36.227 So there's a question of when you speak with it, even if it understands, 01:21:36.310 --> 01:21:37.560 how's that going to work? 01:21:38.020 --> 01:21:41.147 Then I think that there's a lot of stuff about the way that people 01:21:41.230 --> 01:21:42.147 have conversations. 01:21:42.230 --> 01:21:46.860 It isn't typically, I will give a prompt, and now I will wait, and you will speak. 01:21:47.030 --> 01:21:49.080 And then, so it just goes back and forth. 01:21:49.490 --> 01:21:51.657 Um, how many of your sessions when you're 01:21:51.740 --> 01:21:56.037 talking to an agent in voice mode, do you basically, oh, 01:21:56.120 --> 01:21:59.167 it starts saying something, but then like, I don't know, it's like, 01:21:59.250 --> 01:22:02.167 I kind of feel like if I pause for 3 seconds, it's going to think 01:22:02.250 --> 01:22:03.837 I'm done and want to speak. 01:22:03.920 --> 01:22:07.257 And then if it starts speaking, but then I cut back in, is it going to 01:22:07.340 --> 01:22:09.440 think that it's like a whole new question? 01:22:09.930 --> 01:22:12.847 Um, I think there's a lot of the stuff about the conversational dynamics 01:22:12.930 --> 01:22:14.280 that you want to get right. 01:22:14.430 --> 01:22:20.560 But then there's this other issue, which is that For voice overall, 01:22:21.770 --> 01:22:24.810 it's quite high bandwidth for you to communicate with voice, 01:22:25.520 --> 01:22:30.067 but in terms of consuming information, having someone speak to you is very 01:22:30.150 --> 01:22:32.197 inefficient compared to having it be visual. 01:22:32.280 --> 01:22:35.870 Yeah. So I think like that's why, you know, partially when we do these, 01:22:36.870 --> 01:22:39.747 these interfaces, I think it's, it's kind of working on its task. 01:22:39.830 --> 01:22:40.790 I fired it off now. 01:22:41.370 --> 01:22:45.090 So I think the ideal interface is probably not just strictly voice-voice. 01:22:45.590 --> 01:22:49.800 Although I think voice output from you is pretty useful. 01:22:50.300 --> 01:22:54.307 Yeah, I think, I think there very clearly already is an explosion of voice input. 01:22:54.390 --> 01:22:57.040 Like, I think you even see this in coding where like— 01:22:57.350 --> 01:22:59.097 Yeah, people are speaking to the— Yeah, yeah, yeah. 01:22:59.180 --> 01:23:03.350 Most of the researchers just kind of like whisper into their keyboard because it is 01:23:03.440 --> 01:23:07.520 like, as Mark's mentioning, it's like the highest bandwidth way to get your like 01:23:07.610 --> 01:23:10.567 thoughts out, or usually the lowest friction way to get your thoughts out. 01:23:10.650 --> 01:23:11.500 Yeah, yeah, yeah. 01:23:11.650 --> 01:23:15.050 I think the most extreme version of this, by the way, is I think how 01:23:16.200 --> 01:23:18.867 college students now use AI is maybe the most interesting, 01:23:18.950 --> 01:23:22.290 where they just— I think a very common modality of how to use 01:23:23.040 --> 01:23:28.920 AI is that they just will completely word vomit for 10, 15 minutes, 01:23:29.170 --> 01:23:32.800 being full stream of consciousness, chain of thought of all the things. 01:23:33.050 --> 01:23:35.057 And then have AI clean it up and infer. Yeah. 01:23:35.140 --> 01:23:40.350 And have AI effectively do all of this organization of it and pulling things 01:23:40.680 --> 01:23:42.397 together and structuring everything. 01:23:42.480 --> 01:23:43.557 It's quite interesting. 01:23:43.640 --> 01:23:46.770 It's a very interesting, I think, modality of tech usage. 01:23:47.100 --> 01:23:49.227 Yeah. But yeah, I think voice input has been big. 01:23:49.310 --> 01:23:54.027 I do think agents is like, that's a big step change, because I think 01:23:54.110 --> 01:23:57.447 the mental model is just very different. 01:23:57.530 --> 01:24:02.120 Like, if it's just like an informational thing that you're talking to, that like, 01:24:02.540 --> 01:24:04.997 you will clearly get into situations where it just feels useless. 01:24:05.080 --> 01:24:08.957 Whereas now that it can do things for you, it's much closer to the experience 01:24:09.040 --> 01:24:10.217 of talking to a human. Totally. 01:24:10.300 --> 01:24:11.133 Or to a person. 01:24:11.550 --> 01:24:16.757 And so I think there's all these interesting uncanny valleys in interacting 01:24:16.840 --> 01:24:22.470 with AIs that until they've been able to take actions have felt kind of strange. 01:24:22.560 --> 01:24:28.147 And now that we have agents, I think one of the cool things about Muse 01:24:28.230 --> 01:24:30.227 is this confluence of a number of things. 01:24:30.310 --> 01:24:34.740 I think The fact that Muse is embodied and it like is this little guy 01:24:35.320 --> 01:24:38.987 that like you feel like is working on your behalf is also only possible 01:24:39.070 --> 01:24:44.160 and a coherent like representation and embodiment of the product because 01:24:44.660 --> 01:24:48.580 it has persistent memory and it has the ability to do things for you 01:24:48.670 --> 01:24:51.667 and the ability to take actions and has like a little bit of a personality. 01:24:51.750 --> 01:24:54.797 Like, is the confluence of all these things coming together that make 01:24:54.880 --> 01:24:56.580 this like embodiment concept— Yes. 01:24:57.180 --> 01:24:58.630 Even work in the first place. 01:24:59.180 --> 01:25:01.427 Then it turns out, okay, actually this is the real glasses, right? 01:25:01.510 --> 01:25:05.227 It's like, okay, a lot of people want it even with just like no 01:25:05.310 --> 01:25:06.910 display or a small display. Yes. 01:25:07.770 --> 01:25:10.770 The camera and audio is enough of a useful product. 01:25:11.110 --> 01:25:12.020 Yeah, yeah. 01:25:12.110 --> 01:25:16.690 And I— and then we started shipping the display glasses that aren't the kind 01:25:16.780 --> 01:25:19.777 of wide field of view full augmented reality, which, I mean, 01:25:19.860 --> 01:25:25.117 I thought would be awesome too, but even the small display is pretty powerful. 01:25:25.200 --> 01:25:29.000 So I think you'll start getting some interesting interactions there that are 01:25:29.460 --> 01:25:32.000 kind of fully multimodal, right? 01:25:32.130 --> 01:25:36.337 So it's voice in and out and It'll be able to see what you see, and it'll be 01:25:36.420 --> 01:25:38.840 able to kind of paint the pixels. 01:25:39.590 --> 01:25:42.240 And one of the— I mean, the other exciting thing that 01:25:43.140 --> 01:25:45.590 we've done here is Muse Gadgets, which is really, 01:25:45.890 --> 01:25:51.940 I think, unleashing the ecosystem to try cool device formats and form factors 01:25:52.270 --> 01:25:53.397 that could be really exciting. 01:25:53.480 --> 01:25:56.647 Like, since releasing that, we've gotten actually even a number 01:25:56.730 --> 01:26:00.197 of phone manufacturers, you know, talk to us about excited You 01:26:00.280 --> 01:26:02.407 know, we're like, we're just going to integrate Muse Gadgets 01:26:02.490 --> 01:26:04.867 into our phone and see how well it works. And like, we're really excited about that. 01:26:04.950 --> 01:26:07.997 Like, I think, um, I think part of what is cool about 01:26:08.080 --> 01:26:11.997 the current moment of technology is that like, it's very clear 01:26:12.080 --> 01:26:13.247 agents are really exciting. 01:26:13.330 --> 01:26:16.500 It's really clear that like, this should unlock lots of things. 01:26:16.710 --> 01:26:18.547 Will you guys come back to the household devices? 01:26:18.630 --> 01:26:21.637 Like, this is obviously a personal device, but, um, we actually had 01:26:21.720 --> 01:26:25.010 during COVID the, um, Portress— the Portal. 01:26:25.100 --> 01:26:28.557 Portal. There we go. Yeah, I mean, like, he's finding the Portal inside. 01:26:28.640 --> 01:26:30.977 One of the gadgets that we just shipped was the little HomeLink. 01:26:31.060 --> 01:26:34.060 It's like the size of a little, like, nickel or something, 01:26:34.230 --> 01:26:36.900 but with a USB, right? Yeah. 01:26:37.070 --> 01:26:41.277 Um, uh, so you can basically connect that, and then your Muse can connect 01:26:41.360 --> 01:26:42.157 to everything in your home. 01:26:42.240 --> 01:26:47.280 This is actually one of the most fun things too is just having it connect to 01:26:47.580 --> 01:26:49.577 your Sonos and connect to all your lights. 01:26:49.660 --> 01:26:53.167 I mean, I have it basically like when it's time for me to start 01:26:53.250 --> 01:26:56.457 winding down in the evening, I just have it like dim all the lights 01:26:56.540 --> 01:26:59.630 across the house and it's nice. 01:27:00.090 --> 01:27:01.967 How are we going to fund the AI buildout? 01:27:02.050 --> 01:27:05.557 Everyone's worried about, you know, Google went free cash flow negative. 01:27:05.640 --> 01:27:08.687 You know, you were talking about 1 gigawatt and then 5 gigawatts 01:27:08.770 --> 01:27:09.977 and everything like this. 01:27:10.060 --> 01:27:15.310 Meta's now free cash flow negative yet, but there's always more CapEx to fund. 01:27:15.690 --> 01:27:18.480 And so what's the answer? Where is Susan Lee? 01:27:18.570 --> 01:27:24.447 Exactly. No, I mean, look, I think that there are going 01:27:24.530 --> 01:27:26.787 to be a lot of ways to create value from this. 01:27:26.870 --> 01:27:32.000 So what we're trying to do with Muse specifically is make it so that it can be 01:27:32.700 --> 01:27:35.037 widely used, because I think that there are going to be 01:27:35.120 --> 01:27:39.797 some kind of valuable network effects at scale as people start being able 01:27:39.880 --> 01:27:43.047 to interact with each other's, and that's going to be great. 01:27:43.130 --> 01:27:45.927 So I think there are all these reasons why you want it to get to large scale. 01:27:46.010 --> 01:27:47.387 Also, you know, we were talking about partners. 01:27:47.470 --> 01:27:50.017 Partners are going to want to integrate more the more people are using it. 01:27:50.100 --> 01:27:52.850 So it will get more valuable if we can get it to scale. 01:27:53.060 --> 01:27:56.810 The best way to do that is to basically have a model where it is affordable 01:27:56.980 --> 01:27:58.360 and free for people to use. 01:27:58.940 --> 01:28:02.640 But there are going to be a lot of ways that I think we can scale revenue. 01:28:03.650 --> 01:28:05.860 Subscriptions is going to be a big one. 01:28:06.070 --> 01:28:09.830 You know, basically, just because the base version of it is 01:28:09.950 --> 01:28:12.997 free and we want the free version to be excellent doesn't mean that there are 01:28:13.080 --> 01:28:16.417 things that just naturally aren't going to be economical to include in that. 01:28:16.500 --> 01:28:20.667 So, you know, if you want the kind of— you know, we talked about this earlier. 01:28:20.750 --> 01:28:25.257 If you want to start using Muse Wonder in Muse tomorrow, then, you know, 01:28:25.340 --> 01:28:28.307 the way that you'll do that is you'll pay for a subscription. 01:28:28.390 --> 01:28:30.590 So I think that that will be a big business. 01:28:31.060 --> 01:28:33.597 There are a bunch of other things around commerce that we're quite excited about 01:28:33.680 --> 01:28:37.817 too, where we basically think— we believe that Muse is actually going to make people 01:28:37.900 --> 01:28:42.187 money and save people money by using it, and that over time there are lots 01:28:42.270 --> 01:28:46.360 of different ways to effectively take a small cut of what you are saving or 01:28:46.450 --> 01:28:50.740 making for people, and that will end up being a very large revenue stream. 01:28:51.280 --> 01:28:52.697 And then there's all the other things, right? 01:28:52.780 --> 01:28:56.790 So, you think Muse will generally just cost negative money for people? 01:28:57.040 --> 01:28:58.417 because it helps them out. 01:28:58.500 --> 01:29:00.547 Well, there are already all these sites that I think are interesting where people 01:29:00.630 --> 01:29:03.967 are like, go look through my emails, find a bunch of subscriptions I don't 01:29:04.050 --> 01:29:06.257 actually use, cancel them, whatever. 01:29:06.340 --> 01:29:10.967 Um, yeah, so yeah, I mean, that, um, I don't know if that's the thing that we 01:29:11.050 --> 01:29:13.887 would monetize, but, but like, but you get, you get, you kind of get the 01:29:13.970 --> 01:29:15.557 point and where it helps you make money. 01:29:15.640 --> 01:29:18.437 I mean, one of the ideas that we kind of talk about, I mean, this is, 01:29:18.520 --> 01:29:21.437 this is something that I think you've been kind of obsessed with, is making it like 01:29:21.520 --> 01:29:23.570 a business in a box for small businesses. 01:29:24.400 --> 01:29:26.657 So someone can just kind of— it can help them administer all these 01:29:26.740 --> 01:29:27.567 parts of what they're doing. 01:29:27.650 --> 01:29:30.867 And you can imagine that being very valuable and being very profitable, 01:29:30.950 --> 01:29:33.160 both for the business owner and for Meta. 01:29:35.080 --> 01:29:37.480 But I think one of the things that's interesting 01:29:38.120 --> 01:29:43.587 is, like, OK, if we're building the models to build the personal agents 01:29:43.670 --> 01:29:47.007 and the business agent, there is this large surplus of value 01:29:47.090 --> 01:29:49.047 because now the model, it's very general, right? 01:29:49.130 --> 01:29:50.887 So it doesn't just do those things. 01:29:50.970 --> 01:29:53.270 We'll also have the API that'll be profitable. 01:29:53.850 --> 01:29:58.180 A big part of why we want to start the Meta Enterprise Platform 01:29:58.270 --> 01:30:01.687 and have CJ Desai join us to build this is we have this view, which is like, 01:30:01.770 --> 01:30:05.720 all right, yeah, sure. So traditionally Meta hasn't been an enterprise company, 01:30:05.940 --> 01:30:08.817 but we believe that a huge part of the challenge is building 01:30:08.900 --> 01:30:10.867 the leading models in the first place. And they're very general. 01:30:10.950 --> 01:30:16.790 And once we do that, if we make a significant investment in building 01:30:17.240 --> 01:30:21.497 an enterprise group that can provide that in the way that enterprises need 01:30:21.580 --> 01:30:24.087 to consume it, then I mean, I think a lot of enterprises are 01:30:24.170 --> 01:30:25.467 going to want to use Muse Wonder. 01:30:25.550 --> 01:30:27.837 It's going to be an excellent model and affordable. 01:30:27.920 --> 01:30:31.797 And like, it's so, um, so I think that that is going to end 01:30:31.880 --> 01:30:33.640 up being a very big business too. 01:30:33.930 --> 01:30:38.470 And all of these things put together, coupled with the fact that we were already 01:30:38.560 --> 01:30:41.230 starting off as a very profitable business that can, 01:30:41.730 --> 01:30:45.937 that has good credit and can kind of underwrite these large investments, 01:30:46.020 --> 01:30:50.237 I think puts us in a pretty good position as a lab to be able to build out 01:30:50.320 --> 01:30:54.027 the compute that we need to build leading products, build up the network around 01:30:54.110 --> 01:30:59.617 this, and position the company to really lead in the area for decades to come. 01:30:59.700 --> 01:31:00.167 Okay. 01:31:00.250 --> 01:31:02.577 So, you think ultimately there'll just be a lot of revenue from these products, 01:31:02.660 --> 01:31:04.837 and so they'll be self-funding in fairly short order? 01:31:04.920 --> 01:31:05.707 Yeah. 01:31:05.790 --> 01:31:07.547 I mean, that's a much shorter answer than mine. 01:31:07.630 --> 01:31:10.257 It's like— But you're saying people are underestimating— 01:31:10.340 --> 01:31:13.640 The question was, Are we going to be able to make money from this? 01:31:13.930 --> 01:31:15.180 The answer better be yes. 01:31:16.100 --> 01:31:19.637 I think the history— Yes. The answer is yes. 01:31:19.720 --> 01:31:21.977 But the history is very front-loaded expense, right? 01:31:22.060 --> 01:31:25.397 Well, the history of AI as an industry, I think, to some degree is like, 01:31:25.480 --> 01:31:28.937 you know, people have always sort of been hand-wringing about, you know, how are 01:31:29.020 --> 01:31:30.527 these businesses going to make money? 01:31:30.610 --> 01:31:34.400 And I think, you know, every year people are sort of generally 01:31:34.490 --> 01:31:35.780 surprised at the upside. 01:31:36.320 --> 01:31:39.270 That superintelligence is Is a useful, incredibly valuable— 01:31:39.370 --> 01:31:42.077 Yeah, exactly. This is what— this is kind of Meta's whole playbook, right? 01:31:42.160 --> 01:31:45.000 Is we build something, we get it to a billion people, 01:31:45.210 --> 01:31:49.627 then we scale the revenue while growing our revenue primarily 01:31:49.710 --> 01:31:53.220 in the meantime by continuing to scale the things that are already 01:31:53.380 --> 01:31:54.967 kind of at scale and growing. Yeah. 01:31:55.050 --> 01:31:58.220 I think this is also one of the dynamics, like, I think, 01:31:59.310 --> 01:32:01.637 you know, I think Dario's talked about this as well, 01:32:01.720 --> 01:32:04.307 but like, just in general, because it requires so much advanced 01:32:04.390 --> 01:32:06.650 planning to build these models. 01:32:07.020 --> 01:32:09.860 Like, if you look at the R&D expense at any given time, 01:32:10.230 --> 01:32:13.777 that is like over— an overlapping R&D expense of multiple model 01:32:13.860 --> 01:32:15.280 generations at once. Yes. 01:32:15.650 --> 01:32:19.200 And, you know, these, they require a long time to work on. 01:32:19.370 --> 01:32:22.247 And then ultimately, you know, you see like immense payoff 01:32:22.330 --> 01:32:23.577 when, when you ultimately deliver them. 01:32:23.660 --> 01:32:27.290 And this is kind of what we saw with the Muse Spark family of models. 01:32:27.380 --> 01:32:30.730 Like, you know, again, we started working on these a long time ago. 01:32:31.050 --> 01:32:34.547 And are seeing like incredible consumer adoption through the Muse product. 01:32:34.630 --> 01:32:38.057 The other thing is that we use the models to accelerate our core business, right? 01:32:38.140 --> 01:32:41.137 So I mean, you have this core business that's hundreds of billions of dollars 01:32:41.220 --> 01:32:42.887 and still growing extremely quickly. 01:32:42.970 --> 01:32:46.890 And now, like, if you can grow it a few percent faster by using these models 01:32:46.980 --> 01:32:50.977 to accelerate these things, then that ends up funding a huge amount 01:32:51.060 --> 01:32:52.910 of this too as you're scaling it out. 01:32:53.190 --> 01:32:56.340 The Meta core business is a, is a like 8th wonder of the world. 01:32:56.530 --> 01:32:58.067 It's extremely good business. It really is. 01:32:58.150 --> 01:33:03.990 is quite incredible and is growing at sort of astonishing rates. 01:33:05.080 --> 01:33:09.210 Yeah, you guys really cooked with that one. 01:33:10.040 --> 01:33:13.167 Yeah, I think you're also getting at something, Alex, which is like there's 01:33:13.250 --> 01:33:16.880 an interesting accounting artifact where if you go build a data center, 01:33:17.340 --> 01:33:20.887 you have CapEx and you get to depreciate that over a useful life of many years. 01:33:20.970 --> 01:33:24.187 And people debate what that useful life is, but the point is you do get 01:33:24.270 --> 01:33:28.727 to depreciate it, whereas a lot of the AI costs are also just 01:33:28.810 --> 01:33:31.910 in researcher salaries, and those salaries are being spent on, 01:33:32.110 --> 01:33:35.780 you know, model training that also has a useful life of many years, 01:33:36.110 --> 01:33:37.987 but you do not get to capitalize generally. 01:33:38.070 --> 01:33:41.577 And that's also a financial advantage for us building out the data centers as 01:33:41.660 --> 01:33:44.990 opposed to a bunch of the other companies that are paying clouds, 01:33:45.200 --> 01:33:48.300 whereas they're paying the annual rate, but then at some point 01:33:49.250 --> 01:33:52.797 the GPUs will be fully depreciated, but we're probably still going to be using 01:33:52.880 --> 01:33:56.797 them for a while because they I mean, the value of even like an H100 at this 01:33:56.880 --> 01:33:59.130 point is higher than it was last year, right? 01:33:59.220 --> 01:34:01.820 Just because if compute continues being constrained. 01:34:02.510 --> 01:34:06.140 So yeah, I think it's— yeah, I think— 01:34:06.970 --> 01:34:08.727 Any advice for us with OpenRadar? 01:34:08.810 --> 01:34:10.227 We have agreed to acquire them. 01:34:10.310 --> 01:34:11.147 It has not closed yet. 01:34:11.230 --> 01:34:17.440 Yeah, I think part of the exciting— part of what we're starting to see 01:34:17.530 --> 01:34:20.567 in terms of the models, especially the frontier models that are 01:34:20.650 --> 01:34:24.300 being developed, like Muse Wonder versus, let's say, a Fable or an Astra, 01:34:24.450 --> 01:34:30.537 is that we actually are starting to focus in quite different directions 01:34:30.620 --> 01:34:31.417 from the other models. 01:34:31.500 --> 01:34:33.797 And I think there's a lot of intentionality that goes into the model 01:34:33.880 --> 01:34:36.587 development process that we've discussed as part of this, you know, 01:34:36.670 --> 01:34:39.920 in this conversation, that is genuinely different 01:34:40.010 --> 01:34:41.507 from how other models are focusing. 01:34:41.590 --> 01:34:43.807 And so, you know, when these models kind of get 01:34:43.890 --> 01:34:48.347 collapsed to the same set of benchmarks, that elides all of the— 01:34:48.430 --> 01:34:53.187 This is a good point. The interesting, um, sort of like, uh, texture that differs 01:34:53.270 --> 01:34:54.267 between the various models. 01:34:54.350 --> 01:34:58.357 And so I think one thing that is an exciting opportunity for OpenRadar, 01:34:58.440 --> 01:35:02.860 generally speaking, is like, you know, um, helping to Helping developers 01:35:02.990 --> 01:35:06.577 understand what these sort of subtle differences are so that they can make 01:35:06.660 --> 01:35:10.110 the most informed decisions about what models ultimately they select. 01:35:10.290 --> 01:35:13.390 And then also you can imagine the flip side of that, which is, 01:35:14.500 --> 01:35:16.950 you have all these developers using these models. 01:35:17.040 --> 01:35:20.587 And so those developers are great people to ask about what are the interesting 01:35:20.670 --> 01:35:25.927 textual differences between these models that are more than just tweet length. 01:35:26.010 --> 01:35:30.930 Like, this model's good, this model's bad, more than just that level of insight. 01:35:31.020 --> 01:35:35.310 And I think this is like getting to the wine sommelier 01:35:35.980 --> 01:35:39.277 version or form of understanding the models and their sort of relative 01:35:39.360 --> 01:35:41.987 strengths and weaknesses, I think would be quite exciting. 01:35:42.070 --> 01:35:45.117 Okay, so based on your experience, the OpenRadar view that there will be 01:35:45.200 --> 01:35:49.037 a diversity of models in the future and they're optimized for different tasks and— 01:35:49.120 --> 01:35:52.670 I agree with that. Exactly. That's just your guys' experience based on— 01:35:53.200 --> 01:35:56.497 Yeah, we are focusing On different things. 01:35:56.580 --> 01:36:01.210 We care a lot more about personal and professional agents, my observation, 01:36:01.340 --> 01:36:04.930 than the other major labs, which is good. 01:36:05.180 --> 01:36:06.927 There should be differentiation in the economy. 01:36:07.010 --> 01:36:09.177 This is the point of capitalism to some extent. 01:36:09.260 --> 01:36:10.727 Yeah. So you're going to say something? 01:36:10.810 --> 01:36:12.397 No, I just really agree with that point. 01:36:12.480 --> 01:36:16.517 I mean, I think that part of what I love about you guys working on OpenRouter 01:36:16.600 --> 01:36:20.567 and being able to scale it is just the more infrastructure that can get 01:36:20.650 --> 01:36:23.527 developed, that supports a diversity of players 01:36:23.610 --> 01:36:25.710 in the ecosystem, I think is good overall. 01:36:26.240 --> 01:36:30.697 I also, for the same reason, am quite a fan of some of the recent NVIDIA 01:36:30.780 --> 01:36:33.580 moves to support the open source ecosystem. 01:36:34.040 --> 01:36:38.120 I don't know, but this just kind of goes with our whole theory and philosophy 01:36:38.210 --> 01:36:39.957 about, like, what does a good future look like? 01:36:40.040 --> 01:36:43.260 Mm-hmm. We think that a good future is one where you achieve a, 01:36:44.510 --> 01:36:47.430 like, you basically get prosperity by empowering people. 01:36:47.640 --> 01:36:49.677 That's kind of the personal superintelligence thing that we're doing, 01:36:49.760 --> 01:36:52.010 but there's the whole business version. Yeah. 01:36:52.470 --> 01:36:55.270 But for a lot of the important safety topics, 01:36:55.850 --> 01:36:59.690 like, our view is that the way to get there is having a diversity 01:37:00.150 --> 01:37:03.567 and having checks and balances by kind of having a lot of different things out 01:37:03.650 --> 01:37:06.697 there and accessible to people, not like just having a couple of good 01:37:06.780 --> 01:37:08.867 models that are only in a couple companies' hands. 01:37:08.950 --> 01:37:12.540 I think that that would be— from my perspective, I think that is 01:37:13.370 --> 01:37:19.290 probably a riskier configuration for the future than lots of people 01:37:19.380 --> 01:37:20.877 having access to different things. 01:37:20.960 --> 01:37:22.417 Do you guys have a theory as to why that is? 01:37:22.500 --> 01:37:28.170 Where I feel like maybe people predicted more of a winner-takes-all dynamic in AI, 01:37:28.430 --> 01:37:32.260 or one person clearly— like, a story that people told back in 2023 is 01:37:32.350 --> 01:37:36.270 that one AI lab will start to develop an advantage, 01:37:36.600 --> 01:37:39.357 and that will allow them to get recursive self-improvement going, 01:37:39.440 --> 01:37:42.977 And then they'll just pull further and further away from everyone else. 01:37:43.060 --> 01:37:47.320 Whereas it has remained extremely competitive where 01:37:47.610 --> 01:37:49.447 there's kind of multiple people at the frontier. 01:37:49.530 --> 01:37:53.870 You guys with Wonder today are coming along with your frontier-class model. 01:37:54.450 --> 01:37:58.750 Open source models have done an impressive job of keeping up, all told. 01:37:59.330 --> 01:38:03.000 And so we had this story a few years ago that someone was just going 01:38:03.090 --> 01:38:05.920 to run away with the prize. Why hasn't that happened? 01:38:06.920 --> 01:38:11.930 I think there's a lot of underlying reasons and mechanics that drive this. 01:38:12.430 --> 01:38:14.310 One certainly is just execution. 01:38:14.600 --> 01:38:20.017 I think for any given frontier lab, there are periods of strong 01:38:20.100 --> 01:38:24.020 execution and weak execution. And I think that is true across all the labs. 01:38:24.110 --> 01:38:27.060 And so that sort of starts to average itself out over time. 01:38:28.110 --> 01:38:29.710 I think another—Very diplomatic. 01:38:31.200 --> 01:38:34.870 Sorry. Yeah. Is it that some of the Players in the lead screwed up? 01:38:35.660 --> 01:38:37.617 Well, I think if all players, 01:38:37.700 --> 01:38:39.867 when they were in the lead, executed perfectly, they would 01:38:39.950 --> 01:38:41.870 probably be able to stay in the lead. 01:38:42.370 --> 01:38:46.710 So I think that's one factor. I think the power and strength of distillation is 01:38:46.800 --> 01:38:48.967 certainly another factor that's certainly benefiting the open source 01:38:49.050 --> 01:38:51.507 ecosystem quite a bit. And I think this is just like a— 01:38:51.590 --> 01:38:55.760 There's an anti-hegemonic force in the phenomenon of distilling a model. 01:38:56.010 --> 01:38:59.350 Yeah. And this is just a quality of the technology. 01:39:00.640 --> 01:39:04.017 It's not like anyone planned for distillation to work super duper well, 01:39:04.100 --> 01:39:10.817 but it just turns out that you can get a lot of capabilities naturally 01:39:10.900 --> 01:39:12.527 through some form of distillation. 01:39:12.610 --> 01:39:15.197 And this is, broadly speaking, distillation is quite 01:39:15.280 --> 01:39:20.080 a good force for the entire ecosystem because it means that you can pack very 01:39:20.370 --> 01:39:22.917 strong intelligence capabilities into smaller and smaller models, 01:39:23.000 --> 01:39:25.550 which enables us to serve them at much wider scale. 01:39:25.880 --> 01:39:28.917 And that's certainly one of the very exciting stories is just 01:39:29.000 --> 01:39:33.547 like really small models today are like, you know, much more powerful than the 01:39:33.630 --> 01:39:35.680 biggest models a year ago or 2 years ago. 01:39:36.800 --> 01:39:41.730 And then I think the last is that there's just sort of this— I think there's like, 01:39:42.230 --> 01:39:48.270 I actually think a lot of people in the ecosystem want this future 01:39:48.360 --> 01:39:51.610 where it is not just this like, you know, definitely. 01:39:51.900 --> 01:39:56.907 One final company in all of capitalism kind of kind of end outcome. 01:39:56.990 --> 01:39:58.367 Like, I think this is one of the reasons— 01:39:58.450 --> 01:40:01.617 Not just individuals, but companies putting dollars against things. 01:40:01.700 --> 01:40:05.707 Yeah. And I think, like, you know, this is actually one of the reasons why 01:40:05.790 --> 01:40:10.127 a lot of people were excited to join Meta and Meta Superintelligence Labs 01:40:10.210 --> 01:40:12.217 in the first place, was that they were just sort of like, 01:40:12.300 --> 01:40:15.047 you know, I want to work at a place that is focused 01:40:15.130 --> 01:40:16.967 on giving this technology to billions of people. 01:40:17.050 --> 01:40:19.470 That is, like, not the stated goal necessarily 01:40:19.680 --> 01:40:20.807 of all the other frontier labs. 01:40:20.890 --> 01:40:23.437 And so, I think there's, like, a lot of sort of just, like, 01:40:23.520 --> 01:40:26.890 natural forces where many people in the industry are sort of, 01:40:26.980 --> 01:40:31.317 like, grappling with, okay, what is the actual world state 01:40:31.400 --> 01:40:32.947 that I want to live in in the future? 01:40:33.030 --> 01:40:37.237 And there's lots of sort of, like, decisions that people make at all levels 01:40:37.320 --> 01:40:40.697 in terms of where they work or who they do business with or what business deals they 01:40:40.780 --> 01:40:45.250 do that sort of, like, drive towards a more equalized outcome. 01:40:46.040 --> 01:40:51.800 You get into the Some of the MSL startup, which I also wanted to ask about, which is 01:40:52.840 --> 01:40:55.880 it just seems fundamentally very hard still to— 01:40:57.050 --> 01:41:01.140 again, we talked about what goes into developing a frontier model, but also 01:41:01.640 --> 01:41:04.890 from a standing start, like when was the standing start for MSL? 01:41:05.930 --> 01:41:08.770 Well, we launched Llama 4 01:41:09.270 --> 01:41:12.980 on April 4th, I think it was, 2025. 01:41:13.650 --> 01:41:19.240 Right. And then I think, what was it, April 20th 01:41:20.410 --> 01:41:22.490 or 21st was, I think, when I was like, 01:41:22.580 --> 01:41:29.420 hey, when I started calling people for advice on what to do. 01:41:29.790 --> 01:41:34.377 So, I remember I was in a staff meeting and I got, like, the WhatsApp 01:41:34.460 --> 01:41:35.927 notification, like, have time for a call? 01:41:36.010 --> 01:41:41.260 I was like, huh, interesting. Okay, so MSL is a little over 2 years old. 01:41:41.760 --> 01:41:45.390 No, no, no, 1 year. 1 year. Is it '25? Oh, it's '26. 01:41:45.640 --> 01:41:47.057 Oh, sorry, I'm living in the future. 01:41:47.140 --> 01:41:50.190 Yeah, you are in the future. MSL is a little over 1 year old. 01:41:50.560 --> 01:41:55.940 Um, yeah, what are your reflections a little over a year in on what was 01:41:56.730 --> 01:42:00.110 load-bearing, as the IAs would say, in how it came together? 01:42:00.490 --> 01:42:04.830 Well, I mean, I think that there was this whole thing early on which is just like, 01:42:05.410 --> 01:42:07.367 first we had to pull together this core group. 01:42:07.450 --> 01:42:07.577 Right? 01:42:07.660 --> 01:42:14.460 So, um, so Nat and Alex and I, we were like talking a bit up front, um, 01:42:15.300 --> 01:42:19.300 and, you know, at first we were, you know, brainstorming. 01:42:19.720 --> 01:42:23.717 Yeah, like I was talking to Nat about, could we— is there a way that we can like 01:42:23.800 --> 01:42:27.057 structure this so that way it's like a win for Alex's shareholders and like 01:42:27.140 --> 01:42:28.640 we can like get him over here? 01:42:29.060 --> 01:42:32.730 And then, so then we were talking about, uh, like what would the plan be? 01:42:33.230 --> 01:42:37.777 And, and then basically I was just, I was in this mode where I was just like 01:42:37.860 --> 01:42:40.107 asking a lot of different people what we should do. 01:42:40.190 --> 01:42:43.780 And like every time that, that we would talk, 01:42:44.820 --> 01:42:48.617 I'd come away with like 3 more things of like, oh yeah, this is what we should do. 01:42:48.700 --> 01:42:52.200 It's like very like high signal, very like a lot of strategic clarity. 01:42:52.670 --> 01:42:56.800 And so we were brainstorming and working on this for a bit. 01:42:57.210 --> 01:42:59.967 And then Nat and Daniel joined and we pulled together a few 01:43:00.050 --> 01:43:01.930 of the the initial researchers. 01:43:02.050 --> 01:43:05.510 But like, in the beginning, it was sort of like the Supernova thing 01:43:05.640 --> 01:43:08.437 where we just kind of like jammed a bunch of people together. 01:43:08.520 --> 01:43:10.977 And I don't know, it was like, I don't think either of us 01:43:11.060 --> 01:43:13.017 slept that much for the first couple of months. 01:43:13.100 --> 01:43:16.690 No. I like, don't even, I kind of have this mental block almost 01:43:16.900 --> 01:43:19.570 where I like, I kind of almost 01:43:20.940 --> 01:43:23.947 cleared my calendar for like a month or 2 01:43:24.030 --> 01:43:28.280 and was just meeting researchers and talking to them. 01:43:28.450 --> 01:43:32.910 And I remember even before we announced the deal where you were going to come 01:43:33.000 --> 01:43:35.877 over, we were basically like, you were recruiting people and people 01:43:35.960 --> 01:43:38.910 were like, what is going on? Like, it's like, what is this? 01:43:39.050 --> 01:43:42.927 But, um, but then there was the whole question, which was a really 01:43:43.010 --> 01:43:45.590 hands-on data labeler. 01:43:46.800 --> 01:43:48.927 But, but that was, that was this whole question 01:43:49.010 --> 01:43:50.557 too, where it's like, all right, well, like, 01:43:50.640 --> 01:43:52.807 okay, so if we're not happy with the trajectory of the current 01:43:52.890 --> 01:43:58.270 program, Like, who were the right set of people to lead the program going forward? 01:43:58.730 --> 01:44:02.237 And Alex and I had spent a bunch of time talking over the years. 01:44:02.320 --> 01:44:05.737 I mean, maybe like, whatever, once every few months or something we'd talk. 01:44:05.820 --> 01:44:09.920 And I was always just very impressed by how he thought about things strategically. 01:44:10.080 --> 01:44:14.870 And I kind of had this view of like, all right, yeah, currently he's doing 01:44:15.620 --> 01:44:18.000 this very operational data business. 01:44:19.960 --> 01:44:26.257 But it always felt to me like he was the sharpest entrepreneur of his 01:44:26.340 --> 01:44:29.930 generation, of the ones who I had had a chance to meet. 01:44:32.310 --> 01:44:34.937 And like, it always just struck me, it's like, all right, like, 01:44:35.020 --> 01:44:37.977 this is a guy who, yeah, he's like running this like kind of very 01:44:38.060 --> 01:44:40.610 grindy operational business, but like— 01:44:40.730 --> 01:44:42.527 Lovely business still, by the way. What's up? 01:44:42.610 --> 01:44:48.777 We're big investors. It's not a— it's a— there's more Commentary on the difficulty 01:44:48.860 --> 01:44:49.737 of running the business. 01:44:49.820 --> 01:44:52.907 When you run the 8th wonder of the world, Alex, a lot of businesses look 01:44:52.990 --> 01:44:54.740 like a grindy operational business. 01:44:56.330 --> 01:44:58.170 It's just like my view of things. 01:44:58.290 --> 01:45:02.920 It's like grindy, grindy. No, no, no. 01:45:04.840 --> 01:45:08.467 But no, so it's, so I just kind of had this view. 01:45:08.550 --> 01:45:11.930 I'm like, all right, like, even though this is sort of 01:45:13.220 --> 01:45:16.270 I think maybe when we got started, a lot of people were like, 01:45:16.560 --> 01:45:18.067 okay, like, how is this going to work? 01:45:18.150 --> 01:45:22.437 It's like the kind of like the social media guy is working 01:45:22.520 --> 01:45:23.940 with the like data guy. 01:45:24.190 --> 01:45:27.157 Like, is that like, are we going to build a frontier lab here or what? 01:45:27.240 --> 01:45:30.157 But, but I mean, but my, but I mean, my like really strong 01:45:30.240 --> 01:45:33.447 conviction, because I talked to like all the leaders of all the labs, like all the 01:45:33.530 --> 01:45:35.367 heads of research, like all this stuff. 01:45:35.450 --> 01:45:38.537 And like, I was like, no, Alex is the guy. Like, he's the fucking guy. 01:45:38.620 --> 01:45:40.820 And like, and we're going to, and basically, 01:45:40.920 --> 01:45:46.260 like, we're going to basically be able to, to, to go pull together a team and recruit 01:45:46.720 --> 01:45:48.547 the people that we need to go do this. 01:45:48.630 --> 01:45:54.387 And like, we need this kind of very, this kind of like strategic clarity 01:45:54.470 --> 01:45:58.020 that he has around the different ways that we want to approach this, 01:45:58.140 --> 01:46:01.060 the kind of extreme focus on talent density, 01:46:01.770 --> 01:46:06.357 on kind of like the clarity of communicating the different vision, 01:46:06.440 --> 01:46:09.090 which we kind of fashioned together over this period. 01:46:10.610 --> 01:46:13.450 And a bunch of things that I think were 01:46:14.240 --> 01:46:17.950 kind of extreme but have basically worked, 01:46:18.200 --> 01:46:23.290 like having this, this like very flat culture was something 01:46:23.380 --> 01:46:25.877 that you were kind of really obsessed with, right? 01:46:25.960 --> 01:46:28.757 It's, I mean, you have like, was it almost 200 direct reports? 01:46:28.840 --> 01:46:30.927 That's like, we have, it's not, it doesn't, it's not quite 01:46:31.010 --> 01:46:32.137 that extreme in practice. 01:46:32.220 --> 01:46:35.137 I think that there's like, like team leads and all, and all this stuff. 01:46:35.220 --> 01:46:36.437 My muse helps me manage that. 01:46:36.520 --> 01:46:41.017 Yeah, well, that's— But like, but there were these series of things 01:46:41.100 --> 01:46:45.980 that I think quite early we, you kind of articulated, and I was like, 01:46:46.770 --> 01:46:53.280 okay, this is like kind of extreme, but it seems like directionally right, 01:46:53.740 --> 01:46:55.077 and like we should go do this. 01:46:55.160 --> 01:46:57.457 And, you know, I went and like talked to other people who were more like 01:46:57.540 --> 01:47:01.667 the people you'd conventionally think of, and it's like, it was just very much like, 01:47:01.750 --> 01:47:04.417 I don't know what you'd expect expect a big company to operate. 01:47:04.500 --> 01:47:07.007 And I was like, no, we want to do something that's— 01:47:07.090 --> 01:47:11.590 As you describe this, I'll note 4 things that I think even other 01:47:12.300 --> 01:47:15.300 opinionated, effective founder CEOs would not have done. 01:47:16.140 --> 01:47:20.640 One, you just called the recent prior effort by 01:47:21.770 --> 01:47:26.187 a bunch of people at the company a failure and not good enough. 01:47:26.270 --> 01:47:29.027 I mean, you were more diplomatic than that, but the act of standing up off 01:47:29.110 --> 01:47:32.527 the trajectory— Yeah, but the act of standing up So, you know, 01:47:32.610 --> 01:47:34.237 people can read between the lines. 01:47:34.320 --> 01:47:36.577 So sometimes, yeah, sometimes you have to reboot a team. 01:47:36.660 --> 01:47:37.493 That's one. 01:47:37.830 --> 01:47:43.540 Thing 2 is you are willing to pay ungodly amounts of money for effectively talent. 01:47:43.830 --> 01:47:47.337 Like, you're paying a lot of money to bring Alex and some other people, but, 01:47:47.420 --> 01:47:50.460 you know, a lot of money for Alex in particular over. 01:47:52.720 --> 01:47:57.387 Thing 3— And 49% of this very good grindy business. 01:47:57.470 --> 01:48:01.020 Yeah, exactly. But let's keep the story straight. 01:48:01.520 --> 01:48:04.190 So there's that. 01:48:04.690 --> 01:48:08.980 Thing 3 is, I'll note, Alex had never trained a frontier model before. 01:48:09.360 --> 01:48:13.527 And so you were taking some amount of a bet on Alex's potential to do 01:48:13.610 --> 01:48:17.320 something where he had done stuff in AI, but no offense, 01:48:17.410 --> 01:48:20.657 he just had not done the specific thing that you were being hired to do. 01:48:20.740 --> 01:48:23.537 And then 4th, you were saying you cleared your calendar 01:48:23.620 --> 01:48:25.327 and just only did this for 2 months. 01:48:25.410 --> 01:48:28.837 I think a lot of CEOs would've had the business review and would have had 01:48:28.920 --> 01:48:32.920 the board thing and they would have had the customer stuff and things like that. 01:48:33.130 --> 01:48:37.387 Those 4 seem to me pretty interesting where I think a lot 01:48:37.470 --> 01:48:40.017 of CEOs would not have done those. Yeah, that's interesting. 01:48:40.100 --> 01:48:42.637 I definitely think— I mean, one thing you may not appreciate this 01:48:42.720 --> 01:48:46.390 about yourself, but I think one thing that is quite special about Marc is 01:48:47.730 --> 01:48:51.357 the degree to which, like, once he concludes that something is 01:48:51.440 --> 01:48:56.150 the right thing or something makes sense, he just like fully shifts everything 01:48:56.240 --> 01:48:58.140 to be consistent with that belief set. 01:48:58.610 --> 01:49:04.540 And it is quite like, to your point, I think like the inertia of most, even 01:49:05.000 --> 01:49:08.417 very successful founders who like are opinionated and edgy or whatnot, 01:49:08.500 --> 01:49:11.750 would just be to like, oh, I still have like my whole calendar. 01:49:11.920 --> 01:49:15.920 And like, so I'll just like talk about doing this thing and hope that somebody 01:49:16.050 --> 01:49:21.097 does it kind of situation. Whereas I think, No, I think when Marc concludes 01:49:21.180 --> 01:49:23.887 that something is the thing he has to do or makes a lot 01:49:23.970 --> 01:49:29.147 of sense, then he really does, like, he has, like, maybe very high 01:49:29.230 --> 01:49:30.227 learning rate in that sense. 01:49:30.310 --> 01:49:34.260 Like, he's very willing to, like, pivot entire things to be consistent with it. 01:49:34.610 --> 01:49:39.947 And I think that, like, you know, one very underappreciated thing 01:49:40.030 --> 01:49:43.447 is the degree to which so much of the company had to pivot 01:49:43.530 --> 01:49:50.207 to support MSL as well. Like, I think that while it was a kind of like this more 01:49:50.290 --> 01:49:55.710 startup-y effort within a larger company, you know, the needs of MSL 01:49:55.800 --> 01:49:58.967 were still quite large. Like, we needed a huge amount of help on infrastructure. 01:49:59.050 --> 01:50:01.347 We needed these large-scale data centers to be built. 01:50:01.430 --> 01:50:05.060 We needed like lots of support from many other parts of the company. 01:50:06.140 --> 01:50:09.850 And especially as Muse has been successful, we've also needed lots of— 01:50:09.980 --> 01:50:12.357 Yeah. Deep collaboration with the rest of the company. 01:50:12.440 --> 01:50:18.530 And this is one thing that I think kind of only is possible because Mark is 01:50:18.940 --> 01:50:23.237 the founder CEO, and he can ensure that we have kind 01:50:23.320 --> 01:50:29.540 of like a global optimization towards this sort of like AI maximalist version of Meta 01:50:30.120 --> 01:50:33.040 versus the natural inertia of an organization towards 01:50:34.040 --> 01:50:37.130 just local optimization ad infinitum. 01:50:37.800 --> 01:50:41.930 But it's also a lot of how we work together is it's important to me to be 01:50:42.390 --> 01:50:46.807 in the details of this work so that we can get everything around the company 01:50:46.890 --> 01:50:48.850 to go do what we need to do. Right. 01:50:48.980 --> 01:50:51.477 And so it's like, I mean, yeah, I'll have opinions about this stuff, but 01:50:51.560 --> 01:50:53.977 I'm not like, I'm not the lead researcher. Right. 01:50:54.060 --> 01:50:58.820 It's like, it's, you know, it's, but I think that's kind of a lot of, 01:50:58.940 --> 01:51:02.907 I think how we work together well is that, yeah, it's, I mean, we're, 01:51:02.990 --> 01:51:05.697 we're kind of like, quite in the details of all of this stuff, 01:51:05.780 --> 01:51:08.660 but running the lab is a big thing. 01:51:08.990 --> 01:51:11.667 Running the rest of the company in order to support this is like 01:51:11.750 --> 01:51:13.250 the effort is a big thing. 01:51:13.670 --> 01:51:16.797 You almost like, you can't really get this done, I think, without 01:51:16.880 --> 01:51:18.000 both of those pieces. 01:51:18.710 --> 01:51:22.427 And one thing that I think was quite important about when we set 01:51:22.510 --> 01:51:27.260 the whole thing up is Meta, the company, invested a huge amount 01:51:27.350 --> 01:51:31.140 to stand up MSL, and then we very clearly told everyone 01:51:31.230 --> 01:51:33.140 like, hey, there's no deadlines. 01:51:33.400 --> 01:51:37.520 We are going to take things at the right pace, which I think on the one hand could 01:51:37.610 --> 01:51:39.777 be like, you know, absolutely terrifying. 01:51:39.860 --> 01:51:42.947 It's like you've just like stood up this extremely expensive unit and you're 01:51:43.030 --> 01:51:46.407 telling everyone like, hey— Yeah, and you're worried inside a larger company 01:51:46.490 --> 01:51:49.367 that like, okay, 7 years from now maybe something will ship, right? 01:51:49.450 --> 01:51:50.283 Yeah, totally. 01:51:50.410 --> 01:51:54.917 And so, but I think that like, first, I think one of the cool things 01:51:55.000 --> 01:51:56.957 about, you know, focusing on talent density and hiring all 01:51:57.040 --> 01:52:01.087 these very brilliant people who have done so much already, that's very impressive, 01:52:01.170 --> 01:52:05.180 is, you know, everybody had like a lot of motivation to ship 01:52:05.640 --> 01:52:09.677 and everybody wanted to sort of like leave their mark, so to speak, on the field. 01:52:09.760 --> 01:52:13.520 And so everybody was like heavily, highly self-motivated 01:52:14.020 --> 01:52:15.570 to move as quickly as possible. 01:52:15.850 --> 01:52:20.230 And one of the things that's been funny and I've had to continue reminding 01:52:20.320 --> 01:52:25.900 the team of is like, you know, it is a team of a lot of insecure overachievers. 01:52:26.200 --> 01:52:29.617 So like at any one point, if I ask them like, hey, 01:52:29.700 --> 01:52:32.850 how good do you think— where do you think we'll be in 3 months? 01:52:33.160 --> 01:52:36.127 I always get the most like— Everyone always says it's awful. It's awful. 01:52:36.210 --> 01:52:38.507 We're screwed. Yeah. These like very pessimistic answers. 01:52:38.590 --> 01:52:42.210 And then I record it and I like 3 months later I'm like, hey, 01:52:42.760 --> 01:52:43.757 you know, we just shipped this model. 01:52:43.840 --> 01:52:47.677 Let me just show you what all of you said, where all of you said we 01:52:47.760 --> 01:52:51.560 would be in 3 3 months ago. And, uh, and this is— 01:52:51.770 --> 01:52:53.897 And has that caused them to get any more optimistic or no? 01:52:53.980 --> 01:52:56.227 No, they're still— they're very self-critical. 01:52:56.310 --> 01:52:59.510 I think it's, it's not, it's not that they're pessimistic about— 01:53:00.110 --> 01:53:04.277 I think that the average person seems quite optimistic about MSL and very 01:53:04.360 --> 01:53:06.777 self-critical about their own project. Yeah. 01:53:06.860 --> 01:53:09.990 Which I think is actually quite a healthy, a healthy dynamic. 01:53:10.780 --> 01:53:12.950 Um, but, but the trend continues. 01:53:13.040 --> 01:53:16.707 Like, I think, I think, and, and this is, I mean, in many ways Muse Wonder is like 01:53:16.790 --> 01:53:21.170 this incredible milestone for everyone because it is a— 01:53:21.290 --> 01:53:23.797 The team was very, very critical. Yeah. 01:53:23.880 --> 01:53:25.920 That was working on it along the way. 01:53:26.130 --> 01:53:28.840 When did you guys know that Muse Wanderer was good? 01:53:31.600 --> 01:53:33.220 Um, that's a good question. 01:53:33.390 --> 01:53:37.140 I think that we, uh, so there's all these intermediate drawings. 01:53:37.230 --> 01:53:39.147 You're peeking in the oven and it's like, I don't know, it seems 01:53:39.230 --> 01:53:40.187 kind of gooey still. 01:53:40.270 --> 01:53:42.987 And yeah, yeah, going back to it, I mean, there are all these intermediate 01:53:43.070 --> 01:53:46.657 intermediate signals that you can get along the way that things are good or bad. 01:53:46.740 --> 01:53:48.860 And what are the intermediate signals? 01:53:49.530 --> 01:53:53.240 So in pre-training, you have a lot of perplexity-based evaluation. 01:53:53.370 --> 01:53:57.660 So you have a different regime of evals that can help you in pre-training. 01:53:57.750 --> 01:54:01.840 And then you have the more traditional benchmarks that can help you towards RL. 01:54:03.250 --> 01:54:06.007 But no, I mean, this model, 01:54:06.090 --> 01:54:08.847 kind of as you mentioned, has been cooking for quite a while. 01:54:08.930 --> 01:54:12.637 And You know, a lot of— I think one of the funniest 01:54:12.720 --> 01:54:18.730 things about AI research is that, you know, there's a lot of times where 01:54:18.850 --> 01:54:22.397 people just sort of like are kind of squinting at a bunch of dots and being 01:54:22.480 --> 01:54:27.357 like, yeah, looks like it's good, or, uh, looks a little bit off. 01:54:27.440 --> 01:54:31.860 Like, one of the funniest things is the people have protractors, 01:54:33.490 --> 01:54:38.827 like analog protractors that they put against you know, a set of lines on their 01:54:38.910 --> 01:54:40.870 monitor to like measure an angle. 01:54:41.210 --> 01:54:45.250 And then they, um, so he's not come for this job yet. 01:54:45.340 --> 01:54:47.507 So I think there's, I think there's like a huge amount 01:54:47.590 --> 01:54:52.677 of like, uh, like it's very, um, there's more vibes, um, driving like, okay. 01:54:52.760 --> 01:54:56.477 But when did we have this conversation where, when did you know it was good? 01:54:56.560 --> 01:55:01.980 Um, well, we had this debate after it was done pre-training where it was okay. 01:55:03.690 --> 01:55:09.820 How long will it take in RL before it is better than Muse Spark? 01:55:10.650 --> 01:55:13.450 And I remember the folks on the pre-training team 01:55:14.070 --> 01:55:16.200 were just trying to set expectations. 01:55:16.330 --> 01:55:18.830 They're like, look, sometimes this takes a while. 01:55:19.370 --> 01:55:20.770 And then it was immediately. 01:55:21.250 --> 01:55:24.540 And so we were like, okay, I think we're on a good path. 01:55:25.170 --> 01:55:29.170 That to me was sort of—Yeah. I think when when people got— yeah. 01:55:29.260 --> 01:55:30.420 We've been excited. 01:55:30.670 --> 01:55:32.717 I think we've been pretty excited about it for a few months now. 01:55:32.800 --> 01:55:34.600 Yeah. But there's a lot to do there. 01:55:35.470 --> 01:55:39.930 But going back to the beginning, I think one of the things that was very 01:55:41.270 --> 01:55:44.370 interesting and challenging in standing the thing up was like, 01:55:44.560 --> 01:55:47.940 you really just wanted this like very high talent density effort. 01:55:49.020 --> 01:55:53.360 And we were just kind of pulling some of the best people from Meta in and 01:55:53.650 --> 01:55:55.327 recruiting a lot of the best people. 01:55:55.410 --> 01:55:57.737 And I mean, you were asking, it's like, all right, well, like, okay, 01:55:57.820 --> 01:56:00.990 for people who, like, if you were asking before, it's like, 01:56:01.080 --> 01:56:04.580 well, Alex hadn't necessarily trained models like this. 01:56:04.750 --> 01:56:08.290 But to me, one of the big things was like all of the best people were 01:56:08.380 --> 01:56:09.880 very excited to work with him. 01:56:10.460 --> 01:56:12.297 And I think like, it's, I mean, you, 01:56:12.380 --> 01:56:16.217 I'm sure see this in all of, in a lot of the things that you do, 01:56:16.300 --> 01:56:19.807 which is like, you know, it's like you, most people 01:56:19.890 --> 01:56:22.390 are not kind of one-dimensional, right? 01:56:22.980 --> 01:56:25.100 Yeah, maybe you do a specific thing, 01:56:27.310 --> 01:56:30.690 but your ability to grow and expand is 01:56:30.900 --> 01:56:36.947 kind of largely based on how effectively can you just do people who are excellent 01:56:37.030 --> 01:56:40.530 in different domains want to come work with you on different things. 01:56:40.740 --> 01:56:42.497 Yeah, and the great people have followership. 01:56:42.580 --> 01:56:46.040 Yeah. And so I think that there was a little bit of that, 01:56:47.830 --> 01:56:52.550 but also just It's even for the people that don't know you yet, 01:56:53.130 --> 01:56:56.880 or like, or aren't, you know, don't know what you're about. 01:56:56.970 --> 01:57:00.177 Like, are you like an impressive person and a compelling 01:57:00.260 --> 01:57:02.360 person that people want to come work with? 01:57:02.890 --> 01:57:06.810 Do people feel comfortable that you're going to create this environment? 01:57:06.980 --> 01:57:10.897 And I think you were like very thoughtful about this early on. 01:57:10.980 --> 01:57:14.737 We were kind of— Some of the stuff I think you kind of had 01:57:14.820 --> 01:57:17.870 fully formed as beliefs of how you thought we should do this. 01:57:18.240 --> 01:57:21.657 But I also remember when we were having the initial recruiting 01:57:21.740 --> 01:57:24.410 conversations, we were sort of in real time 01:57:25.080 --> 01:57:28.457 shaping— it's like, okay, well, this construct that we're talking about 01:57:28.540 --> 01:57:32.167 with this person isn't quite going to work, but like, they seem good, 01:57:32.250 --> 01:57:33.127 and like, they have this thought. 01:57:33.210 --> 01:57:36.050 And it was just like, like, really kind of coalescing 01:57:36.510 --> 01:57:38.637 this vision about what the shape of it should be in real time. 01:57:38.720 --> 01:57:40.467 I mean, I don't know, I mean, I'm sure you remember that. 01:57:40.550 --> 01:57:44.520 this, where it was like there were some of the initial concepts of like, 01:57:45.100 --> 01:57:50.400 I mean, I think probably the most extreme thing was I think you had the concept of 01:57:50.900 --> 01:57:54.150 the talent density and all and that kind of piece. 01:57:54.690 --> 01:57:58.150 But I think we weren't sure upfront that it was going to have 01:57:58.240 --> 01:58:00.110 to be a full reboot, right? 01:58:00.280 --> 01:58:04.790 It's like that basically started when we were out talking to 01:58:05.410 --> 01:58:07.330 a lot of the kind of best people. 01:58:07.540 --> 01:58:11.247 And we realized that, hey, like, in order to make this as attractive 01:58:11.330 --> 01:58:14.300 as possible, it should feel like, okay, we're starting this 01:58:14.460 --> 01:58:15.710 thing again from scratch. 01:58:15.800 --> 01:58:18.757 And yeah, we'll pull in a lot of the great people from the Llama effort 01:58:18.840 --> 01:58:20.340 and FAIR and all of that too. 01:58:21.010 --> 01:58:25.307 But that was a very intentional thing that we basically figured out and that you 01:58:25.390 --> 01:58:28.897 figured out as we were out talking to the initial set of people of like 01:58:28.980 --> 01:58:30.727 what was going to make this compelling. 01:58:30.810 --> 01:58:34.477 And it ended up being really smart and like making a huge difference 01:58:34.560 --> 01:58:37.737 in the culture of the lab and how it got and how it got started. 01:58:37.820 --> 01:58:40.527 But I don't know, it's like being able to be that dynamic 01:58:40.610 --> 01:58:43.860 about figuring out, like, what does the shape of this need to be? 01:58:44.280 --> 01:58:47.290 It's not like that. I think that's a very uncommon trait. 01:58:47.450 --> 01:58:49.117 It's a very entrepreneurial trait, right? 01:58:49.200 --> 01:58:52.620 So, when I was thinking about, like, all right, like, this field is 01:58:52.710 --> 01:58:54.257 going to keep on evolving, right? 01:58:54.340 --> 01:58:56.877 So, what is the thing that's going to be important for someone who's 01:58:56.960 --> 01:59:01.967 going to run a lab well? It's not like, are they an expert in whatever the latest 01:59:02.050 --> 01:59:06.850 modality is of, like, or kind of of training technique for a lab. 01:59:07.390 --> 01:59:09.600 It's, is the person a clear 01:59:11.140 --> 01:59:14.440 thinker and able to kind of dynamically 01:59:14.810 --> 01:59:18.777 update and move in, like, as kind of an entrepreneur would? 01:59:18.860 --> 01:59:21.907 I think this is just like an inherently very entrepreneurial space. 01:59:21.990 --> 01:59:26.530 Has this experience changed your view on the value of 01:59:30.200 --> 01:59:34.080 going external, where both your reboot, 01:59:34.460 --> 01:59:38.050 a bunch of the people you brought in were external, like Alex and 01:59:38.500 --> 01:59:39.840 Nat and Daniel and folks. 01:59:40.340 --> 01:59:44.340 But then also you're describing deciding whether you needed a reboot 01:59:44.970 --> 01:59:47.970 based on going to the outside 01:59:49.140 --> 01:59:52.020 intelligence of the aggregate views 01:59:52.520 --> 01:59:55.647 of the researchers in the industry and getting a sense for the— But just Do 01:59:55.730 --> 02:00:01.317 you feel like you'll carry forward that approach for future unrelated projects? 02:00:01.400 --> 02:00:04.067 Because obviously the risk of a large company is that everyone's like, 02:00:04.150 --> 02:00:08.100 oh, we got a great approach. Yeah, let me show you our 100-page document on it. 02:00:08.330 --> 02:00:14.870 I mean, obviously you can't— a lot of what we had to do here was 02:00:15.120 --> 02:00:16.667 rotate a lot of the rest of the company. 02:00:16.750 --> 02:00:19.627 And like for things like the infrastructure team that now, 02:00:19.710 --> 02:00:22.007 you know, it's like building these huge clusters to support this. 02:00:22.090 --> 02:00:23.990 I mean, that's obviously a huge thing. 02:00:25.130 --> 02:00:28.260 So huge parts of the company kind of had to pivot to do this. 02:00:28.430 --> 02:00:32.557 And obviously you can't do too many of those at the same time, right? 02:00:32.640 --> 02:00:34.840 Only one thing can be the top priority. Yes. 02:00:35.190 --> 02:00:39.060 But I think on balance, it has made me somewhat more optimistic. 02:00:39.150 --> 02:00:43.107 I mean, we brought in this guy Kunal Shah, who, you know, it's like, 02:00:43.190 --> 02:00:45.990 I think he's like one of the top entrepreneurs in India, 02:00:46.150 --> 02:00:48.987 building this great company CRED, and And now he's running WhatsApp, 02:00:49.070 --> 02:00:51.570 which I always thought should do more in commerce, 02:00:51.700 --> 02:00:54.787 and we were sort of underserved there, and India is the biggest 02:00:54.870 --> 02:00:56.210 country that we're in. 02:00:57.670 --> 02:00:59.797 But, I mean, that's— it's a bit of a different 02:00:59.880 --> 02:01:02.837 situation, because we're not, like, pivoting 02:01:02.920 --> 02:01:04.507 the whole company towards WhatsApp. 02:01:04.590 --> 02:01:07.387 We just wanted to bring more entrepreneurial energy there. 02:01:07.470 --> 02:01:09.677 This was, like, a level 10 out of 10 difficulty thing. 02:01:09.760 --> 02:01:12.137 I don't want to talk about it in the past tense, as if it's done, because it's still 02:01:12.220 --> 02:01:14.057 a level 10 out of 10 difficulty thing. Yeah, yeah, sure. 02:01:14.140 --> 02:01:16.890 But I think 02:01:18.100 --> 02:01:21.817 the number of pieces that kind of had 02:01:21.900 --> 02:01:27.570 to work, it was quite all-encompassing. 02:01:27.820 --> 02:01:30.697 And I think that it's both difficult for the people doing it, 02:01:30.780 --> 02:01:33.700 but I do think that that also puts some level of strain on 02:01:34.490 --> 02:01:37.037 the rest of the company, on the financial model of the company. 02:01:37.120 --> 02:01:40.320 I think like you— so I wouldn't want to conflate those 2 pieces. 02:01:40.540 --> 02:01:43.587 Right? It's like I think there's yeah, it's good to have new blood, and it's good 02:01:43.670 --> 02:01:46.547 to have people who understand the context. 02:01:46.630 --> 02:01:50.637 And you know, you were talking about people who've been around Meta for a while 02:01:50.720 --> 02:01:52.057 and understand the ups and downs. 02:01:52.140 --> 02:01:54.477 And it's like I don't know if you've worked at Meta for 10 years, 02:01:54.560 --> 02:01:55.897 you've seen some shit, right? 02:01:55.980 --> 02:02:00.897 It's like it's like we've gone through a lot of hard times, and and like I think 02:02:00.980 --> 02:02:03.080 that that's kind of important context too. 02:02:03.190 --> 02:02:07.070 But but no, I I think a lot of this is just. 02:02:08.240 --> 02:02:11.907 trying to, you know, you're— whenever you're building 02:02:11.990 --> 02:02:16.617 a team, you're— you inherently, you know, it's like Alex had a theory for what 02:02:16.700 --> 02:02:20.667 the team shape needed to be, and like, I guess I had a theory for what 02:02:20.750 --> 02:02:23.670 like the shape of the leaders needed to be for it. 02:02:23.960 --> 02:02:29.180 And, um, and, you know, it's like a year plus in, it like seems okay. 02:02:29.340 --> 02:02:32.217 Like, it's like, first of all, it was just good to try something different, right? 02:02:32.300 --> 02:02:34.177 Mm-hmm. And then different than what everyone else was doing. 02:02:34.260 --> 02:02:41.187 Yes. And yeah, I mean, that's— 02:02:41.270 --> 02:02:45.777 I think there's some other dimensions here that are worth mentioning too. 02:02:45.860 --> 02:02:49.150 Like, one of the— like, when deciding we had to do the full 02:02:49.240 --> 02:02:54.197 reboot, like, you know, a lot of it was, you know, one of the main inputs was 02:02:54.280 --> 02:02:56.747 external, but also one of the inputs— I don't think I've ever— 02:02:56.830 --> 02:02:59.667 actually, I can't remember if I told you this story, but one of the first meetings 02:02:59.750 --> 02:03:05.587 I had When I joined, I was in this meeting with some of the leads 02:03:05.670 --> 02:03:06.820 of the existing effort. 02:03:07.210 --> 02:03:10.297 And I was like, okay, I think the first thing we need to do is 02:03:10.380 --> 02:03:15.140 train a really good reasoning model by some timeframe. 02:03:15.260 --> 02:03:18.770 I don't know when, but this is clearly the most important milestone. 02:03:19.270 --> 02:03:23.140 And one of the key leads in the group 02:03:24.060 --> 02:03:29.650 who no longer works at Meta, but was like, oh, we don't need a reasoning model. 02:03:29.820 --> 02:03:32.570 Do any of our consumers need a reasoning model? 02:03:33.320 --> 02:03:34.697 I feel like we don't need to do a reasoning model. 02:03:34.780 --> 02:03:36.117 We could just do a normal model. 02:03:36.200 --> 02:03:39.370 And I had this heart-sinking moment. 02:03:39.700 --> 02:03:41.790 Oh, no, I've joined the wrong company. 02:03:42.370 --> 02:03:45.460 I was like, oh, I've got to take 10 steps back. 02:03:46.040 --> 02:03:49.170 And I think this is kind of a funny anecdote. 02:03:49.300 --> 02:03:50.837 I remember calling Nat after this meeting. 02:03:50.920 --> 02:03:55.180 I was like, Nat, like, this is what they just said in the meeting. 02:03:55.340 --> 02:04:00.680 But I think it was specifically an indicator of something that I think, 02:04:01.100 --> 02:04:04.397 like, Mark, when he talked to people, like, certainly also started to realize, 02:04:04.480 --> 02:04:08.067 which is like, we needed leadership in the company to be 02:04:08.150 --> 02:04:11.607 much more AI-pilled and much more sort of like forward-leaning 02:04:11.690 --> 02:04:15.527 and front-leaning on like where the— how quickly the technology's moving, 02:04:15.610 --> 02:04:18.077 where it's going to go to, what the possibilities are. 02:04:18.160 --> 02:04:21.580 And effectively dramatically more AI-pilled. 02:04:22.080 --> 02:04:27.957 And I think as Marc started pivoting and shifting 02:04:28.040 --> 02:04:32.140 the whole company towards this direction, and it was very clear that this is like, 02:04:33.010 --> 02:04:36.340 I think Marc got AI-pilled, and then everyone saw that he was 02:04:36.430 --> 02:04:39.847 AI-pilled and everyone realized, like, oh, this is where the company's going. 02:04:39.930 --> 02:04:45.230 Then I think you had a lot of the existing leaders like very quickly 02:04:45.730 --> 02:04:50.230 pick it up and change. And like Adam Mosseri, for example, who runs Instagram, 02:04:50.570 --> 02:04:56.317 has one of the most like aesthetic but also like effective kind 02:04:56.400 --> 02:05:01.077 of like personal vibe code and personal mini app setup of anyone that I've seen. 02:05:01.160 --> 02:05:04.870 Like, I think it's like he's like really gone quite far in the direction. 02:05:05.330 --> 02:05:09.167 Mm-hmm. And I think a lot of the leaders at Meta who have been at Meta a long 02:05:09.250 --> 02:05:11.667 time are, like, all very adaptive. 02:05:11.750 --> 02:05:15.007 Like, I think, you know, kind of to the prior point, they've been in— Yeah. 02:05:15.090 --> 02:05:18.180 A highly turbulent, like, period of technology. 02:05:18.390 --> 02:05:22.220 Now, AI is, like, maybe, you know, 5 or 10 times more turbulent 02:05:22.310 --> 02:05:24.307 and faster moving than those past waves. 02:05:24.390 --> 02:05:28.100 But I think they're, like, they have adapted to new trends many times. 02:05:28.190 --> 02:05:31.647 And so I think there's, like, I've been very impressed, actually, 02:05:31.730 --> 02:05:35.780 by seeing how quickly a lot of Meta leadership 02:05:36.990 --> 02:05:40.287 If I compare the kind of conversations I would have with them when I first joined 02:05:40.370 --> 02:05:43.077 versus the kind of conversations I have now— They're now super AI-built. Yeah. 02:05:43.160 --> 02:05:47.620 And it's very impressive, honestly, for that to happen at the large scale 02:05:47.870 --> 02:05:50.870 of Meta, which is one of the largest companies in the world. 02:05:51.670 --> 02:05:53.297 Yeah, we got them there pretty quickly. 02:05:53.380 --> 02:05:56.177 It was, yeah, but I mean, like you're saying, it's like if 02:05:56.260 --> 02:06:00.180 you start coding, then it's helpful. 02:06:01.760 --> 02:06:05.680 But I think a big part of what, a big part of the culture that we were 02:06:05.770 --> 02:06:11.850 trying to create is also, you know, when we were talking about how you have 02:06:11.940 --> 02:06:15.570 all these direct reports, the point of it isn't that 02:06:16.650 --> 02:06:20.360 you have the direct reports, it's that there aren't, it's the absence 02:06:20.450 --> 02:06:22.737 of non-technical managers, right? 02:06:22.820 --> 02:06:25.157 So, so I don't think you would say that you're 02:06:25.240 --> 02:06:28.660 like, you know, that you're providing a ton of managerial support 02:06:28.750 --> 02:06:31.497 to 200 people at once. I mean, that's not a feature of the thing. 02:06:31.580 --> 02:06:36.010 Alex, well known for— Career chats and emotional, how are you doing, checking. 02:06:36.510 --> 02:06:39.840 But I think it's just well known among researchers that, you know, 02:06:39.930 --> 02:06:40.967 there's this common pattern. 02:06:41.050 --> 02:06:45.600 It's like you go from being very technical to you go to a role where you're, 02:06:46.020 --> 02:06:50.310 you're not like in the details anymore, and then you very quickly become not 02:06:50.440 --> 02:06:54.860 technical or don't kind of understand it much faster than you probably realize. 02:06:55.020 --> 02:06:58.470 And it's quite dangerous. It's such a fast-moving domain, especially. 02:06:59.110 --> 02:07:01.360 I think that, like, this whole thing of just, 02:07:01.450 --> 02:07:04.327 like, you have to stay technical, people have to use the agents, 02:07:04.410 --> 02:07:06.330 people should be writing code. Yeah. 02:07:06.780 --> 02:07:10.537 I think that that's— that has been a really important part of this whole thing. 02:07:10.620 --> 02:07:14.120 Have you had any very funny disagreements between the two of you guys? 02:07:14.250 --> 02:07:15.547 We have a lot of disagreements. 02:07:15.630 --> 02:07:18.177 Yeah, I mean, we're working on naming up until very recently. 02:07:18.260 --> 02:07:21.717 Naming is a good bike. It's one of the things I appreciate about Alex. 02:07:21.800 --> 02:07:23.967 It's like one of his— I mean, what this is like, 02:07:24.050 --> 02:07:26.930 I think it goes back to your your philosophy, 02:07:27.310 --> 02:07:29.937 you're talking about this for recruiting back at Scale, is like hire 02:07:30.020 --> 02:07:31.397 people who really care, right? 02:07:31.480 --> 02:07:34.150 I think there's some expletives in it, but it's fine. 02:07:34.400 --> 02:07:36.150 But I— 02:07:36.770 --> 02:07:38.980 But Alex really cares, it turns out. 02:07:39.070 --> 02:07:42.777 I think that's like one of the things that I thought was, you know, I think 02:07:42.860 --> 02:07:44.407 that that's like really important, right? 02:07:44.490 --> 02:07:49.367 It's, so I think part of what you kind of get when you like have 02:07:49.450 --> 02:07:53.917 the supernova of like all these different talented but very strong-willed people 02:07:54.000 --> 02:07:57.090 together is like, you're going to have like fun fights, um, 02:07:57.380 --> 02:08:00.337 and not fun fights, but like, but it's like to some degree I'd rather 02:08:00.420 --> 02:08:04.630 have people push for things that they believe in and like argue over things 02:08:04.760 --> 02:08:07.387 than, you know, just have people think that they're like blindly 02:08:07.470 --> 02:08:08.767 executing something, you know. 02:08:08.850 --> 02:08:11.147 It's like people need to feel ownership. So yeah. 02:08:11.230 --> 02:08:13.580 How is Mark a different boss than you expected? 02:08:14.560 --> 02:08:16.897 Um, well, I don't have a huge comparison set. 02:08:16.980 --> 02:08:18.817 I think I can leave now. Yeah, exactly. 02:08:18.900 --> 02:08:23.030 Yeah, no, it's No, I think that like the— 02:08:23.900 --> 02:08:26.530 I think it has been very— 02:08:28.950 --> 02:08:33.040 one of the very cool things about like working with Mark over the past year 02:08:33.410 --> 02:08:38.290 has just been watching like the true degree of his boldness 02:08:38.540 --> 02:08:41.877 and like, and this trait that we were talking about before of like, you know, 02:08:41.960 --> 02:08:48.637 once he internalizes certain beliefs, like the— speed and magnitude of willing 02:08:48.720 --> 02:08:55.350 to change prior actions and pivot towards the worldview that is 02:08:55.440 --> 02:08:56.767 consistent with those beliefs. 02:08:56.850 --> 02:09:02.070 I think these are pretty unusual traits, generally speaking, even 02:09:02.780 --> 02:09:04.737 among this base of founders. 02:09:04.820 --> 02:09:08.277 Mark actually ingests new information and fully updates his worldview and then 02:09:08.360 --> 02:09:10.747 acts in accordance with the newly installed worldview. 02:09:10.830 --> 02:09:14.000 Very Bayesian. And then willing to take— Very Bayesian. 02:09:14.200 --> 02:09:17.000 Very Bayesian. I've never gotten such a nice compliment. 02:09:17.540 --> 02:09:20.340 What lovely priors you have. 02:09:21.000 --> 02:09:25.587 Yeah. Posterior. You're not allowed to talk about posteriors anymore after 02:09:25.670 --> 02:09:27.050 the whole incident. Yeah. 02:09:27.680 --> 02:09:31.560 No, but then I think also just the degree of boldness. 02:09:31.760 --> 02:09:35.267 And I think that, like, the— this is— it was, like, 02:09:35.350 --> 02:09:38.727 actually quite puzzling to me at first, the degree to which you were, like, 02:09:38.810 --> 02:09:40.357 willing to make these massive bets. 02:09:40.440 --> 02:09:43.147 It was something actually Nat and I would often, like, talk about. 02:09:43.230 --> 02:09:45.860 I was like, Wow, we're really going big here. 02:09:46.860 --> 02:09:49.460 And both of us would be like, would you go this big? 02:09:49.660 --> 02:09:53.410 Would you go this big? And so I think that this, I think it's like, I mean, 02:09:53.790 --> 02:09:58.120 that is quite a notable trait and something I really admire. 02:09:58.870 --> 02:10:03.380 And I think is very, I actually, if I 02:10:04.090 --> 02:10:06.420 look back at the history of Meta and 02:10:07.010 --> 02:10:11.390 I imagine what the scenario would've been in all these pivotal moments, 02:10:11.800 --> 02:10:15.220 I can see how like this is kind of the movie, so to speak. 02:10:15.310 --> 02:10:19.017 Yeah. Or like this is kind of how a lot of those situations go where it's 02:10:19.100 --> 02:10:22.520 like there's— I feel like probably 02:10:23.860 --> 02:10:26.190 the way I imagine is like there clearly is 02:10:26.280 --> 02:10:29.130 moments where like something very, very big is happening. 02:10:30.360 --> 02:10:34.697 And it may not even really be showing in the metrics or it may not be impacting 02:10:34.780 --> 02:10:37.630 user counts or impacting revenue or whatever it might be. 02:10:37.750 --> 02:10:39.247 It's not really impacting things. 02:10:39.330 --> 02:10:41.997 But clearly, if you believe that thing is going to happen, 02:10:42.080 --> 02:10:45.330 then it has massive implications for the company and the product. 02:10:47.340 --> 02:10:52.257 And I think my post hoc retelling of Meta's history is 02:10:52.340 --> 02:10:56.220 that I think Meta is a company where every time that happens, Meta takes 02:10:56.310 --> 02:10:58.010 it extremely, extremely seriously. 02:10:58.270 --> 02:11:01.810 And that has enabled the company to innovate and continue evolving 02:11:02.600 --> 02:11:05.450 with a very fast-moving and fast-changing tech landscape. 02:11:06.520 --> 02:11:08.990 Mark is very Bayesian. Yeah, very Bayesian. 02:11:09.190 --> 02:11:12.610 So Bayesian. Life is short, just do the right thing. 02:11:13.200 --> 02:11:16.990 Yeah. And I think AI is, I mean, I think AI is maybe that, like, 02:11:17.160 --> 02:11:18.207 you know, times 100 or whatnot. 02:11:18.290 --> 02:11:23.540 But, like, I think the, you know, a lot of CEOs 02:11:24.210 --> 02:11:27.710 would've told you last, you know, last summer, oh yeah, I think AI is 02:11:27.800 --> 02:11:29.127 the most important thing for my business. 02:11:29.210 --> 02:11:32.757 Like, I think maybe every CEO would say that, But then you look at, like, 02:11:32.840 --> 02:11:34.887 what they're actually doing and how consistent are their 02:11:34.970 --> 02:11:36.220 actions with that belief. 02:11:36.470 --> 02:11:41.980 And I think this is, like, a, like, quite, quite special thing about Meta and Mark. 02:11:42.980 --> 02:11:44.687 Last question, because we're past all of our bedtimes. 02:11:44.770 --> 02:11:46.060 I'm so flattered. Yeah. 02:11:47.150 --> 02:11:50.250 What do you make of all of Alex's deranged posting on Twitter? 02:11:50.900 --> 02:11:56.570 I mean, I think most of it is great, but, but I, I guess my main view— 02:11:56.780 --> 02:12:00.540 But it doesn't matter how How much you fail, it matters how win— 02:12:00.660 --> 02:12:01.827 how big you win when you win. 02:12:01.910 --> 02:12:06.457 I think with— so I think with communication, I think it's 02:12:06.540 --> 02:12:08.340 most important to just be authentic. 02:12:08.710 --> 02:12:11.717 And I think when you're trying to be authentic, some of the things 02:12:11.800 --> 02:12:13.347 don't come out the way you want. 02:12:13.430 --> 02:12:18.100 And I think that that is a— that it is better to make the trade in that direction 02:12:18.560 --> 02:12:22.850 than it is to be overly scripted and not be authentic. 02:12:22.940 --> 02:12:25.390 But I mean, most of it, I just feel is hilarious. 02:12:25.560 --> 02:12:29.520 So, yeah, what do you think about my deranged tweets? 02:12:30.150 --> 02:12:32.900 I'm not a heavy X user, but— 02:12:33.570 --> 02:12:36.700 But you are back on X now. I'm back to posting. 02:12:36.820 --> 02:12:39.287 I mean, it's like a bunch of the researchers are there and we want 02:12:39.370 --> 02:12:42.820 to be over there, but everything I post there, I also post on Threads 02:12:43.580 --> 02:12:45.370 and it's all good. 02:12:45.460 --> 02:12:50.380 But every time I load X, I have like 4 Alex posts and it just like cracks me up. 02:12:50.500 --> 02:12:52.377 It's like some of these, Like, it's like, where do you— we're 02:12:52.460 --> 02:12:53.547 just talking about this. 02:12:53.630 --> 02:12:55.967 Well, you're now a clearinghouse, I presume, where people send you good stuff. 02:12:56.050 --> 02:12:57.887 I was like, where do you come up with this stuff? 02:12:57.970 --> 02:13:00.850 It's like, yeah, this is—yeah. How does it work? 02:13:01.600 --> 02:13:04.727 Um, most of the— like, uh, we were, we were talking about this with my partner 02:13:04.810 --> 02:13:09.357 yesterday, and he was, you know, I think one observation was just, um, 02:13:09.440 --> 02:13:13.987 there was actually probably still to this moment, I— my, my, uh, thought pattern 02:13:14.070 --> 02:13:15.777 oftentimes is, oh, that's funny. 02:13:15.860 --> 02:13:17.907 I wonder what Jolly would think about that. 02:13:17.990 --> 02:13:21.030 And then, you know, you get good memes out of that. 02:13:21.200 --> 02:13:25.870 That being said, I have experienced the sort of like brutal 02:13:26.410 --> 02:13:30.170 metabolism of the internet and how quickly it like metabolizes memes. 02:13:30.540 --> 02:13:34.630 And so the sort of like, you know, you really have to stay 02:13:35.840 --> 02:13:40.390 on the cutting edge of memeing to— It's easy to fall behind. Yeah. 02:13:40.640 --> 02:13:43.190 Which is maybe a good summary of this conversation. 02:13:44.310 --> 02:13:47.057 It's easy to fall behind, you know, you gotta stay at the frontier. 02:13:47.140 --> 02:13:50.600 Stay in the details. Yeah, stay technical and stay at the frontier. 02:13:50.690 --> 02:13:52.937 Yeah. Yeah. Alrighty, we'll leave it there. All right. 02:13:53.020 --> 02:13:57.450 Awesome. Thanks, guys. Thank you. Okay, wait, so what is everyone drinking? 02:13:57.610 --> 02:14:03.410 Well, that. What is this? That's a keg of Guinness that I brought you. 02:14:03.950 --> 02:14:05.787 This is Guinness from? Dublin. 02:14:05.870 --> 02:14:08.707 Oh, wow. You were just there? Yeah, I was just there this week. Oh, awesome. 02:14:08.790 --> 02:14:11.507 Yeah. Well, thank you. We got the local stuff. Yeah, there you go. 02:14:11.590 --> 02:14:15.217 Yeah, no, I figured this would be a good occasion to Get the Guinness going. 02:14:15.300 --> 02:14:16.133 Yeah.