1 00:00:00,000 --> 00:00:04,750 At the end of last year, we're like, all right, we are quite 2 00:00:05,130 --> 00:00:06,880 We haven't seen the models yet. 3 00:00:07,010 --> 00:00:08,927 You know, we're going to write this check for like, I don't know, 4 00:00:09,010 --> 00:00:11,507 like $100 billion, $200 billion. 5 00:00:11,590 --> 00:00:14,640 I'm like, Alex, are you sure this is going to work? 6 00:00:15,140 --> 00:00:16,640 It's like—Cheers. 7 00:00:17,600 --> 00:00:21,810 Cheers. Yeah. Cheers. Cheers. Cheers. 8 00:00:24,520 --> 00:00:27,360 So, give me the update on Muse. 9 00:00:27,480 --> 00:00:31,700 If you were looking at the third-party reports, 10 00:00:32,070 --> 00:00:38,580 you would see that this might be the fastest-growing agent or AI app 11 00:00:39,290 --> 00:00:40,790 that I think has existed yet. 12 00:00:41,540 --> 00:00:46,210 What are your guys' personal favorite Muse use cases, either yours or other people's? 13 00:00:47,510 --> 00:00:49,257 I was actually talking about this at dinner yesterday. 14 00:00:49,340 --> 00:00:52,720 My favorite thing is I have my Muse, whose name is Oiler, 15 00:00:53,760 --> 00:00:57,270 constantly source increasingly obscure teas 16 00:00:57,770 --> 00:01:01,980 from increasingly obscure vendors of tea. 17 00:01:02,100 --> 00:01:06,980 So at this point he's, on a weekly basis, my Muse is buying 18 00:01:07,400 --> 00:01:14,070 some random tea from some farm in Taiwan or some farm in Japan 19 00:01:14,360 --> 00:01:16,460 and sourcing it over and bringing it back. 20 00:01:16,660 --> 00:01:18,410 What's your favorite Muse use case? 21 00:01:18,830 --> 00:01:21,330 I don't know. I mean, I use it for a ton of stuff. 22 00:01:21,500 --> 00:01:22,333 I still like... 23 00:01:23,000 --> 00:01:26,170 My favorite both use case and probably eval for the product 24 00:01:26,590 --> 00:01:28,340 is having it coach me in MMA. 25 00:01:28,710 --> 00:01:34,340 So I put cameras in my gym, and I basically... 26 00:01:35,220 --> 00:01:38,727 It watches the cameras when I'm in there, and it gives me feedback in between 27 00:01:38,810 --> 00:01:42,147 the rounds, which is a very hard problem, it turns out, because not 28 00:01:42,230 --> 00:01:43,600 only does it have to... 29 00:01:43,940 --> 00:01:50,817 It basically has to process all the video quickly enough because you only 30 00:01:50,900 --> 00:01:52,607 get 1 minute in between the rounds. 31 00:01:52,690 --> 00:01:54,197 Oh, it coaches you between the rounds? Yeah. 32 00:01:54,280 --> 00:01:56,537 I want feedback. I want feedback on how to win. 33 00:01:56,620 --> 00:01:57,617 You know what I mean? It's like, come on. 34 00:01:57,700 --> 00:01:58,907 Did you have to instruct it... 35 00:01:58,990 --> 00:02:02,120 I've wondered about this too, because looking at videos is 36 00:02:02,620 --> 00:02:03,920 a really good AI use case. 37 00:02:04,580 --> 00:02:07,837 It feels like you can't just give it a multi-gigabyte video file. 38 00:02:07,920 --> 00:02:10,320 So do you tell it how to slice stills out of it? 39 00:02:10,630 --> 00:02:12,730 No, it basically... Yeah, it pulls stills. 40 00:02:12,920 --> 00:02:15,220 I think it kind of figured that out by itself. 41 00:02:15,340 --> 00:02:20,850 Um, it turns out that Muse has pretty good native understanding 42 00:02:21,100 --> 00:02:25,230 of MMA, so that's good. So like the multi-modal— The pre-training. 43 00:02:25,600 --> 00:02:30,857 Yeah, someone on the team knew. Now... 44 00:02:30,940 --> 00:02:32,657 Just kind of out there in the corpus. 45 00:02:32,740 --> 00:02:38,740 I think it's a good test of the kind of coding ability that... 46 00:02:39,070 --> 00:02:41,867 Because really basically what it's doing, it's like, you know, 47 00:02:41,950 --> 00:02:45,707 within the agent, I'm not seeing any code, but it basically is writing this whole 48 00:02:45,790 --> 00:02:47,917 video processing, image processing pipeline. 49 00:02:48,000 --> 00:02:50,547 Yes. And I'm constantly telling it like, all right... 50 00:02:50,630 --> 00:02:53,427 Like, look you took 30 seconds to give me that feedback. 51 00:02:53,510 --> 00:02:55,597 I only have a 60-second break. Like, you gotta go faster. 52 00:02:55,680 --> 00:02:57,780 Just optimize yourself, optimize yourself. 53 00:02:58,090 --> 00:03:02,057 And it's, I think, a good example of how stuff is 54 00:03:02,140 --> 00:03:05,267 going to go in the future where basically everyone is actually 55 00:03:05,350 --> 00:03:06,607 going to be writing tons of code. 56 00:03:06,690 --> 00:03:08,977 They're just not going to know that they're writing code. 57 00:03:09,060 --> 00:03:12,020 What kind of MMA steers does a model give you? 58 00:03:12,900 --> 00:03:16,567 Um, you want to see? Yeah. Basically, let me see if I can pull some stuff out. 59 00:03:16,650 --> 00:03:22,617 But basically it gives me a lot of stuff about balance and, like, hip position. 60 00:03:22,700 --> 00:03:26,327 Or it's like when you're kind of in side control here. 61 00:03:26,410 --> 00:03:28,997 It's just like you didn't get this underhook 62 00:03:29,080 --> 00:03:30,207 in this place so you're going to... 63 00:03:30,290 --> 00:03:33,250 You're vulnerable to this getting rolled over here. 64 00:03:33,340 --> 00:03:35,677 I'll pull up some stuff. You guys should try this. 65 00:03:35,760 --> 00:03:38,597 It was definitely one of his first use cases actually of even 66 00:03:38,680 --> 00:03:40,387 the initial prototypes. Yeah. Yeah. 67 00:03:40,470 --> 00:03:43,720 I mean, I did this with OpenClaw before we even had Muse working. 68 00:03:43,810 --> 00:03:45,140 It was just like... 69 00:03:45,310 --> 00:03:49,770 Then that's why I was saying it's kind of like become an eval for this too. 70 00:03:50,060 --> 00:03:52,527 It's good. I actually didn't know that it was an eval for the product. 71 00:03:52,610 --> 00:03:55,570 So noted. Yeah. There you go. 72 00:03:57,280 --> 00:03:58,180 How is all this... 73 00:03:59,110 --> 00:04:02,870 In the model's training data, like the fine-grained, 74 00:04:03,370 --> 00:04:07,700 this is what an MMA fighter looks like when they're appropriately 75 00:04:07,790 --> 00:04:10,497 balanced, and this is when they're like too much on the back foot. 76 00:04:10,580 --> 00:04:13,750 Just, I'm wondering, how has it consumed so many images of 77 00:04:14,460 --> 00:04:17,210 appropriate versus inappropriate MMA posture? 78 00:04:17,380 --> 00:04:17,837 Here you can see it. 79 00:04:17,920 --> 00:04:22,090 It just pulls out frames from it and it's like too much on my back leg there. 80 00:04:22,600 --> 00:04:24,810 Yeah, but is this the emergent intelligence? 81 00:04:24,930 --> 00:04:26,680 Like it does actually... 82 00:04:27,480 --> 00:04:30,140 Again, does it know from seeing lots of images of MMA 83 00:04:30,560 --> 00:04:33,027 or does it know because it understands the concept of where you're 84 00:04:33,110 --> 00:04:36,317 meant to be balancing? It's read books about MMA, and then it sees that you're 85 00:04:36,400 --> 00:04:37,407 too much on the back foot, or— 86 00:04:37,490 --> 00:04:43,870 Well, we did train the models with a lot of visual perception data. 87 00:04:44,280 --> 00:04:47,957 Yeah. So, I mean, I think that that's like kind of making good multimodal models. 88 00:04:48,040 --> 00:04:52,290 I mean, this gets into like, I think, a core approach thing that we have, 89 00:04:52,380 --> 00:04:55,180 which is we're training the models end-to-end to be good 90 00:04:55,590 --> 00:04:56,797 personal agents, right? 91 00:04:56,880 --> 00:05:00,760 So like the question is why would Meta want to build these models? 92 00:05:01,130 --> 00:05:04,260 First of all, I think just superintelligence overall is going 93 00:05:04,350 --> 00:05:07,307 to allow us to run the company better and build all kinds of different things. 94 00:05:07,390 --> 00:05:11,767 But for the products that we're building, there's certain capabilities and certain 95 00:05:11,850 --> 00:05:17,397 mixes of skills that basically are just much more applicable to these cases. 96 00:05:17,480 --> 00:05:20,947 So I mean, this is one of the things that Alex focused on from the beginning 97 00:05:21,030 --> 00:05:23,827 was like, all right, we really want to make sure that our 98 00:05:23,910 --> 00:05:27,407 multimodal program is good because that's disproportionately important for the types 99 00:05:27,490 --> 00:05:29,667 of things that people use all these services for. 100 00:05:29,750 --> 00:05:31,800 How do you train a good multimodal model? 101 00:05:32,420 --> 00:05:34,707 Is this where Meta just has an advantage because you guys have so 102 00:05:34,790 --> 00:05:36,707 much image and video data? It's very helpful. 103 00:05:36,790 --> 00:05:42,677 Yeah. And I think ultimately, a lot of this, it comes down to how are 104 00:05:42,760 --> 00:05:43,597 you focused on it, right? 105 00:05:43,680 --> 00:05:47,927 I think because of the fact that we wanted to build personal superintelligence. 106 00:05:48,010 --> 00:05:51,270 And fundamentally, I think most people in the world really 107 00:05:51,600 --> 00:05:53,687 experience the world in a highly multimodal way. 108 00:05:53,770 --> 00:05:58,940 They like talking to other people and listening to things and seeing videos 109 00:05:59,480 --> 00:06:01,107 and filming things of their own. 110 00:06:01,190 --> 00:06:04,780 I think most consumers have this highly multimodal experience. 111 00:06:04,910 --> 00:06:09,907 And especially earlier this year, there was actually a point of tension 112 00:06:09,990 --> 00:06:14,830 almost in the industry because I think so much of the consensus view within the AI 113 00:06:14,920 --> 00:06:19,130 industry was that it was just all about coding and coding was all that mattered. 114 00:06:19,800 --> 00:06:23,130 And I think that for the success of a product like 115 00:06:23,220 --> 00:06:28,010 Muse and the success of a lot of these consumer use cases, it really depends on 116 00:06:28,180 --> 00:06:30,480 the ability to be strong and multimodal. Yeah. 117 00:06:30,970 --> 00:06:33,017 I think we're just kind of scratching the surface of that. 118 00:06:33,100 --> 00:06:35,767 I mean, a lot of the initial theory that we had was that, 119 00:06:35,850 --> 00:06:39,647 I mean, most people don't actually like reading that much, right? 120 00:06:39,730 --> 00:06:43,900 So even the text-based interaction model, I think it's something that people do. 121 00:06:43,990 --> 00:06:45,280 I mean, everyone texts. 122 00:06:46,320 --> 00:06:49,537 But when you think about stuff like the glasses that we're working on, 123 00:06:49,620 --> 00:06:52,020 or even our other products that are very visual, 124 00:06:52,750 --> 00:06:56,417 the glasses basically will eventually just be a pure multimodal product, right? 125 00:06:56,500 --> 00:07:01,420 It's like you can let it hear what you are listening to, and it can talk to you, 126 00:07:01,630 --> 00:07:03,730 and it can see what you see if you let it. 127 00:07:03,880 --> 00:07:06,680 And then eventually, I don't even know if we're going to 128 00:07:07,260 --> 00:07:08,347 have an app store on the glasses. 129 00:07:08,430 --> 00:07:12,310 It'll just be like a multimodal generation that kind of creates whatever 130 00:07:13,270 --> 00:07:14,810 you need right in your vision. 131 00:07:15,480 --> 00:07:17,727 And I imagine something like that will probably end up powering a lot 132 00:07:17,810 --> 00:07:20,360 of the apps that we build too. 133 00:07:20,980 --> 00:07:23,780 But yeah, I mean, we're still in the early days of that. 134 00:07:24,900 --> 00:07:27,900 But for now, the perception work has been really impressive. 135 00:07:28,700 --> 00:07:31,700 I want to ask you guys about the product space here in AI 136 00:07:32,160 --> 00:07:33,700 and try something out on you. 137 00:07:33,830 --> 00:07:37,707 If I was to describe to you, say, with Stripe, what I think a winning 138 00:07:37,790 --> 00:07:41,207 product looks like, it would take me pages and pages to talk about because we have 139 00:07:41,290 --> 00:07:43,377 the developer experience and we have to talk about all the international 140 00:07:43,460 --> 00:07:46,257 heterogeneity of payment methods. Then we have to talk about stablecoins. 141 00:07:46,340 --> 00:07:47,887 Like, it would take me a long time. 142 00:07:47,970 --> 00:07:51,800 Whereas I notice a phenomenon in AI where it feels like the winning products 143 00:07:52,050 --> 00:07:56,057 collapse very quickly onto an agreed-upon product space 144 00:07:56,140 --> 00:07:59,517 that everyone is pursuing, and so transformers worked, and then 145 00:07:59,600 --> 00:08:02,770 like the chat agent UI really worked. 146 00:08:03,230 --> 00:08:04,780 And now we are seeing... 147 00:08:04,980 --> 00:08:09,860 You mentioned OpenClaw earlier, the kind of OpenClaw persistent agent, 148 00:08:09,950 --> 00:08:11,990 has a computer, able to do things. 149 00:08:12,120 --> 00:08:13,867 You know, computer use out there in the world view. 150 00:08:13,950 --> 00:08:15,787 We're collapsing on that product space. 151 00:08:15,870 --> 00:08:18,720 But there's something about AI where the winning products 152 00:08:19,000 --> 00:08:23,880 tend to be very simple, pursued by all the companies that are taking it seriously. 153 00:08:24,300 --> 00:08:26,297 You could describe on the back of a postcard... 154 00:08:26,380 --> 00:08:27,927 Like you could describe very concisely. 155 00:08:28,010 --> 00:08:30,927 And obviously, implementing them is massively complicated. 156 00:08:31,010 --> 00:08:32,800 Like, we'll get into the new model. 157 00:08:32,890 --> 00:08:35,927 There's, you know, huge amounts of frontier work 158 00:08:36,010 --> 00:08:37,477 that goes into making a good version. 159 00:08:37,560 --> 00:08:42,650 But the winning brand—now you're talking about the glasses or the future versions. 160 00:08:42,940 --> 00:08:48,190 But somehow there's a very small number of clearly correct products in AI 161 00:08:48,780 --> 00:08:53,570 as opposed to it being this, like, very broad and complicated product space. 162 00:08:54,490 --> 00:08:56,290 I kind of disagree. Sure. 163 00:08:57,870 --> 00:08:59,957 Well, first of all, I think part of the theory 164 00:09:00,040 --> 00:09:03,790 from the beginning was that this kind of agent or whatever form it took was 165 00:09:03,880 --> 00:09:09,630 going to be quite a different thing from this text-based assistant, right? 166 00:09:09,720 --> 00:09:12,427 Every company has one of those. I mean, ours is Meta AI. 167 00:09:12,510 --> 00:09:14,807 It reaches a ton of people, but I mean, it's like the same thing 168 00:09:14,890 --> 00:09:16,347 that all these companies have. 169 00:09:16,430 --> 00:09:18,597 And we, I think, 170 00:09:18,680 --> 00:09:23,230 specifically think that that is used for very informational purposes, 171 00:09:23,350 --> 00:09:26,817 but people don't think about it as like a thing that gets stuff done for them. 172 00:09:26,900 --> 00:09:28,647 It's retrieval, not delegation or something. 173 00:09:28,730 --> 00:09:31,700 Yeah, and even like when you say it has a computer... 174 00:09:31,950 --> 00:09:33,947 Yeah, but there's lots of different ways to do that, right? 175 00:09:34,030 --> 00:09:39,370 You know, I think that it's only kind of this most recent wave that 176 00:09:39,870 --> 00:09:44,037 gives the agent a dedicated VM, which I think is actually incredibly 177 00:09:44,120 --> 00:09:49,667 important for making you feel confident that the security and privacy of your 178 00:09:49,750 --> 00:09:53,967 information is—it's not just like, okay, it's not connecting to some elastic 179 00:09:54,050 --> 00:09:56,680 compute that just everyone uses. 180 00:09:57,050 --> 00:10:00,730 But my claim isn't that it's easy to implement, nor that it's obvious— 181 00:10:00,850 --> 00:10:02,017 I think there are going to be a lot of things. 182 00:10:02,100 --> 00:10:04,437 Yeah. Okay. You think there's going to be a lot of different things? 183 00:10:04,520 --> 00:10:07,187 Because again, my claim is that now that people see Muse, 184 00:10:07,270 --> 00:10:12,567 everyone looks at that and says, "Oh yeah, that's clearly how you want the AI agent 185 00:10:12,650 --> 00:10:16,100 product to work, and that the future AI agent products that we'll see 186 00:10:16,370 --> 00:10:19,080 will be fairly Muse-shaped." So I think... 187 00:10:19,410 --> 00:10:23,160 My take on this is, I think we're just so early in the realm 188 00:10:23,290 --> 00:10:27,080 of what AI products will look like that I think a lot of the interesting 189 00:10:27,210 --> 00:10:30,547 differentiation about the products kind of gets collapsed in how 190 00:10:30,630 --> 00:10:31,927 we view the products today. 191 00:10:32,010 --> 00:10:35,507 I think the traditional way in which we wrote down and think about 192 00:10:35,590 --> 00:10:36,840 product specs, that is... 193 00:10:38,010 --> 00:10:41,720 Those aren't the relevant dimensions and levers and axes 194 00:10:42,140 --> 00:10:46,560 to consider the important design decisions on AI-native products. 195 00:10:47,100 --> 00:10:52,110 And so, Muse, for example, I think so much attention was put in the design 196 00:10:52,320 --> 00:10:54,070 of the model for the product. 197 00:10:54,360 --> 00:10:57,867 And there were a lot of very intentional decisions around, in all these situations, 198 00:10:57,950 --> 00:11:00,200 how exactly should the model behave? 199 00:11:00,370 --> 00:11:02,787 And what's the exact personality of the model in these situations? 200 00:11:02,870 --> 00:11:05,787 And how should it handle these sorts of tool-use scenarios? 201 00:11:05,870 --> 00:11:07,920 And these are very intentional decisions. 202 00:11:08,420 --> 00:11:14,297 And probably in the same way that the full spec of how we wanted the model to behave 203 00:11:14,380 --> 00:11:17,257 in all these scenarios would be an extremely long document. 204 00:11:17,340 --> 00:11:23,060 But because of just how we think about products today, it's just AI. 205 00:11:23,810 --> 00:11:26,097 It's kind of like how when mobile first started happening, 206 00:11:26,180 --> 00:11:28,357 it was like, "Oh, it's just an app and everything is just 207 00:11:28,440 --> 00:11:32,520 an app." Well, it turns out there's a lot of nuance in what exactly an app is. 208 00:11:33,360 --> 00:11:37,117 And we're kind of in this phase where, oh, it's just AI, or it's just an agent, 209 00:11:37,200 --> 00:11:39,487 or it's just an agent with a VM or something like that. 210 00:11:39,570 --> 00:11:44,410 But the interesting differentiations we have to tease out. 211 00:11:44,700 --> 00:11:46,917 Okay. So you guys think I'm making the classic mistake, which is like, 212 00:11:47,000 --> 00:11:49,337 my domain is complex and this domain is, again, from a product 213 00:11:49,420 --> 00:11:51,337 point-of-view, it's simple. All payment processors seem the same. 214 00:11:51,420 --> 00:11:53,880 Yeah, no, I really agree with what you're saying. 215 00:11:54,050 --> 00:11:57,260 One thing that we talked about from early on, is that basically 216 00:11:58,010 --> 00:12:00,967 there is this pattern where I think the products are getting thinner 217 00:12:01,050 --> 00:12:03,307 and the model is getting to be more of the product, right? 218 00:12:03,390 --> 00:12:04,677 So one of the things that we say internally is like, 219 00:12:04,760 --> 00:12:05,897 the model is the product. Yes. 220 00:12:05,980 --> 00:12:08,187 Right? So, and when you're talking about an agent, 221 00:12:08,270 --> 00:12:10,920 a lot of the complex product work that went into Muse 222 00:12:11,310 --> 00:12:14,527 is basically around the privacy and security model for handling your data 223 00:12:14,610 --> 00:12:15,857 and the things that get connected to it. 224 00:12:15,940 --> 00:12:18,737 And that, I think, is something that we're just very proud of that we 225 00:12:18,820 --> 00:12:19,657 really focused on. 226 00:12:19,740 --> 00:12:22,867 But that's a lot of the product surface that we need to innovate on. 227 00:12:22,950 --> 00:12:26,580 But the capabilities, the personality, a lot of the stuff is like 228 00:12:26,950 --> 00:12:28,247 you're training into the model. 229 00:12:28,330 --> 00:12:32,290 And whether they're the people in the lab who are working on that or the people 230 00:12:32,540 --> 00:12:34,790 on Nat's team who are working on the product, 231 00:12:34,960 --> 00:12:38,717 I think when Nat is thinking about what is the key stuff that we need to make 232 00:12:38,800 --> 00:12:42,307 the product feel good, he's talking about post-training the model, right? 233 00:12:42,390 --> 00:12:43,177 Not like... 234 00:12:43,260 --> 00:12:46,137 I mean, of course the UI of the product matters too, but the most important 235 00:12:46,220 --> 00:12:48,810 part of the UI is the product. Yeah, exactly. 236 00:12:48,930 --> 00:12:53,610 I think there's like, there's still very viscerally, I think from a user, 237 00:12:53,860 --> 00:12:56,480 there's a feeling of using Muse over time. 238 00:12:57,070 --> 00:13:00,567 And maybe some meaningful portion of that is just the reliability 239 00:13:00,650 --> 00:13:02,157 of Muse to handle certain tasks. 240 00:13:02,240 --> 00:13:05,330 But then another meaningful portion is how is it proactive? 241 00:13:05,490 --> 00:13:07,287 And in what context is it proactive? 242 00:13:07,370 --> 00:13:10,537 And when it talks to you, how do you feel about how it talks to you? 243 00:13:10,620 --> 00:13:13,080 And so there's clearly a lot of... 244 00:13:14,250 --> 00:13:17,717 And I'm starting to see more and more of this actually show up on the X feed, 245 00:13:17,800 --> 00:13:20,837 which are people are starting to tease out these, like... 246 00:13:20,920 --> 00:13:24,260 They're all proactive agents, but what are the actual meaningful 247 00:13:24,350 --> 00:13:25,810 differences between them? 248 00:13:26,180 --> 00:13:28,687 And these, I think, are the interesting product decisions, 249 00:13:28,770 --> 00:13:32,477 which, you know, the funny thing is how do you specify these product decisions? 250 00:13:32,560 --> 00:13:36,460 Well, it's ultimately in model design decisions, which ultimately boil down to 251 00:13:36,820 --> 00:13:40,067 training methods or data or evals or, you know, all these things. 252 00:13:40,150 --> 00:13:44,280 Yes. As we talk about product space, one notable Muse detail is 253 00:13:44,370 --> 00:13:45,947 that there is no model selector. 254 00:13:46,030 --> 00:13:48,410 and no effort selector. 255 00:13:49,120 --> 00:13:52,577 Do you guys think that's just obviously correct in a consumer AI product, or— 256 00:13:52,660 --> 00:13:54,160 I just want to make it simple. 257 00:13:54,250 --> 00:13:55,850 Yeah. I think that we've been... 258 00:13:57,960 --> 00:14:01,110 I think somewhat infamously, a lot of the AI products have been 259 00:14:01,630 --> 00:14:07,927 designed by a highly technical group that do not have these sort of randomly 260 00:14:08,010 --> 00:14:10,980 sampled American in their mental model. 261 00:14:11,520 --> 00:14:12,640 And so the products... 262 00:14:13,060 --> 00:14:18,900 I mean, the naming of some of the models is like so egregious. 263 00:14:19,020 --> 00:14:19,817 The numbers go backwards? 264 00:14:19,900 --> 00:14:22,567 Some of the times the numbers go backwards and sometimes it's just like a totally 265 00:14:22,650 --> 00:14:24,657 different word and you have another set of words. 266 00:14:24,740 --> 00:14:27,140 And like, I just like, how is somebody who's not 267 00:14:27,740 --> 00:14:32,827 carefully reading API design specs supposed to know 268 00:14:32,910 --> 00:14:35,497 what they're supposed to pick? Like, it's just far too confusing. 269 00:14:35,580 --> 00:14:37,917 Okay, we're talking about models a lot. Tell me about the new model. 270 00:14:38,000 --> 00:14:38,960 It sounds awesome. 271 00:14:39,670 --> 00:14:41,587 Yeah, I mean, I think we're quite proud of this. 272 00:14:41,670 --> 00:14:45,720 Um, yeah, the new model—Muse Wonder—it's a new class of model 273 00:14:45,930 --> 00:14:47,007 that is at the frontier. 274 00:14:47,090 --> 00:14:49,097 So we've basically been working on this since the beginning. 275 00:14:49,180 --> 00:14:53,597 Basically, when you're doing these pre-trains, you don't just go to whatever 276 00:14:53,680 --> 00:14:55,477 the biggest cluster is that you have. 277 00:14:55,560 --> 00:14:59,110 You want to work your way up the ladder training models that are 278 00:14:59,650 --> 00:15:04,610 maybe 4 or 5 times bigger, and then you just keep on repeating that process. 279 00:15:05,450 --> 00:15:11,367 So the first set of models that we've had Muse Spark was all based on this initial 280 00:15:11,450 --> 00:15:14,000 pre-training that we did in, what was it, December? 281 00:15:14,120 --> 00:15:15,497 Yeah, the end of last year. Yeah. 282 00:15:15,580 --> 00:15:16,917 And, and it's actually... 283 00:15:17,000 --> 00:15:21,457 I think we've all just been quite surprised and impressed with how 284 00:15:21,540 --> 00:15:25,967 much intelligence and capability the team has been able to build 285 00:15:26,050 --> 00:15:30,050 into a model which is like frankly not that big, right? 286 00:15:30,140 --> 00:15:33,680 I mean, and then after that we were working on this next model, 287 00:15:33,970 --> 00:15:36,897 and the pre-training has been done for a bit, and we've been working 288 00:15:36,980 --> 00:15:39,400 on the post-training, and now it is here. 289 00:15:40,230 --> 00:15:42,360 So it's the next class of model. 290 00:15:42,480 --> 00:15:44,277 It is significantly bigger. 291 00:15:44,360 --> 00:15:48,740 So I think you you will feel that in terms of the nuance in which it can think. 292 00:15:50,030 --> 00:15:50,657 And it really. 293 00:15:50,740 --> 00:15:55,077 A lot of the areas that we focused on were making it the best personal 294 00:15:55,160 --> 00:15:57,080 and professional agentic model. 295 00:15:57,210 --> 00:16:01,710 Some of the different things that you can optimize for that's it for us, and then 296 00:16:02,380 --> 00:16:06,757 yeah, a lot of the basic capabilities that we focused on building in. 297 00:16:06,840 --> 00:16:11,800 Very good memory, very good ability to persist through complicated tasks. 298 00:16:11,890 --> 00:16:15,557 But a ton of work went into this across the lab and the product 299 00:16:15,640 --> 00:16:16,980 teams and all of this. 300 00:16:17,310 --> 00:16:21,310 And I think it's also just sort of a testament to the the team being able 301 00:16:21,400 --> 00:16:24,857 to methodically figure out the process to scale these models. 302 00:16:24,940 --> 00:16:28,320 So I think this is also just, you know, 303 00:16:30,150 --> 00:16:33,157 a milestone on the way towards the time 304 00:16:33,240 --> 00:16:36,367 between each subsequently bigger class of models, I think will 305 00:16:36,450 --> 00:16:37,950 now start to compress. 306 00:16:38,080 --> 00:16:40,497 And that's very exciting for the type of stuff that we can build. 307 00:16:40,580 --> 00:16:42,337 At Meta or in the industry generally? 308 00:16:42,420 --> 00:16:46,710 At Meta. I mean, I think you're seeing something like this everywhere else, but— 309 00:16:46,800 --> 00:16:48,800 Is this an RSI effect or something else? 310 00:16:49,050 --> 00:16:51,590 Well, I think at Meta, part of this is just... 311 00:16:52,220 --> 00:16:53,053 We needed to... 312 00:16:53,510 --> 00:16:56,427 Well, I think quite famously, we were not happy with the trajectory 313 00:16:56,510 --> 00:16:59,847 that we were on with Llama 4. We kind of like rebuilt the whole team. 314 00:16:59,930 --> 00:17:02,977 That's been this kind of crazy thing over the last year and a half that we should 315 00:17:03,060 --> 00:17:04,487 probably spend a bunch of time on. 316 00:17:04,570 --> 00:17:04,817 Oh, we'll get to that. Yeah, yeah. 317 00:17:04,900 --> 00:17:11,067 Some war stories. But the team basically had to go build up like different tiers 318 00:17:11,150 --> 00:17:16,910 of models to get to this one. And now that we are kind of at the frontier, 319 00:17:17,490 --> 00:17:21,497 now everything that we do going forward will just be pushing the frontier, 320 00:17:21,580 --> 00:17:25,087 as opposed to there's a lot of background work that we needed to do before that. 321 00:17:25,170 --> 00:17:27,500 I don't know, what would you say? Yeah. 322 00:17:27,590 --> 00:17:29,547 Well, I think we're really excited about it. 323 00:17:29,630 --> 00:17:35,467 We will have 2 models in this series that are going to be available. 324 00:17:35,550 --> 00:17:39,140 So there's Muse Wonder and Muse Wonder Full. 325 00:17:39,850 --> 00:17:41,307 Laugh? Yeah, all right, there you go. 326 00:17:41,390 --> 00:17:45,310 Yeah, very good. Uh, which is basically a much higher— Studio band version. 327 00:17:45,400 --> 00:17:47,437 Yeah, no, a much higher-end model. 328 00:17:47,520 --> 00:17:49,777 It's the full version of Muse 1. 329 00:17:49,860 --> 00:17:53,740 I know, I got it. Yeah, yeah, it's not— It's 2 words. Wonder full version. 330 00:17:56,410 --> 00:17:58,577 You're clearly a father. What's up? 331 00:17:58,660 --> 00:18:00,787 This is an excellent dad joke. Yeah, there you go. 332 00:18:00,870 --> 00:18:01,827 Um, and, and the performance... 333 00:18:01,910 --> 00:18:05,830 I mean, we've seen like incredible performance across a slate of evaluations 334 00:18:05,920 --> 00:18:08,210 on personal, professional agent tasks. 335 00:18:08,300 --> 00:18:10,467 It's state-of-the-art based on our measurement. 336 00:18:10,550 --> 00:18:14,217 And then there's a number of, I think, specific highlights which we're 337 00:18:14,300 --> 00:18:17,700 really excited about. Which we've designed into the model over time. 338 00:18:17,890 --> 00:18:21,890 So active memory, like Mark mentioned, is one area where 339 00:18:22,600 --> 00:18:26,687 we think that to build personal agents, personal professional agents, 340 00:18:26,770 --> 00:18:31,480 like memory is just one of the clear requirements to build reliable agents 341 00:18:31,690 --> 00:18:33,657 and is one of the areas where we've invested, I think, 342 00:18:33,740 --> 00:18:36,987 disproportionately actually compared to many of the other efforts. 343 00:18:37,070 --> 00:18:40,407 And we have our own internal evals of Memory Bench which demonstrate 344 00:18:40,490 --> 00:18:42,090 a lot of the capabilities there. 345 00:18:42,910 --> 00:18:45,310 What does memory mean in a model to a layperson? 346 00:18:46,250 --> 00:18:47,497 Is it just how much stuff it can... 347 00:18:47,580 --> 00:18:50,337 Like the working memory of if you're working on a task, how much stuff 348 00:18:50,420 --> 00:18:52,170 the model can kind of keep in its— 349 00:18:52,840 --> 00:18:54,387 Yeah. How's it different from context? 350 00:18:54,470 --> 00:18:57,090 I think it's in the experience of using the model. 351 00:18:57,390 --> 00:19:01,597 How well does it remember things that you mentioned to it over days, weeks, 352 00:19:01,680 --> 00:19:05,017 or months, and then leverage those in the process of doing future tasks? 353 00:19:05,100 --> 00:19:07,860 I see. This is the stuff that's buggy AF in OpenClaw. 354 00:19:08,150 --> 00:19:10,700 Like, I think we're all extremely fond of OpenClaw. 355 00:19:11,150 --> 00:19:15,067 Very happy OpenClaw user, but it ultimately does this compaction 356 00:19:15,150 --> 00:19:18,327 where it has like the long-term memory and the working short-term memory or whatever. 357 00:19:18,410 --> 00:19:22,207 And I think once people use OpenClaw for a while, they find the memory pretty buggy. 358 00:19:22,290 --> 00:19:27,540 Yeah. So we've designed basically both the model and a system for the model to use 359 00:19:27,710 --> 00:19:31,110 to develop a much stronger form of memory. We call it active memory. 360 00:19:31,250 --> 00:19:34,880 Yeah. In addition, as Mark mentioned on the persistence— 361 00:19:35,010 --> 00:19:37,507 But so this is outside the model, like the model exists and it's— 362 00:19:37,590 --> 00:19:39,927 No, no, but some of the capabilities that you're talking about, 363 00:19:40,010 --> 00:19:43,137 like the compaction, are things that the model does, right? 364 00:19:43,220 --> 00:19:45,687 So basically being able to do this well, both with— 365 00:19:45,770 --> 00:19:47,807 It's not a harness, it's in the model itself. 366 00:19:47,890 --> 00:19:50,230 Yes, it's in the model itself. Exactly, yeah. 367 00:19:50,810 --> 00:19:55,490 So it's aware of its memory as a core system. 368 00:19:55,820 --> 00:19:59,870 And it's about how it's able to leverage various sources to be able to 369 00:20:00,950 --> 00:20:03,957 conduct tasks in the future, as well as understand its ability to do 370 00:20:04,040 --> 00:20:06,997 compaction, how it should do that compaction most effectively, et cetera. 371 00:20:07,080 --> 00:20:09,957 Okay. But this is a big deal because I think a lot of people's early experience 372 00:20:10,040 --> 00:20:12,840 of AI agents has been buggy memory. 373 00:20:13,130 --> 00:20:14,750 And so, yeah, improving that is— 374 00:20:14,920 --> 00:20:17,257 And to be clear, there's still plenty of room to improve. 375 00:20:17,340 --> 00:20:20,217 Our evaluations show this, but this is something that we're quite 376 00:20:20,300 --> 00:20:22,557 proud of, and we think this is a meaningful contribution. 377 00:20:22,640 --> 00:20:23,847 Okay. So active memory is one thing. 378 00:20:23,930 --> 00:20:24,637 Yeah. Active memory. 379 00:20:24,720 --> 00:20:28,807 Active focus is another, which is effectively just the ability for the model 380 00:20:28,890 --> 00:20:31,520 to, when conducting tasks, 381 00:20:31,610 --> 00:20:34,360 be able to carry them through, especially as we've seen 382 00:20:34,730 --> 00:20:39,030 Muse users as well as Muse Code users, give higher and higher complexity 383 00:20:39,240 --> 00:20:41,947 and more and more ambiguous and more and more complex tasks, 384 00:20:42,030 --> 00:20:45,117 we want the model to be able to conduct those and do those very effectively. 385 00:20:45,200 --> 00:20:48,700 And this is one of the things actually in early dogfooding and testing 386 00:20:48,790 --> 00:20:52,297 that a lot of people have noticed about the model that is sort of special 387 00:20:52,380 --> 00:20:55,930 and stands out with respect to other models that they've tried or used. 388 00:20:57,170 --> 00:20:59,630 The third one is domain expertise. 389 00:20:59,720 --> 00:21:04,510 What is focus in AI terms? Is it like putting a lot of compute on the task? 390 00:21:04,600 --> 00:21:07,310 Is it not getting distracted with other— 391 00:21:08,180 --> 00:21:09,937 Kind of, persistence? Yeah, like persistence. 392 00:21:10,020 --> 00:21:14,027 It's basically like the ability to conduct complete tasks over very long horizons. 393 00:21:14,110 --> 00:21:19,820 Okay. And then when faced with various setbacks along the way, 394 00:21:20,240 --> 00:21:24,070 the ability to identify other solutions to those problems in an aligned way. 395 00:21:24,830 --> 00:21:25,740 Okay. 396 00:21:26,410 --> 00:21:31,247 Then there's... It's actually a big step up in coding capabilities, 397 00:21:31,330 --> 00:21:32,377 so frontier coding capabilities. 398 00:21:32,460 --> 00:21:37,420 So it is actually state-of-the-art on DeepSWE and, SWE-Marathon 399 00:21:38,420 --> 00:21:41,887 and does very well on other benchmarks like Terminal-Bench 4 and many others. 400 00:21:41,970 --> 00:21:45,430 So we think that users who use it for coding, 401 00:21:45,640 --> 00:21:49,557 which will be a minority of all of our users, but the ones who do will see 402 00:21:49,640 --> 00:21:51,440 a big improvement and a big step up. 403 00:21:51,680 --> 00:21:55,940 Okay, so we should also just be using Muse Wonder Full for coding at Stripe? 404 00:21:56,440 --> 00:21:59,277 Potentially, yeah. I mean, let's go out there and— 405 00:21:59,360 --> 00:22:01,567 But like if it's state of the art on SWE-bench, yeah. 406 00:22:01,650 --> 00:22:04,277 We'd be missing out. Yeah, no, it's a big improvement on coding. 407 00:22:04,360 --> 00:22:07,907 I mean, obviously, we want people to try it and see how they 408 00:22:07,990 --> 00:22:11,540 experience it, but we've been very pleased in our own internal testing. 409 00:22:12,160 --> 00:22:14,627 Okay, so this is— Yeah, I think people will like it for a lot of things. 410 00:22:14,710 --> 00:22:19,417 Okay, so this is both just your latest frontier model, like people can come, 411 00:22:19,500 --> 00:22:22,217 you know, developers can come directly to you, they can use it on OpenRouter, 412 00:22:22,300 --> 00:22:26,850 like they can just consume it as a model, and it's also a big upgrade 413 00:22:27,680 --> 00:22:33,020 inside of Muse that people will just feel as active users of Muse. 414 00:22:33,560 --> 00:22:33,727 Yeah. 415 00:22:33,810 --> 00:22:36,647 Well, what we're going to do in Muse, because it's a lot bigger and it's more 416 00:22:36,730 --> 00:22:43,530 expensive to serve, we're going to start off primarily in paid subscriptions. 417 00:22:43,740 --> 00:22:45,827 So what we want to do with Muse as a product, we want it to be 418 00:22:45,910 --> 00:22:47,240 free for everyone to use. 419 00:22:47,740 --> 00:22:51,090 So we have this huge amount of free tokens that people can use with 420 00:22:51,290 --> 00:22:52,740 with Muse Spark as the model. 421 00:22:53,200 --> 00:22:58,500 But if you want to use Wonder for most of the hard tasks, then 422 00:23:00,250 --> 00:23:02,710 at least for now, you'll use the paid version. 423 00:23:03,170 --> 00:23:06,757 But, you know, I mean, eventually we'll, you know how quickly the field moves. 424 00:23:06,840 --> 00:23:09,807 Yes. And we hope to, you know, at some point not too far from now, 425 00:23:09,890 --> 00:23:13,770 have a model that is as good as Wonder, but in the Spark form factor. 426 00:23:14,270 --> 00:23:17,607 And that will be a great thing to be able to serve hundreds of millions 427 00:23:17,690 --> 00:23:20,687 of people around the world. Okay, so you go to Muse, you subscribe to the—what's it 428 00:23:20,770 --> 00:23:22,737 called—like the premium or pro plan, or... 429 00:23:22,820 --> 00:23:26,447 Pay money. Yeah. And just don't stop until you've taken out your credit card. 430 00:23:26,530 --> 00:23:28,487 Yeah. And then you'll be able to... 431 00:23:28,570 --> 00:23:30,827 You'll select it, or it'll just magically— Oh, it just gets better. 432 00:23:30,910 --> 00:23:32,537 It just gets better for you. Yeah. 433 00:23:32,620 --> 00:23:39,540 And then it'll also be available via API, on the Meta Model API and OpenRouter. 434 00:23:40,210 --> 00:23:45,010 And then it'll also be available for Muse Code users. 435 00:23:46,170 --> 00:23:50,430 And then Muse Wonder Full will be available to subscribers of Muse Code. 436 00:23:51,560 --> 00:23:53,720 So it's pretty exciting. 437 00:23:53,810 --> 00:23:56,897 I think we want to, as Mark mentioned, it is a much larger model. 438 00:23:56,980 --> 00:23:59,607 And so we're working through how to serve that most effectively 439 00:23:59,690 --> 00:24:00,977 to as many people as possible. 440 00:24:01,060 --> 00:24:03,897 But we're excited to get it in the hands of as many people as possible. 441 00:24:03,980 --> 00:24:07,697 It's a big step forward. I mean, this is like, and we really with Muse Spark, 442 00:24:07,780 --> 00:24:10,947 we were very happy with the performance that we were able to deliver as 443 00:24:11,030 --> 00:24:11,947 we were getting up to this point. 444 00:24:12,030 --> 00:24:17,000 But like, this is, the beginning of, like, MSL kind of being at the frontier. 445 00:24:18,080 --> 00:24:20,707 We feel like the models are quite competitive with the best 446 00:24:20,790 --> 00:24:24,090 models that are out there. And I think that this is— 447 00:24:24,630 --> 00:24:30,720 Okay, so you think it's like Astra, Fable, and Wonder altogether at the frontier? 448 00:24:31,220 --> 00:24:33,387 I mean, you can look at it empirically by the benchmarks. 449 00:24:33,470 --> 00:24:35,807 Empirically by the benchmarks, one would conclude that. 450 00:24:35,890 --> 00:24:36,723 Yes. I see. Yes. 451 00:24:37,140 --> 00:24:41,310 And I think actually one of the things that we're especially proud of is 452 00:24:41,770 --> 00:24:47,030 we have taken the time, as we've mentioned, to optimize models 453 00:24:47,190 --> 00:24:49,150 for the product form factor of Muse. 454 00:24:49,240 --> 00:24:51,697 So this is the state-of-the-art model. 455 00:24:51,780 --> 00:24:55,077 We think it really is the best model in the world for the use case 456 00:24:55,160 --> 00:24:56,760 of personal professional agents. 457 00:24:56,990 --> 00:24:59,877 Yeah, people inside companies like complaining about Goodhart's Law. 458 00:24:59,960 --> 00:25:03,297 You know, a metric that becomes a target ceases to become a good metric. 459 00:25:03,380 --> 00:25:07,297 You hear complaints that models are benchmaxed and that at a certain point, 460 00:25:07,380 --> 00:25:11,640 kind of overfitting to the benchmarks actually makes the models slightly worse. 461 00:25:12,550 --> 00:25:15,427 As people who actually have to train AI models, is that true? 462 00:25:15,510 --> 00:25:17,887 Are the benchmarks pretty good at grading models? 463 00:25:17,970 --> 00:25:20,937 Do you start to see weird edge failure cases from the— 464 00:25:21,020 --> 00:25:22,687 The benchmarks have to keep moving, right? 465 00:25:22,770 --> 00:25:26,980 Because it happens very quickly, but they just get saturated so quickly. 466 00:25:27,070 --> 00:25:31,280 So, I mean, I do think that— But you think non-saturated benchmarks are fine? 467 00:25:32,110 --> 00:25:37,660 I mean, as we've gone on, we steer the models more towards our internal 468 00:25:37,870 --> 00:25:40,287 benchmarks for the use cases that we're focused on. 469 00:25:40,370 --> 00:25:42,750 So, I mean, early on when we were kind of just 470 00:25:43,380 --> 00:25:45,780 doing the science of trying to climb the ladder, 471 00:25:46,090 --> 00:25:48,967 we cared more about what the external benchmarks were just to make 472 00:25:49,050 --> 00:25:50,467 sure we were on the right track. 473 00:25:50,550 --> 00:25:53,380 And then I'd say, you know, each release that goes by, 474 00:25:54,090 --> 00:25:57,850 the larger share of what we're looking at are internal ones. 475 00:25:58,430 --> 00:26:02,600 And one of the things that we actually quantified in our research blog post as 476 00:26:02,690 --> 00:26:07,147 part of Muse Wonder is this metric for big model smell, 477 00:26:07,230 --> 00:26:10,197 which is kind of, you know, big model smell and feeling of benchmaxing 478 00:26:10,280 --> 00:26:11,197 are kind of like inverses of one another. 479 00:26:11,280 --> 00:26:13,567 This idea that you could just like tell when you're using a really 480 00:26:13,650 --> 00:26:15,157 big model, it just... You can tell. 481 00:26:15,240 --> 00:26:16,947 It just has nuance. It's got that je ne sais quoi. 482 00:26:17,030 --> 00:26:20,367 Yeah. And so we, the way we quantify that is actually almost explicitly what you're 483 00:26:20,450 --> 00:26:23,497 mentioning, which is the gap between in-distribution performance 484 00:26:23,580 --> 00:26:26,087 and out-of-distribution performance. So what is the... 485 00:26:26,170 --> 00:26:29,710 We kind of measure how well does the model perform on tasks 486 00:26:29,800 --> 00:26:33,670 that are in distribution of our RL and post-training versus tasks 487 00:26:33,760 --> 00:26:37,210 that are out of distribution. And what does that gap look like for... 488 00:26:37,850 --> 00:26:40,887 What does that gap look like between the Muse Spark line and the Muse Wonder line? 489 00:26:40,970 --> 00:26:44,477 And we actually see this huge step up from the Muse Spark to the Muse Wonder 490 00:26:44,560 --> 00:26:47,727 line because of, as Mark mentioned, like, it's just a much better base 491 00:26:47,810 --> 00:26:52,277 model powering this new line. And there's an entire je ne sais quoi to that, 492 00:26:52,360 --> 00:26:55,610 but I think we've been able to actually quantify this effect. 493 00:26:56,030 --> 00:26:59,830 And it's an obviously dimension that we're excited to continue improving on. 494 00:27:00,200 --> 00:27:03,867 But if you're saying that external benchmarks have ceased to become 495 00:27:03,950 --> 00:27:07,350 as important, and you care much more about your internal benchmarks. 496 00:27:07,460 --> 00:27:10,860 Is that kind of an argument against benchmaxing, that you think that 497 00:27:11,420 --> 00:27:14,087 just trying to get the top score on the external benchmarks that everyone 498 00:27:14,170 --> 00:27:17,570 talks about is maybe not going to lead you to develop a great model? 499 00:27:17,840 --> 00:27:20,540 Well, yeah, but also we're focused on specific things. 500 00:27:20,680 --> 00:27:24,647 I mean, part of the whole strategy is personal superintelligence and just trying 501 00:27:24,730 --> 00:27:27,767 to build different types of products than other people are trying to build. 502 00:27:27,850 --> 00:27:30,980 So even if we did have the exact same evals as everyone else. 503 00:27:31,360 --> 00:27:32,193 It just— 504 00:27:32,820 --> 00:27:35,697 Yes. And though I would say a personal superintelligence is not like a super 505 00:27:35,780 --> 00:27:38,570 niche interest, like it's probably something that other 506 00:27:38,660 --> 00:27:40,120 models want to be able to do. 507 00:27:40,990 --> 00:27:44,037 I think that there are things that are very different about it. 508 00:27:44,120 --> 00:27:47,120 I mean, I think like the personality probably matters more. 509 00:27:47,250 --> 00:27:51,207 I mean, one of the things that we talk about is this idea of discretion, right? 510 00:27:51,290 --> 00:27:54,240 So if you want to like use the model to get something done, 511 00:27:55,050 --> 00:27:59,010 It needs to be able to know a lot about you and get things done in the world 512 00:27:59,510 --> 00:28:01,887 while disclosing as little as possible. 513 00:28:01,970 --> 00:28:04,847 So, you know, if you're— you have an allergy or you're 514 00:28:04,930 --> 00:28:06,847 pregnant or something, you're trying to, like, get a, 515 00:28:06,930 --> 00:28:09,517 you know, like, a table at a restaurant and you don't want to, like, 516 00:28:09,600 --> 00:28:11,937 have to tell them your medical condition. Yes, yeah. 517 00:28:12,020 --> 00:28:14,897 You know, to make sure that it's going to be a good place for you. 518 00:28:14,980 --> 00:28:16,157 Like, okay, how do you handle that? 519 00:28:16,240 --> 00:28:20,320 And I mean, humans have common sense and the models so far 520 00:28:20,740 --> 00:28:24,747 If you're building a coding agent that works in an environment where everyone has 521 00:28:24,830 --> 00:28:28,577 access to the same context, discretion isn't particularly important. 522 00:28:28,660 --> 00:28:32,920 So there are all these things that I think are just kind of more important for us. 523 00:28:33,290 --> 00:28:34,123 Yes. 524 00:28:34,550 --> 00:28:37,467 Aspects of coding where the code needs to work, 525 00:28:37,550 --> 00:28:41,637 but you don't— the consumers who are using it may not even ever see the code. 526 00:28:41,720 --> 00:28:45,387 I mean, that probably matters more in a personal superintelligence context 527 00:28:45,470 --> 00:28:47,517 rather than serving software developers. 528 00:28:47,600 --> 00:28:50,060 So I think these things are just a bit different. 529 00:28:50,480 --> 00:28:52,857 Yeah. Ultimately, we care about making useful models. 530 00:28:52,940 --> 00:28:56,070 So we want models when you put them into people's hands, 531 00:28:56,440 --> 00:28:58,400 they have a great experience using. 532 00:28:58,650 --> 00:29:02,657 And I think that is the most important metric for any model that we 533 00:29:02,740 --> 00:29:03,940 or other people produce. 534 00:29:04,120 --> 00:29:08,910 I think that really focusing on the external benchmarks, 535 00:29:09,410 --> 00:29:12,560 if you really dig into the supply chain a little bit, it's some 536 00:29:12,880 --> 00:29:17,800 PhD student who made these benchmarks, a benchmark as part of their, 537 00:29:18,460 --> 00:29:20,137 as the course of them doing their PhD. 538 00:29:20,220 --> 00:29:24,427 And so you're kind of outsourcing all of your product thinking to that PhD 539 00:29:24,510 --> 00:29:26,427 student who happened to make a viral benchmark. 540 00:29:26,510 --> 00:29:28,060 Right. So clearly that's wrong. 541 00:29:28,180 --> 00:29:30,080 Yeah. And so that's why we care a lot. 542 00:29:30,350 --> 00:29:34,060 A huge part of what a lot of people at Meta and MSL are focused 543 00:29:34,150 --> 00:29:38,690 on is crafting amazing evals. And now that we have them, we can work on our own. 544 00:29:38,820 --> 00:29:41,900 But if you're building a you're ramping up your lab, right? 545 00:29:41,990 --> 00:29:45,447 It's like those public ones are not a crazy place to start, but it's 546 00:29:45,530 --> 00:29:46,617 just not where you want to end up. 547 00:29:46,700 --> 00:29:48,827 You're just at that point, you're just looking for signs of life. 548 00:29:48,910 --> 00:29:52,810 But then when you're like really in this mode, as we are now, where it's like, 549 00:29:53,080 --> 00:29:55,627 you know, we want to be very intentional about how 550 00:29:55,710 --> 00:29:59,797 we're improving our model over time to build better and better, better products. 551 00:29:59,880 --> 00:30:03,047 Like a lot of the cases that Mark just mentioned, like these are 552 00:30:03,130 --> 00:30:07,470 actually detailed very rigorously in like the spreadsheet of all the little 553 00:30:07,560 --> 00:30:11,807 capabilities and nuances of the product that we think really 554 00:30:11,890 --> 00:30:13,100 matter to the end user. 555 00:30:13,190 --> 00:30:16,437 And maybe each of them matters like 2% to the user, but in aggregate, 556 00:30:16,520 --> 00:30:21,070 that's the obsession that will ultimately result in these models that feel magical. 557 00:30:21,940 --> 00:30:24,737 Can you guys describe for me just what goes into training an RL model? 558 00:30:24,820 --> 00:30:27,080 Or sorry, a leading-edge frontier model? 559 00:30:27,330 --> 00:30:30,367 I'm just curious in that I think a lot of people listening 560 00:30:30,450 --> 00:30:34,037 to this have a rough idea where there's there's a pre-training 561 00:30:34,120 --> 00:30:34,917 and a post-training step. 562 00:30:35,000 --> 00:30:38,150 So pre-training is you kind of ingest this huge amount of data. 563 00:30:38,250 --> 00:30:40,377 You look at a lot of pictures and these are cats and these are dogs. 564 00:30:40,460 --> 00:30:44,507 And after a while you start to develop a sense for what is catness and what is 565 00:30:44,590 --> 00:30:47,300 dogness and what is fur and everything like that. 566 00:30:47,720 --> 00:30:50,477 And so that's kind of the pre-training step where you just hoover up data 567 00:30:50,560 --> 00:30:52,687 and maybe you can talk about the data acquisition and just kind 568 00:30:52,770 --> 00:30:53,930 of what goes into that. 569 00:30:54,100 --> 00:31:00,940 And then there's this RL stage, which is the newish advance 570 00:31:01,030 --> 00:31:02,930 over kind of the past few years of AI. 571 00:31:03,320 --> 00:31:07,737 And again, it sounds like that is where a lot of the specific nuance of this model 572 00:31:07,820 --> 00:31:12,910 has come in, where you guys have chosen better domains, 573 00:31:15,120 --> 00:31:17,960 better hills to climb for the model. 574 00:31:18,330 --> 00:31:21,750 But just at a high level, describe how many people, 575 00:31:21,840 --> 00:31:25,240 how long it takes, what the steps are, how you think about those in— 576 00:31:26,300 --> 00:31:27,887 How much time do you have? Yeah, at a high level. 577 00:31:27,970 --> 00:31:32,057 This happens to be What we spend all of our time on. 578 00:31:32,140 --> 00:31:35,680 Exactly. And so summarizing— YouTube courses. 579 00:31:36,270 --> 00:31:39,977 Yeah, I think at a very high level, these are the stages. 580 00:31:40,060 --> 00:31:41,860 It's pre-training and post-training. 581 00:31:42,530 --> 00:31:46,400 Ironically, training is the superset of all of it. 582 00:31:46,780 --> 00:31:49,487 But that's— so how many people are on this whole effort, like when you 583 00:31:49,570 --> 00:31:50,247 develop these models? 584 00:31:50,330 --> 00:31:54,330 So this, I think you're getting into what is interesting about MSL 585 00:31:54,830 --> 00:31:58,627 and the approach that we've taken, which is, I mean, From the very beginning, 586 00:31:58,710 --> 00:32:01,297 I think when I first reached out to you, I think we were just looking at this 587 00:32:01,380 --> 00:32:04,297 because we were like, just like, wow, okay, how far we've come. 588 00:32:04,380 --> 00:32:06,177 And I was just like, I pulled up the screenshot of like 589 00:32:06,260 --> 00:32:07,927 when I first reached out to Alex. 590 00:32:08,010 --> 00:32:10,767 My first thing was like, hey, do you have time to catch up and talk? 591 00:32:10,850 --> 00:32:12,057 And then it's like, all right, we talked for— 592 00:32:12,140 --> 00:32:14,477 That's how they get you. 44 minutes or whatever. 593 00:32:14,560 --> 00:32:19,350 And then after that call, 2 days later, Alex sends me this long thing, 594 00:32:19,440 --> 00:32:23,940 which is like, well, here's what I think I would do if I were in your position. 595 00:32:24,530 --> 00:32:29,447 And the very first thing is that you don't need a lot of people, and 596 00:32:29,530 --> 00:32:31,327 talent density is the thing that matters. 597 00:32:31,410 --> 00:32:34,990 So I'd say, like, from the beginning of the plan, this was, 598 00:32:35,080 --> 00:32:37,287 like, the core of your strategy, right? 599 00:32:37,370 --> 00:32:39,620 It's basically, like, you actually don't need 600 00:32:40,120 --> 00:32:44,210 a massive number of people, but what you do need is for, 601 00:32:44,880 --> 00:32:48,830 like, each of the people to be as good as possible, which kind of informed this 602 00:32:49,130 --> 00:32:51,380 kind of crazy recruiting push and, like, very 603 00:32:52,720 --> 00:32:56,227 intense filtering that we had to take, you know, some of the best people 604 00:32:56,310 --> 00:33:00,770 from inside Meta and a very intense process to get people from outside Meta. 605 00:33:01,350 --> 00:33:05,020 But I actually think like a lot of what has made MSL special 606 00:33:05,480 --> 00:33:09,317 and why a lot of people have wanted to join is because I think it's in some 607 00:33:09,400 --> 00:33:12,537 ways it's just like more fun to work on a talent and smaller effort. 608 00:33:12,620 --> 00:33:14,077 It feels like the startup. Yes. 609 00:33:14,160 --> 00:33:15,010 That's operating. 610 00:33:16,330 --> 00:33:18,950 In some ways it has all the benefits of a startup 611 00:33:19,040 --> 00:33:21,690 with the stability and resources of a larger company. 612 00:33:21,830 --> 00:33:25,417 And this was like this environment that I think this was like point one in your kind 613 00:33:25,500 --> 00:33:29,550 of plan for what, um, for what Alex was like, this is what we need to go do. 614 00:33:29,760 --> 00:33:33,007 So I don't know the answer of like how big is a pre-training team, you know, 615 00:33:33,090 --> 00:33:34,677 it's, it doesn't need to be that big. 616 00:33:34,760 --> 00:33:38,470 But I think if you look at, you know, some of the other labs, um, it's 617 00:33:38,560 --> 00:33:39,937 probably quite big in some other places. 618 00:33:40,020 --> 00:33:44,437 So, um, I think part of the reason why, uh, these models are doing well is because 619 00:33:44,520 --> 00:33:48,777 you You have people who we're betting on and they have a lot of resources 620 00:33:48,860 --> 00:33:50,487 because we're taking our compute. 621 00:33:50,570 --> 00:33:53,657 We're not peanut buttering it across a lot of competing efforts. 622 00:33:53,740 --> 00:33:59,450 We're basically just investing in the key people, a lot of compute per researcher, 623 00:33:59,870 --> 00:34:01,750 and then they can do their work. 624 00:34:02,250 --> 00:34:04,750 Was Wonder the one big training effort? 625 00:34:04,960 --> 00:34:06,627 That was where most of the training was? 626 00:34:06,710 --> 00:34:09,217 It was the next pre-train. It was climbing up this ladder. 627 00:34:09,300 --> 00:34:11,967 So, I mean, we have this tiering system and then kind of Spark. 628 00:34:12,050 --> 00:34:14,507 I mean, internally, like, Avocado was the code name 629 00:34:14,590 --> 00:34:18,470 for the Muse Spark line, and then Watermelon was the code name for this one. 630 00:34:18,640 --> 00:34:20,437 And we're, like, out of fruits now. We're out of fruits. 631 00:34:20,520 --> 00:34:24,390 Yeah, so we're going to have to switch to a, to a new convention. 632 00:34:24,730 --> 00:34:28,440 But, um, I think the only fruit that is technically larger is the durian, 633 00:34:28,860 --> 00:34:30,357 and you don't want to go to the durian. No, no, no, no. 634 00:34:30,440 --> 00:34:31,987 Yeah, yeah, yeah. No, you want to stay far away. 635 00:34:32,070 --> 00:34:34,320 A durian is bigger than a watermelon? 636 00:34:34,450 --> 00:34:36,117 I think it's definitely heavier. That's horrifying. 637 00:34:36,200 --> 00:34:38,700 Definitely heavier. Spikier. Spikier, yeah. 638 00:34:39,370 --> 00:34:44,410 But no, yeah, I think as Mark mentioned, like, you know, each of the— I mean, 639 00:34:44,500 --> 00:34:48,417 each of the stages I'll describe, like, it's tens of people working on it 640 00:34:48,500 --> 00:34:50,590 at Meta versus, you know, hundreds at— 641 00:34:50,710 --> 00:34:54,137 So Meta is a huge company, but this is like a really tiny effort inside of Meta. 642 00:34:54,220 --> 00:34:56,387 Yeah. Oh yeah, I know. The research team is less than 200 people. 643 00:34:56,470 --> 00:35:00,887 Yeah, it's very small. It's intentionally very, very tight. I think that the— 644 00:35:00,970 --> 00:35:04,520 It's closer to 100, right? And then there's a bunch of InfoFolks. Yeah. 645 00:35:04,850 --> 00:35:06,017 It's a lot of infra work. 646 00:35:06,100 --> 00:35:11,270 So I think at a very high level, to go back to the, to the, to the what is 647 00:35:11,400 --> 00:35:13,237 it question, you know, how does it happen? 648 00:35:13,320 --> 00:35:15,657 How does model happen? That's basically what I'm asking. 649 00:35:15,740 --> 00:35:17,110 Where do models come from? 650 00:35:19,570 --> 00:35:24,160 You take a huge amount of data for pre-training, and here you're— 651 00:35:24,750 --> 00:35:28,500 And it's here, like, roughly, it's like all the, 652 00:35:28,960 --> 00:35:31,797 you know, public internet and like all the books you can get your hands 653 00:35:31,880 --> 00:35:34,177 on and just like as much data as possible. But different types. 654 00:35:34,260 --> 00:35:37,800 Plus the proprietary But also different modalities, right? 655 00:35:37,890 --> 00:35:40,267 So if you want it to be good at perception, then you want to make sure 656 00:35:40,350 --> 00:35:44,310 it has a lot of images and audio and video and this kind of stuff. 657 00:35:44,850 --> 00:35:47,100 Um, you know, if you want it to 658 00:35:47,770 --> 00:35:51,027 be good at certain domains, you don't just want to RL that in at the end. 659 00:35:51,110 --> 00:35:53,737 You want to have it have exposure to that from the beginning. 660 00:35:53,820 --> 00:35:57,860 Yeah. Yep. Yeah. So, so, and in this stage, you're kind of, you're much less, 661 00:35:58,280 --> 00:36:01,617 you care a lot less exactly what the data looks like or how it's formatted or 662 00:36:01,700 --> 00:36:03,550 You know, all of everything about it. 663 00:36:03,790 --> 00:36:05,787 But it's just lots and lots of data. 664 00:36:05,870 --> 00:36:08,920 When you talk about this being the best personal agent model, 665 00:36:09,210 --> 00:36:12,337 was there data you had to get for the pre-train where it's like huge 666 00:36:12,420 --> 00:36:15,420 amounts of restaurant reservation, call booking? 667 00:36:15,510 --> 00:36:16,717 I don't know what it is, but like— 668 00:36:16,800 --> 00:36:20,140 You just kind of want to dump in large volumes of the modalities 669 00:36:20,260 --> 00:36:21,467 that people are going to care about. 670 00:36:21,550 --> 00:36:22,767 I mean, there were a bunch of things. 671 00:36:22,850 --> 00:36:25,187 I mean, I don't actually think we talked about this when we were talking 672 00:36:25,270 --> 00:36:27,600 about where Muse Wonder excels. 673 00:36:27,890 --> 00:36:31,227 But some of the areas that it's like, all right, what do people care about, right? 674 00:36:31,310 --> 00:36:34,357 It's, I mean, there are things like shopping that people care about a lot, 675 00:36:34,440 --> 00:36:36,690 personal advice that people care about a lot. 676 00:36:36,900 --> 00:36:39,777 People have personal legal questions, personal health questions, 677 00:36:39,860 --> 00:36:44,490 personal tax questions. Muse Wonder, I mean, is like quite good at those. 678 00:36:44,580 --> 00:36:47,077 I don't know if there's any quantification that we want to give, 679 00:36:47,160 --> 00:36:51,830 but it's, let's say it's— It's very good on those personal advice questions. 680 00:36:51,920 --> 00:36:55,337 Yeah, and so those types of things, I mean, you can RL them to some degree, 681 00:36:55,420 --> 00:36:58,420 but you're like, just want to put in a large amount of data. 682 00:36:58,800 --> 00:37:01,257 Like, you can imagine health is a very— there's a lot of data 683 00:37:01,340 --> 00:37:05,347 out there about health. And, um, in, in pre-training, you just wanted to like see 684 00:37:05,430 --> 00:37:06,597 all these different things. 685 00:37:06,680 --> 00:37:11,020 And, um, so you, you kind of just put these different corpora in. 686 00:37:11,650 --> 00:37:17,190 Yeah. So, so that— so as, um, as Mark was mentioning, like, the, uh, we, 687 00:37:18,360 --> 00:37:22,487 uh, one of the highlights of Muse Wonder actually is its ability on domain 688 00:37:22,570 --> 00:37:24,117 various domain expertises. Yes. 689 00:37:24,200 --> 00:37:29,117 And I think our philosophy here is that, you know, the best personal agent should 690 00:37:29,200 --> 00:37:31,127 feel like a team of experts in your corner. 691 00:37:31,210 --> 00:37:33,960 Like, it should feel like it is your legal team. 692 00:37:34,170 --> 00:37:38,050 Yes. And your finance team, and your medical team, and, and, and, and, and. 693 00:37:38,420 --> 00:37:42,047 And so, um, on a bunch of these domains, we, you know, Muse Wonder is actually 694 00:37:42,130 --> 00:37:45,557 state of the art on, I think, um, a lot of the finance evals, as well 695 00:37:45,640 --> 00:37:48,100 as a lot of the, um, uh, legal evals. 696 00:37:48,640 --> 00:37:50,020 the state-of-the-art model. 697 00:37:50,140 --> 00:37:53,567 And that, you know, these, these domain areas, that starts in pre-training. 698 00:37:53,650 --> 00:37:57,690 Those, you know, we look for a lot of domain data to put into pre-training. 699 00:37:57,820 --> 00:38:01,360 Yes. But a lot of the, a lot of the art of pre-training— I mean, 700 00:38:01,650 --> 00:38:05,487 pre-training is, um, there's this one Rune tweet where it's like pre-training is full 701 00:38:05,570 --> 00:38:08,867 of the researchers who like chew glass and stare out into the distance 702 00:38:08,950 --> 00:38:10,917 and into the, into the abyss or whatnot. 703 00:38:11,000 --> 00:38:14,200 But, but I think there's a truth to that because it is like, um, 704 00:38:14,380 --> 00:38:17,960 It is both the highest stakes of the research areas because you, 705 00:38:18,460 --> 00:38:22,797 you know, we invested a huge percentage of our compute into training Muse Wonder, 706 00:38:22,880 --> 00:38:24,307 into pre-training Muse Wonder. 707 00:38:24,390 --> 00:38:26,930 And it was very high stakes. 708 00:38:27,050 --> 00:38:30,680 I think probably both of us at various points in that process, you know— 709 00:38:30,770 --> 00:38:32,187 You went through troughs of sadness. 710 00:38:32,270 --> 00:38:36,150 But yeah, troughs of uncertainties, you know, all the 7 stages of grief, 711 00:38:36,360 --> 00:38:38,980 you know, happen while you're pre-training a model. 712 00:38:39,070 --> 00:38:44,320 And it is the most sort of like fundamentally scientific in an interesting 713 00:38:44,410 --> 00:38:47,510 way, because, you know, you just don't have very many samples. 714 00:38:47,910 --> 00:38:52,330 So, like, your ability to infer exactly what will happen scientifically 715 00:38:52,700 --> 00:38:54,747 really depends on your ability to sort of like, 716 00:38:54,830 --> 00:38:56,837 you know, big brain reason about everything that's 717 00:38:56,920 --> 00:38:58,297 happening in this process. Okay. 718 00:38:58,380 --> 00:39:01,830 So in the pre-training stage, everyone who's training frontier models 719 00:39:02,260 --> 00:39:05,297 basically is trying to get as much data as possible for the pre-train 720 00:39:05,380 --> 00:39:09,637 with maybe a bent towards If you want Muse Wonder to excel at legal 721 00:39:09,720 --> 00:39:12,097 and finance and tax questions and things like this, 722 00:39:12,180 --> 00:39:15,940 then you want to have as much of that specific corpus as 723 00:39:16,060 --> 00:39:17,277 possible in the pre-train. 724 00:39:17,360 --> 00:39:20,067 Yeah, you still want as much intentionality about where ultimately you 725 00:39:20,150 --> 00:39:23,277 want your model to be, but again, it just, you know, benefits from a lot more data. 726 00:39:23,360 --> 00:39:28,660 Yeah. And this is sort of the massive data, massive compute stage. 727 00:39:29,030 --> 00:39:32,577 Do you hit some saturation in the data you put in the pre-train, or ultimately 728 00:39:32,660 --> 00:39:34,290 is it just more data is better? 729 00:39:35,290 --> 00:39:37,707 Well, there's a relationship between if you want 730 00:39:37,790 --> 00:39:39,940 the model to be bigger, you need more data. 731 00:39:40,340 --> 00:39:44,257 Otherwise, it overfits. Yes, but my point is, like, is there data where it's like, 732 00:39:44,340 --> 00:39:47,347 we can't distract the model with that, we can't clutter up its brain attic 733 00:39:47,430 --> 00:39:50,430 with knowing that stuff, or is it just more data is better? 734 00:39:50,800 --> 00:39:52,970 There are some of these effects, I think. 735 00:39:53,060 --> 00:39:54,727 Yeah, because this is part of the science. 736 00:39:54,810 --> 00:39:56,147 Okay, we're getting into the secret sauce, sorry. 737 00:39:56,230 --> 00:39:58,817 No, it's not even that, it's just like, I mean, I think that there are all these 738 00:39:58,900 --> 00:40:01,107 questions about, like, okay, well, we put a lot of these tokens in, 739 00:40:01,190 --> 00:40:03,277 is there any zero-sumness in a bunch of places there isn't, 740 00:40:03,360 --> 00:40:08,407 or it's actually positive transfer, but then like depending on how it's set 741 00:40:08,490 --> 00:40:10,617 up, like you can have some negative transfer. 742 00:40:10,700 --> 00:40:12,997 Yes. And so I don't know, there are all these interesting questions. 743 00:40:13,080 --> 00:40:18,250 But in general, one of the bigger challenges of scaling bigger models 744 00:40:18,790 --> 00:40:21,740 is it's not just about how many flops and how much compute. 745 00:40:22,290 --> 00:40:24,750 You want to be able to scale data with that. 746 00:40:24,840 --> 00:40:28,590 I mean, this is like part of what, when the original kind of scaling laws 747 00:40:28,680 --> 00:40:31,557 for pre-training were written, I mean, this is sort of what came 748 00:40:31,640 --> 00:40:35,560 out of that is is that. So if the internet isn't growing that quickly 749 00:40:36,310 --> 00:40:39,307 and you want to scale bigger models that can be more intelligent, 750 00:40:39,390 --> 00:40:41,937 I think one of the interesting problems is, all right, where's 751 00:40:42,020 --> 00:40:42,987 the data coming from? Yes. 752 00:40:43,070 --> 00:40:48,110 So I think that probably gets to the end of what we're going to talk about on that. 753 00:40:48,280 --> 00:40:54,700 But we could go to the other— Excellent CEO energy over here. 754 00:40:55,200 --> 00:41:00,287 But so post-training, so then I think Post-training is kind of— is 755 00:41:00,370 --> 00:41:04,040 a very exciting field now, I think, and is an area where 756 00:41:05,250 --> 00:41:08,047 I think actually, I think almost on a continual basis, 757 00:41:08,130 --> 00:41:10,927 the entire field is learning about the optimal recipes to be able 758 00:41:11,010 --> 00:41:12,550 to post-train these models. 759 00:41:12,890 --> 00:41:16,307 And again, for people, like, post-training is where you give the model 760 00:41:16,390 --> 00:41:19,187 a task that you expect it to do, grade it on how it does. 761 00:41:19,270 --> 00:41:21,977 And so if it's a chess-playing model, it plays chess games, learns the— Well, 762 00:41:22,060 --> 00:41:25,647 there's actually 2 stages. There's the SFT, So there's supervised learning. 763 00:41:25,730 --> 00:41:30,400 So that's like a lot of when you take this kind of raw thing and you give it both 764 00:41:31,450 --> 00:41:33,697 the question and the answer that you would want it to have, right? 765 00:41:33,780 --> 00:41:35,697 If like, if you're having a chat conversation, how would you 766 00:41:35,780 --> 00:41:37,037 want it to respond in this case? 767 00:41:37,120 --> 00:41:41,120 So when you're training things like personality, the SFT is actually like 768 00:41:41,210 --> 00:41:43,210 a very important part for molding it. 769 00:41:43,670 --> 00:41:47,370 But then I think most of the advances in the recent years have been in RL. 770 00:41:47,760 --> 00:41:48,593 Yeah. Yeah. 771 00:41:48,710 --> 00:41:52,630 So I think one mental model that can be helpful is like, And SFT is where 772 00:41:52,720 --> 00:41:55,217 you kind of initialize a lot of— SFT is supervised fine-tuning. 773 00:41:55,300 --> 00:41:56,517 Supervised fine-tuning, yeah. 774 00:41:56,600 --> 00:41:59,557 Where you initialize a lot of the behaviors that you want the model 775 00:41:59,640 --> 00:42:01,390 to be able to exhibit later on. 776 00:42:01,730 --> 00:42:04,650 And then, you know, maybe at the end of SFT, 777 00:42:05,110 --> 00:42:08,697 it has those behaviors kind of baked in, but it'll only do it like 10% of the time. 778 00:42:08,780 --> 00:42:13,860 And then in reinforcement learning, you really sort of give the model 779 00:42:14,320 --> 00:42:18,490 reinforcement in various situations to bring the elicitation of those 780 00:42:18,580 --> 00:42:19,880 behaviors dramatically up. 781 00:42:21,120 --> 00:42:24,580 And I think that the— yeah, so I think there's a bunch 782 00:42:24,670 --> 00:42:28,917 of interesting things that are happening in RL and post-training. 783 00:42:29,000 --> 00:42:33,130 I think one is that just the scaling of RL and post-training has been 784 00:42:34,470 --> 00:42:37,887 one of the clear axes, just spending more compute and gathering more data. 785 00:42:37,970 --> 00:42:40,770 Just doing much more of it before you release a model. 786 00:42:40,970 --> 00:42:44,767 Yes, doing much more of it, especially as the pre-trains scale up, doing much more. 787 00:42:44,850 --> 00:42:47,987 It's like anything in scaling laws, like it has to go with the rest 788 00:42:48,070 --> 00:42:50,860 of the factors or it doesn't work. Yeah, in general. 789 00:42:51,280 --> 00:42:53,947 I mean, if I were to give, say, like one broad statement about AI, 790 00:42:54,030 --> 00:42:55,777 it's just scale everything, you know? 791 00:42:55,860 --> 00:42:58,907 But, um, uh, but in particular, you have to scale everything. 792 00:42:58,990 --> 00:43:01,660 Like scaling one thing in isolation does not work. 793 00:43:02,250 --> 00:43:05,960 You, you technically can, you can scale one dimension. 794 00:43:06,420 --> 00:43:08,627 You just will get much more optimal performance if 795 00:43:08,710 --> 00:43:10,167 you scale many things altogether. 796 00:43:10,250 --> 00:43:13,177 Well, I mean, so one way to think about like Muse Wonder, right? 797 00:43:13,260 --> 00:43:14,637 It's like, all right, so we did the pre-train. 798 00:43:14,720 --> 00:43:16,757 Yep. we've done some RL, right? 799 00:43:16,840 --> 00:43:18,177 When you look at it like Muse Spark, right? 800 00:43:18,260 --> 00:43:22,980 So version 1 to 1.1, 1.2, 1.3, we didn't change anything about pre-training, right? 801 00:43:23,220 --> 00:43:25,640 We just continued doing more RL on top of it. 802 00:43:25,810 --> 00:43:29,770 So, um, so you, when you think about the early versions 803 00:43:30,020 --> 00:43:33,900 of a model of a new class, they're actually relatively less, 804 00:43:34,360 --> 00:43:37,660 like the RL to pre-training ratio is going to be lower for RL 805 00:43:38,320 --> 00:43:42,207 than the later versions. That is kind of what the point releases in models are. 806 00:43:42,290 --> 00:43:44,417 Well, I think every company has its own convention. 807 00:43:44,500 --> 00:43:45,577 For us, that's what it is meant to be. 808 00:43:45,660 --> 00:43:49,167 But for a lot of companies, like the big number is the pre-train and then the— 809 00:43:49,250 --> 00:43:51,450 I think it kind of lost that semantic value. 810 00:43:53,090 --> 00:43:54,217 Yeah, I think now you actually can't do that. 811 00:43:54,300 --> 00:43:56,047 Okay, but that is true at Meta at least. 812 00:43:56,130 --> 00:43:57,217 But we haven't even done it. 813 00:43:57,300 --> 00:43:59,677 We kind of decided to not even do it. It's true at Muse Spark at least. 814 00:43:59,760 --> 00:44:01,347 It is true at Muse Spark and Muse Wonder. 815 00:44:01,430 --> 00:44:03,427 It's true that Muse Spark and Muse Wonder are different pre-trains. 816 00:44:03,510 --> 00:44:06,647 And we decided to kick the can down the road on this question for Muse Wonder 817 00:44:06,730 --> 00:44:08,600 by not giving it a number yet. 818 00:44:08,770 --> 00:44:12,170 But when we have our first dot release, we will have to decide that. 819 00:44:13,900 --> 00:44:17,280 Yeah. But yeah, no, I think the RL stage, part of what has been— 820 00:44:18,610 --> 00:44:23,117 part of what's very interesting is first is just producing the environments 821 00:44:23,200 --> 00:44:27,120 and producing the tasks. And I think that is one of the areas where there's 822 00:44:27,210 --> 00:44:29,337 a lot of intentionality in training these models. 823 00:44:29,420 --> 00:44:33,000 It's also one of the areas where you notice very funny things. 824 00:44:33,090 --> 00:44:38,800 Like if you— in a lot of cases, less is actually more when it comes to data in RL. 825 00:44:39,390 --> 00:44:44,180 And so I think I think the sort of, like, the feel and the science of it, like, 826 00:44:44,760 --> 00:44:48,477 is pretty flipped, generally speaking, from pre-training. 827 00:44:48,560 --> 00:44:54,187 Yeah. The other is that I think it can give very magnified kind of instruction 828 00:44:54,270 --> 00:44:55,147 to the model, if that makes sense. 829 00:44:55,230 --> 00:44:58,527 So I think that, like, a lot of safety cases or alignment cases 830 00:44:58,610 --> 00:45:01,027 that we care a lot about are, like, you know, and, you know, 831 00:45:01,110 --> 00:45:05,010 where you would see when you do large high-compute RL runs on frontier models, 832 00:45:05,280 --> 00:45:11,037 are where you apply a lot of RL or optimization pressure in scenarios where 833 00:45:11,120 --> 00:45:15,250 the only way for the model to succeed is to do something misaligned 834 00:45:15,380 --> 00:45:16,530 or do something unsafe. 835 00:45:16,760 --> 00:45:19,847 And so that's another thing that we had to be very careful about and thoughtful 836 00:45:19,930 --> 00:45:24,030 about in training Muse Wonder is ensuring that we curated the RL environments very 837 00:45:24,680 --> 00:45:28,347 thoughtfully to ensure that all of them were eliciting and driving 838 00:45:28,430 --> 00:45:30,557 towards more aligned behavior from the model. 839 00:45:30,640 --> 00:45:32,607 So the point that you're saying about quality, I think, 840 00:45:32,690 --> 00:45:36,147 is an important one because I think a lot of the previous questions back and forth 841 00:45:36,230 --> 00:45:38,567 were about the quantity and have the flavor of like, oh, 842 00:45:38,650 --> 00:45:40,067 you need to scale the quantity. Yes. 843 00:45:40,150 --> 00:45:43,150 But I mean, I do think like a huge part of this is just, is, 844 00:45:43,530 --> 00:45:47,660 is the point that you just made of like the even a smaller amount of high quality. 845 00:45:48,240 --> 00:45:52,080 Yeah, I think, I think in RL, like we, you know, as we've seen 846 00:45:52,670 --> 00:45:57,710 pretty big jumps in our new Spark line, I think these have been like accompanied 847 00:45:57,800 --> 00:46:02,300 with, you know, real meaningful improvements to the underlying recipe. 848 00:46:02,510 --> 00:46:05,510 And each of these improvements in the recipe have been like, 849 00:46:06,350 --> 00:46:11,477 I think the improvement between Muse Spark 1 to Muse Spark 1.3 feels night and day. 850 00:46:11,560 --> 00:46:14,900 I think probably to most people, if you ask them, 851 00:46:16,690 --> 00:46:18,197 do these feel like they came from different pretrains? 852 00:46:18,280 --> 00:46:21,617 I think they probably would say yes, because I think you can— Mm-hmm. 853 00:46:21,700 --> 00:46:24,197 I think it's happening industry-wide for what it's worth. 854 00:46:24,280 --> 00:46:29,700 It's just like we are learning to squeeze so much capability and so much 855 00:46:30,500 --> 00:46:35,210 and elicit so much agentic capability from these existing base models. 856 00:46:35,330 --> 00:46:38,837 Now, that's not to say, like, we believe scaling is very important. 857 00:46:38,920 --> 00:46:41,137 That's why we have to use Wonderland in the first place. 858 00:46:41,220 --> 00:46:44,557 Yes. But, you know, this sort of like, these two things are happening at the same 859 00:46:44,640 --> 00:46:48,847 time, which is also, by the way, for us, this is why we're seeing so 860 00:46:48,930 --> 00:46:49,977 much progress in general. 861 00:46:50,060 --> 00:46:53,517 Like, I think it's the fact that we're scaling both pre-training and scaling our 862 00:46:53,600 --> 00:46:56,860 recipes in RL kind of like concurrently. At a nice clip. 863 00:46:57,360 --> 00:46:59,737 Yeah. And then there's these other scaling axes too. 864 00:46:59,820 --> 00:47:03,027 I mean, like making sure that your agent harness is good and that you can 865 00:47:03,110 --> 00:47:06,200 co-optimize these things is probably just another aspect of, 866 00:47:06,660 --> 00:47:08,120 or dimension for scaling. 867 00:47:08,740 --> 00:47:12,117 But if I'm understanding you guys correctly, are you saying that alignment 868 00:47:12,200 --> 00:47:16,250 is not about coming up with some alignment-specific RL 869 00:47:16,380 --> 00:47:20,380 steps and instead about making sure the entire RL process 870 00:47:20,960 --> 00:47:23,380 is kind of steering The model— 871 00:47:23,970 --> 00:47:26,467 You have to be thoughtful about it in the entire process of RL. 872 00:47:26,550 --> 00:47:27,347 Yeah. 873 00:47:27,430 --> 00:47:30,057 So I think this is a holistic effort, I think. 874 00:47:30,140 --> 00:47:33,540 Yeah. But also, I think different labs have very different theories. 875 00:47:33,930 --> 00:47:36,767 I mean, alignment is a very philosophical concept, right? 876 00:47:36,850 --> 00:47:38,727 Of like, who do you want it to be aligned to? 877 00:47:38,810 --> 00:47:42,980 I mean, a lot of our theory is like, okay, we care more about it being aligned to, 878 00:47:43,070 --> 00:47:46,197 you know, we're trying to give people agents, right? 879 00:47:46,280 --> 00:47:48,630 And it needs to, if you ask it to do something, 880 00:47:49,030 --> 00:47:52,957 you need to have a high confidence that it's going to do what you ask it to do. 881 00:47:53,040 --> 00:47:55,827 Yes. You know, like, you want it to do that with high reliability. 882 00:47:55,910 --> 00:47:58,920 You don't want a moralizing lecture for why you're wrong 883 00:47:59,130 --> 00:48:00,457 for asking it to do the thing. 884 00:48:00,540 --> 00:48:02,877 Obviously, if you're trying to do something that's catastrophic, 885 00:48:02,960 --> 00:48:06,760 it needs to gracefully, without pissing you off, like, not do it. 886 00:48:08,090 --> 00:48:10,640 But in general, you know, we view alignment as more 887 00:48:11,180 --> 00:48:15,890 of a question of instruction and values alignment with the person 888 00:48:15,980 --> 00:48:17,380 who it's intending to serve. 889 00:48:18,980 --> 00:48:22,980 And I think that that's kind of a different mode of thinking about it. 890 00:48:23,070 --> 00:48:25,947 It's very different from, I think, how some of the others do. 891 00:48:26,030 --> 00:48:30,907 But one of the more interesting things I think that is maybe a question related 892 00:48:30,990 --> 00:48:34,580 to your original question is also just how long the process of training 893 00:48:34,740 --> 00:48:35,577 one of these models takes. 894 00:48:35,660 --> 00:48:41,330 Like Muse Wonder, I mean, in general, we've planned Muse Wonder since 895 00:48:42,170 --> 00:48:43,800 last You know, August. 896 00:48:44,380 --> 00:48:48,297 Yeah, so yeah, I remember the offsite when we were— so this is like when 897 00:48:48,380 --> 00:48:49,387 they first presented this. 898 00:48:49,470 --> 00:48:54,097 Yeah, so I think all of the stages that we just talked about are like, you know, you 899 00:48:54,180 --> 00:48:55,887 really work on them for quite a long time. 900 00:48:55,970 --> 00:48:59,767 And, and, um, it's not that we, we didn't start working on Muse Wonder 901 00:48:59,850 --> 00:49:01,187 when we finished Muse Spark. 902 00:49:01,270 --> 00:49:03,937 It's that we've actually just been planning and working on this 903 00:49:04,020 --> 00:49:05,650 for quite a long time. 904 00:49:06,150 --> 00:49:09,447 And we really view Muse Spark, if anything, as like I think we mentioned 905 00:49:09,530 --> 00:49:12,130 this in the first blog post where we wrote about it. 906 00:49:12,240 --> 00:49:15,580 That was a stepping stone to get to Muse Wonder. 907 00:49:16,370 --> 00:49:17,367 And that really was how we thought about it. 908 00:49:17,450 --> 00:49:18,667 Even at the time you thought about it. 909 00:49:18,750 --> 00:49:22,207 Yeah, and in terms of how long it takes, I mean, we're also even just 910 00:49:22,290 --> 00:49:23,507 talking about the research part. 911 00:49:23,590 --> 00:49:27,840 I mean, there's also the compute part where we've been investing in building up, 912 00:49:28,090 --> 00:49:29,757 first of all, a huge amount of compute overall. 913 00:49:29,840 --> 00:49:32,940 But a lot of, you know, part of our philosophy on this is that 914 00:49:33,720 --> 00:49:38,597 you can kind of measure the compute strength of a lab by how big 915 00:49:38,680 --> 00:49:43,730 its largest single computer is, supercomputer, for being able to do these 916 00:49:44,360 --> 00:49:45,067 large training runs. 917 00:49:45,150 --> 00:49:48,780 So, I mean, we built this Prometheus cluster, we call it, in Ohio. 918 00:49:49,030 --> 00:49:52,610 Mm-hmm. That's scaling to larger than a gigawatt. 919 00:49:53,110 --> 00:49:56,247 And now we're building this Hyperion cluster in Louisiana that's 920 00:49:56,330 --> 00:49:57,910 going to scale to 5 gigawatts. 921 00:49:58,870 --> 00:50:03,540 And those actually, predate MSL. 922 00:50:05,000 --> 00:50:08,550 Because like building a whole like data center site from scratch, 923 00:50:09,210 --> 00:50:11,387 at least the plans for those, they're coming online now. 924 00:50:11,470 --> 00:50:15,137 But that was— You guys thought it might be the feed, right? 925 00:50:15,220 --> 00:50:17,477 That you just have used for a lot of compute generally? 926 00:50:17,560 --> 00:50:18,393 No, no, no. 927 00:50:18,850 --> 00:50:23,100 This is kind of predating, even going back to the, to kind of the LLaMA program. 928 00:50:23,190 --> 00:50:27,440 I see. I mean, it was probably in 2023. 929 00:50:29,480 --> 00:50:35,407 or yeah, either 2023, 2024 was when we basically started making 930 00:50:35,490 --> 00:50:36,697 the plans to build those clusters. 931 00:50:36,780 --> 00:50:39,490 They're now kind of fully coming online. 932 00:50:39,790 --> 00:50:43,370 Well, I guess the Louisiana one will come online over a couple years. 933 00:50:44,000 --> 00:50:44,833 Yeah. 934 00:50:45,620 --> 00:50:49,547 But if you want to go really end-to-end on undoing this stuff, then, I mean, 935 00:50:49,630 --> 00:50:54,050 obviously the kind of full stack, and I mean, Meta is a full-stack tech company. 936 00:50:54,180 --> 00:50:58,427 I mean, we build our own data centers, we build our own chips, 937 00:50:58,510 --> 00:51:00,177 we build our own low-level infrastructure. 938 00:51:00,260 --> 00:51:03,017 And now this is kind of the next component, which, I mean, 939 00:51:03,100 --> 00:51:06,147 arguably is probably the most important component in all of this for the next wave 940 00:51:06,230 --> 00:51:09,607 of technology is kind of building really good models and then 941 00:51:09,690 --> 00:51:10,567 delivering them to people. 942 00:51:10,650 --> 00:51:13,700 But I mean, yeah, if you want to go fully end-to-end, I mean, 943 00:51:13,860 --> 00:51:16,530 this is a long process. 944 00:51:16,860 --> 00:51:20,950 I do think this is like a— I mean, maybe it's well appreciated by folks 945 00:51:21,040 --> 00:51:24,997 in the tech industry, but It really is, you know, it feels like models 946 00:51:25,080 --> 00:51:25,877 are coming out constantly. 947 00:51:25,960 --> 00:51:28,757 So maybe you'd have the mental model that like, oh, a company like launches 948 00:51:28,840 --> 00:51:31,627 a model and then they figure out how they're going to make the next model and, 949 00:51:31,710 --> 00:51:32,337 or something like that. 950 00:51:32,420 --> 00:51:35,800 But it's actually these like very long planning processes and these 951 00:51:36,220 --> 00:51:39,967 sort of like very long work streams that kind of all come together at the very 952 00:51:40,050 --> 00:51:41,977 end and then result in these beautiful models. 953 00:51:42,060 --> 00:51:46,850 Yeah. How do you build good products when the feedback cycle is so long? 954 00:51:47,060 --> 00:51:49,227 Like ultimately you start planning the model in August and you don't 955 00:51:49,310 --> 00:51:51,710 know until more than a year later. if it's good. 956 00:51:51,940 --> 00:51:55,567 And, you know, Silicon Valley famously likes to run on metrics, but, like, 957 00:51:55,650 --> 00:51:58,570 ultimately, do you just need someone who 958 00:51:59,570 --> 00:52:02,327 is the aesthete and says, I'm sorry, I don't like it? 959 00:52:02,410 --> 00:52:04,367 It's nice when you have metrics, but I think the, like, 960 00:52:04,450 --> 00:52:08,667 state of being an entrepreneur is, like, how well do you make decisions 961 00:52:08,750 --> 00:52:11,290 in an information-poor environment, right? So— 962 00:52:11,630 --> 00:52:13,257 But this is, like, that on steroids, right? 963 00:52:13,340 --> 00:52:14,337 That on mega steroids. 964 00:52:14,420 --> 00:52:17,590 Yeah, I think that this is— and I think Marc gets a huge amount 965 00:52:17,680 --> 00:52:21,637 of credit for this, like, Because of how long everything takes, 966 00:52:21,720 --> 00:52:27,097 like, there were many points along over the past year since I've been here, 967 00:52:27,180 --> 00:52:30,477 but also, you know, as you mentioned, like, long before that, where 968 00:52:30,560 --> 00:52:34,230 you have to make very large investments or make very big 969 00:52:34,320 --> 00:52:38,527 decisions about, you know, how you're going to invest or develop 970 00:52:38,610 --> 00:52:41,810 the technology or, you know, what decisions you're going to make 971 00:52:41,950 --> 00:52:47,247 long before there's proof points around whether or not even the thing is working. 972 00:52:47,330 --> 00:52:51,040 And so I think this is like, in some sense, it's like 973 00:52:51,130 --> 00:52:52,207 entrepreneurship on hard mode. 974 00:52:52,290 --> 00:52:53,587 It's like the opposite of Lean Startup. 975 00:52:53,670 --> 00:52:56,177 Like, Lean Startup is like, yeah, you like see if you get something working 976 00:52:56,260 --> 00:52:58,597 and then you double down and then you see if that works, then you double down. 977 00:52:58,680 --> 00:53:00,260 Like, you cannot do that in AI. 978 00:53:00,720 --> 00:53:05,307 You have to like weave— I mean, we've been investing into these data 979 00:53:05,390 --> 00:53:07,597 centers since, you know, for years and years and years. 980 00:53:07,680 --> 00:53:08,477 We've had to— 981 00:53:08,560 --> 00:53:11,777 And you'll be radically behind if you wait to see if things are working. 982 00:53:11,860 --> 00:53:15,857 So we have these conversations where it's like, At the end of last year, we're like, 983 00:53:15,940 --> 00:53:19,237 all right, we are quite excited about the progress. 984 00:53:19,320 --> 00:53:21,027 Yeah, we haven't seen the models yet. 985 00:53:21,110 --> 00:53:23,947 Like, it's like, I'm going to, like, you know, we're going to write this check 986 00:53:24,030 --> 00:53:27,407 for, like, I don't know, like $100 billion, $200 billion. 987 00:53:27,490 --> 00:53:30,330 I'm like, Alex, are you sure this is going to work? 988 00:53:31,000 --> 00:53:35,667 It's like, um, yeah, so it was, it was so extreme that, you know, 989 00:53:35,750 --> 00:53:37,507 we had, we had like the whole exec team. 990 00:53:37,590 --> 00:53:38,967 I think Susan scared you a bit. 991 00:53:39,050 --> 00:53:43,677 Yeah, well, yeah, no, well, it was like The joke every time we talked 992 00:53:43,760 --> 00:53:47,887 about it would just be like, yeah, and we hope it's all going to work and be good. 993 00:53:47,970 --> 00:53:50,807 But like, there were literally, like, at the end of last year, other than 994 00:53:50,890 --> 00:53:54,397 me giving a vibe report on how our research was going and, you know, 995 00:53:54,480 --> 00:53:58,820 I could sort of show some plots and whatnot, but it was like, I mean, 996 00:53:59,320 --> 00:54:01,617 I was feeling good about it, but it was certainly not clear 997 00:54:01,700 --> 00:54:03,237 at all that it was going to work. 998 00:54:03,320 --> 00:54:08,497 Yeah. You guys are saying that the essence of training a frontier model is you have 999 00:54:08,580 --> 00:54:12,120 to push your chips all in before you've been dealt the cards. 1000 00:54:12,290 --> 00:54:16,380 You just have to decide that you're going to develop a frontier model 1001 00:54:16,540 --> 00:54:19,340 and you don't get to meter the inputs. At this scale. 1002 00:54:19,840 --> 00:54:22,547 I mean, if we, like, back a few years ago, the clusters didn't have to be 1003 00:54:22,630 --> 00:54:28,140 this big, but now they do. So I think this is part of the excitement 1004 00:54:28,970 --> 00:54:32,520 of booting up a frontier lab now, right? 1005 00:54:33,060 --> 00:54:35,690 Is basically like the ante is extremely high. 1006 00:54:36,110 --> 00:54:40,357 And I think this is— and like, I kind of like viscerally remember this. 1007 00:54:40,440 --> 00:54:42,157 We were in Hawaii at the end of last year. 1008 00:54:42,240 --> 00:54:44,827 It was kind of like the whole exec team got together. 1009 00:54:44,910 --> 00:54:48,160 And yeah, Susan pulled me aside many times and was like, 1010 00:54:48,620 --> 00:54:52,167 you know, I just want to tell you, if this doesn't work, like, you know, 1011 00:54:52,250 --> 00:54:53,997 this is going to be really, really bad. 1012 00:54:54,080 --> 00:54:57,380 It's going to be very, very bad in financial terms. 1013 00:54:58,420 --> 00:55:01,420 I really need to impress on you that this model has to work. 1014 00:55:01,720 --> 00:55:04,927 it has to be good. If it's good, it's all good. We're great. 1015 00:55:05,010 --> 00:55:07,967 But if it's bad, it's really, really, really bad. 1016 00:55:08,050 --> 00:55:10,017 And Susan, who you know— And that was useful motivation? 1017 00:55:10,100 --> 00:55:14,057 It was. Well, it was like, it was one of these very funny moments 1018 00:55:14,140 --> 00:55:16,817 where I was like, you know, obviously the amounts of money 1019 00:55:16,900 --> 00:55:21,857 are like so obscenely large. What are the amounts of money in training? 1020 00:55:21,940 --> 00:55:23,527 Order of magnitude, whatever you can share, but just 1021 00:55:23,610 --> 00:55:25,277 like how much are they costing? Well, you can do the classic math. 1022 00:55:25,360 --> 00:55:26,987 Yeah, no, I mean, a gigawatt cluster. 1023 00:55:27,070 --> 00:55:31,200 I mean, it's Like, I don't know, I think what you'd see out there would be, 1024 00:55:31,490 --> 00:55:35,370 you know, to operate a gigawatt a year, call it on the order of $15 billion. 1025 00:55:36,170 --> 00:55:40,090 Okay, so it's roughly a $15 billion effort to train a bunch of models. 1026 00:55:40,300 --> 00:55:42,350 To operate a gigawatt cluster for a year. 1027 00:55:42,670 --> 00:55:45,427 If you're talking about building the CapEx, it's probably closer to, 1028 00:55:45,510 --> 00:55:48,137 you know, I mean, you pay for it upfront, so it's probably, you know, 1029 00:55:48,220 --> 00:55:50,550 $50 or $60 billion. 1030 00:55:51,140 --> 00:55:57,900 So, but then that's just kind of, one cluster for one part of training. 1031 00:55:58,190 --> 00:56:00,730 Yeah. Um, then you have all the other parts. 1032 00:56:01,400 --> 00:56:04,650 Um, research isn't just doing the run, right? 1033 00:56:04,740 --> 00:56:07,390 It's like, that's like probably an equal amount of it 1034 00:56:07,860 --> 00:56:11,077 is because for each change that you make, you're testing all the ladders all 1035 00:56:11,160 --> 00:56:13,330 the way up to the big, the big scale. 1036 00:56:13,700 --> 00:56:15,790 Um, and then they're serving the stuff. 1037 00:56:16,160 --> 00:56:18,707 So, I mean, part of the stuff that in some ways— 1038 00:56:18,790 --> 00:56:21,840 So tens of billions of dollars to train a Frontier model now. 1039 00:56:23,210 --> 00:56:26,137 Yeah, definitely the whole, I mean, I've seen the financial rollups, 1040 00:56:26,220 --> 00:56:28,557 tens of billions of dollars is a good estimate for— Yeah. 1041 00:56:28,640 --> 00:56:30,597 And then, but then, but then you want to do something with it, right? 1042 00:56:30,680 --> 00:56:34,177 So I actually think like some of the more complicated questions 1043 00:56:34,260 --> 00:56:39,767 that we had were like, all right, so we're training this model and we know, 1044 00:56:39,850 --> 00:56:43,147 we know we want to train the model cuz we, this is like key, but like 1045 00:56:43,230 --> 00:56:44,857 how much product demand do we think we're going to have? 1046 00:56:44,940 --> 00:56:48,407 Right. Because so now it's like, all right, well the product doesn't exist. 1047 00:56:48,490 --> 00:56:51,867 Um, like, and as in you're trying to build data centers to serve Muse 1048 00:56:51,950 --> 00:56:53,407 for a product that doesn't exist yet. Yeah. 1049 00:56:53,490 --> 00:56:55,657 And we have to make it years in advance, right? 1050 00:56:55,740 --> 00:56:58,707 So this is probably where, like, when we were talking about it last year, 1051 00:56:58,790 --> 00:57:01,790 we were just like, huh, there was a moment where I think 1052 00:57:01,880 --> 00:57:05,627 a bunch of people were like, yeah, maybe, maybe we should just 1053 00:57:05,710 --> 00:57:07,337 like be a little more conservative. 1054 00:57:07,420 --> 00:57:09,257 And it's like, no, we're doing this. 1055 00:57:09,340 --> 00:57:11,097 It's like, it's like, we're going to make it work. 1056 00:57:11,180 --> 00:57:12,257 We're going to make it work. We're going to burn the boats. 1057 00:57:12,340 --> 00:57:13,467 Yeah, this is going to work. 1058 00:57:13,550 --> 00:57:19,097 And But it was like, I mean, even, I mean, I too am an entrepreneur 1059 00:57:19,180 --> 00:57:22,780 and founder and have, you know, at various points in my life taken risk. 1060 00:57:22,900 --> 00:57:26,950 I remember the visceral feeling at the time was like, this is a fuck ton of risk. 1061 00:57:27,490 --> 00:57:30,190 Like, I don't know if I really— Oh, it's only riskier. 1062 00:57:30,490 --> 00:57:31,527 Yeah, yeah, it's only— 1063 00:57:31,610 --> 00:57:37,450 well, I do think this is like one of the like things that I think, Mark, 1064 00:57:37,740 --> 00:57:40,077 is very impressive about you, which is I think you have really 1065 00:57:40,160 --> 00:57:45,090 internalized this concept of You know, it's about how big it is when you win, 1066 00:57:45,250 --> 00:57:46,927 not like, you know, failure is okay. 1067 00:57:47,010 --> 00:57:50,930 It's just like how big is the win when you do win? 1068 00:57:51,300 --> 00:57:52,137 And this was, I think— 1069 00:57:52,220 --> 00:57:55,347 Mark has fully internalized power laws in a way that many people have not. 1070 00:57:55,430 --> 00:57:56,387 I think quite deeply, yeah. 1071 00:57:56,470 --> 00:57:58,767 Well, I think the other piece is, I don't actually think it's 1072 00:57:58,850 --> 00:58:02,390 quite as risky as this. I mean, the numbers are big, but I think— 1073 00:58:02,480 --> 00:58:04,477 Meta has a backstop in terms of how it can use compute. 1074 00:58:04,560 --> 00:58:05,817 Yeah, so I think there's 2 things. 1075 00:58:05,900 --> 00:58:09,500 One is if AI works overall, plenty of other people want all our compute. 1076 00:58:09,650 --> 00:58:13,737 Right? So they're willing to pay, like, I mean, if like the cost is whatever, 1077 00:58:13,820 --> 00:58:16,920 $15 billion a gigawatt, we could get like a lot more for that. 1078 00:58:17,120 --> 00:58:19,707 So I think that you, in order for it not to work out, 1079 00:58:19,790 --> 00:58:23,080 it's not just that we have to fail, AI overall has to fail. 1080 00:58:23,250 --> 00:58:24,083 Yes. 1081 00:58:24,670 --> 00:58:29,047 And then even if that happens, we still have all these other 1082 00:58:29,130 --> 00:58:30,717 uses to productively use compute. 1083 00:58:30,800 --> 00:58:34,590 So in the grand scheme of things, we were definitely building out 1084 00:58:35,090 --> 00:58:39,970 this compute for inference ahead of knowing that the product 1085 00:58:40,060 --> 00:58:41,520 was going to be ready. 1086 00:58:41,930 --> 00:58:45,520 But you were AI-pilled enough that you said GPUs online and powered 1087 00:58:45,610 --> 00:58:48,607 will be useful to someone. Yeah, but also like we believed in what we were building. 1088 00:58:48,690 --> 00:58:54,070 I mean, it's like, I think, and you can look at the kind 1089 00:58:54,610 --> 00:58:57,737 of the trajectory where it's like, all right, first there's ChatGPT 1090 00:58:57,820 --> 00:58:59,240 as the text-based assistant. 1091 00:58:59,870 --> 00:59:02,540 Then, you know, Anthropic got coding agents to work. 1092 00:59:03,160 --> 00:59:07,130 And no one had built the kind of personal consumer version of that. 1093 00:59:07,500 --> 00:59:11,127 And it just seemed like this is the thing that Meta should build in the world. 1094 00:59:11,210 --> 00:59:14,467 And I think that that was just like pretty clear to you and me and Nat, 1095 00:59:14,550 --> 00:59:15,967 like from the beginning, right? 1096 00:59:16,050 --> 00:59:18,717 That that's like, that's kind of what we're here to do. 1097 00:59:18,800 --> 00:59:20,470 And so I don't know. 1098 00:59:20,680 --> 00:59:24,480 I mean, there's always like the real question for me on the Muse launch even 1099 00:59:25,140 --> 00:59:28,607 was you never know when you launch something how it's going to be received. 1100 00:59:28,690 --> 00:59:32,197 Right? So there's always the case where you get a home run, which in this 1101 00:59:32,280 --> 00:59:33,570 case it was, right? Yes. 1102 00:59:33,700 --> 00:59:36,237 And it's just like extremely well received right off the bat. 1103 00:59:36,320 --> 00:59:42,450 But the normal case would be you kind of— it's fine, and you have to like iterate 1104 00:59:42,540 --> 00:59:45,590 a few times to get product-market fit before you really grow. 1105 00:59:46,040 --> 00:59:48,127 In some ways, like, I actually think the product, 1106 00:59:48,210 --> 00:59:51,797 we maybe could have even grown it faster, but we didn't have all the growth stuff 1107 00:59:51,880 --> 00:59:54,757 really ready to go when we shipped it because we are like— our median 1108 00:59:54,840 --> 00:59:56,637 expectation was like, all right, we're probably going to have to like 1109 00:59:56,720 --> 00:59:59,177 iterate on this a bit, and like, that'll be That'll be fine. 1110 00:59:59,260 --> 01:00:03,267 But I don't know, I think like if it takes a few iterations, 1111 01:00:03,350 --> 01:00:05,267 it's like not the end of the world. That's normal. 1112 01:00:05,350 --> 01:00:08,187 Yes. Um, you can have launches where it's like you 1113 01:00:08,270 --> 01:00:10,817 get like, it's kind of catastrophic and then it takes a while to recover, 1114 01:00:10,900 --> 01:00:14,317 but that's not good. But, but it's like, but I mean, even if it took a few 1115 01:00:14,400 --> 01:00:15,907 iterations, it would have been fine. 1116 01:00:15,990 --> 01:00:19,827 So I don't know, I guess it just seemed to me like this was going to be— yeah, 1117 01:00:19,910 --> 01:00:22,697 the numbers were big, but I— it seemed like it was probably— 1118 01:00:22,780 --> 01:00:27,037 Was it bonding for the executive team to be Holding hands, 1119 01:00:27,120 --> 01:00:28,620 taking this risk together. 1120 01:00:28,710 --> 01:00:31,007 You're talking about Susan taking UXi? Bonding or bonding moment? 1121 01:00:31,090 --> 01:00:33,757 Is that a bonding moment? It's like, I mean, it's stressful. 1122 01:00:33,840 --> 01:00:35,960 I think it was—Both. Both. Yeah. 1123 01:00:36,050 --> 01:00:38,967 Being in a stressful situation together is generally bonding. 1124 01:00:39,050 --> 01:00:45,557 Well, one thing that I think is quite notable about Meta is the degree to which, 1125 01:00:45,640 --> 01:00:48,687 like, first of all, most of the exec team has been there quite a long time. 1126 01:00:48,770 --> 01:00:50,227 Like, Nat and I were really newbies. 1127 01:00:50,310 --> 01:00:57,190 And And, you know, most people have been through crazy situations many 1128 01:00:57,280 --> 01:00:59,110 times in the past as well. Mm-hmm. 1129 01:00:59,240 --> 01:01:02,907 And I think Meta as a company is like an extremely resilient company that has 1130 01:01:02,990 --> 01:01:07,617 been through like, you know, many situations that also at the time 1131 01:01:07,700 --> 01:01:09,327 would've seemed incredibly daunting. 1132 01:01:09,410 --> 01:01:13,337 And so everyone's kind of like knows the program, so to speak, 1133 01:01:13,420 --> 01:01:18,797 in terms of, you know, this obviously on paper, and especially when like 1134 01:01:18,880 --> 01:01:22,847 Susan Li, our CFO, is sort of like pulling me aside in the meetings, like, does, 1135 01:01:22,930 --> 01:01:26,177 you know, does seem like a particularly scary situation. 1136 01:01:26,260 --> 01:01:28,937 But I think most people were like, yeah, yeah, no, but we're Meta, 1137 01:01:29,020 --> 01:01:29,767 we're going to get through it. 1138 01:01:29,850 --> 01:01:34,230 Like, I think there was like a real camaraderie around that and sort of like 1139 01:01:34,320 --> 01:01:40,197 quite a nice energy around that, which really I did appreciate because I 1140 01:01:40,280 --> 01:01:45,410 think it would be very easy for, you know, it's the other executives at 1141 01:01:45,530 --> 01:01:49,577 the company who are operating and running— Who are paying the bills. 1142 01:01:49,660 --> 01:01:51,707 Yeah, one of the greatest businesses of all time. 1143 01:01:51,790 --> 01:01:52,623 Yes. 1144 01:01:53,170 --> 01:01:58,210 And it'd be very easy for them to be quite annoyed by this whole endeavor. 1145 01:01:58,300 --> 01:02:01,340 But I think everyone is also very excited about— They are. 1146 01:02:01,430 --> 01:02:04,550 What the opportunities for the company with AI are. 1147 01:02:04,680 --> 01:02:11,640 And I think, like, you know, taking a step back, I think one of Mark's jobs as, like, 1148 01:02:11,730 --> 01:02:16,610 the founder and CEO of the company is to ensure that the company is always 1149 01:02:17,400 --> 01:02:21,900 pushing towards the most ambitious and expansive version of what it can be. 1150 01:02:22,410 --> 01:02:28,450 And I think that one of the great things about this entire MSL and AI push is 1151 01:02:28,540 --> 01:02:31,910 that I think this is one of the rawest expressions of that is 1152 01:02:33,790 --> 01:02:39,337 we knew that AI certainly is a very important technology and it's one 1153 01:02:39,420 --> 01:02:40,757 of the bigger technologies right now. 1154 01:02:40,840 --> 01:02:45,177 But if we were able to build it effectively 1155 01:02:45,260 --> 01:02:49,430 and properly integrate it into everything that we do, as well as like, you know, 1156 01:02:49,600 --> 01:02:52,480 as a full-stack technology company, like build it into, 1157 01:02:53,230 --> 01:02:55,857 you know, these incredible products that we know will exist in the future, 1158 01:02:55,940 --> 01:02:58,340 this could be like immensely expansive for Meta. 1159 01:02:58,520 --> 01:03:04,740 And I think this is true. Like, I think Muse is an incredibly exciting product. 1160 01:03:05,030 --> 01:03:08,780 It has like, you know, it can go in 1,000 different directions. 1161 01:03:09,370 --> 01:03:13,710 I think it's one of like probably the most exciting products to just like You know, 1162 01:03:14,120 --> 01:03:17,587 when like late at night and you just think about all the ways it can go. 1163 01:03:17,670 --> 01:03:22,217 Like, I think Nat, Mark, and I just like, you know, shoot across messages late 1164 01:03:22,300 --> 01:03:25,007 at night of just like, oh, we could do this and we could do this and, 1165 01:03:25,090 --> 01:03:27,467 you know, we can make this move and we can like build this product. 1166 01:03:27,550 --> 01:03:30,010 And I think it's just like, you know, very— 1167 01:03:30,140 --> 01:03:33,267 It's a very exciting time in technology right now. 1168 01:03:33,350 --> 01:03:35,437 Yeah, it's very blue ocean in that way. Yeah. 1169 01:03:35,520 --> 01:03:36,353 Yeah. 1170 01:03:36,650 --> 01:03:38,200 I'm struck by, as we talk about 1171 01:03:39,860 --> 01:03:44,530 Muse and just how it clearly feels like 1172 01:03:44,900 --> 01:03:47,660 the next platform, an agent operating on your behalf. 1173 01:03:48,160 --> 01:03:51,700 You've talked for a long time about not wanting to be 1174 01:03:52,870 --> 01:03:56,710 reliant on other platforms and how kind of the experience of building on 1175 01:03:57,500 --> 01:04:01,167 Android and iOS kind of burned you on being dependent on other. 1176 01:04:01,250 --> 01:04:05,297 And often when you would talk about, say, Reality Labs, is the importance of owning. 1177 01:04:05,380 --> 01:04:08,430 the platform of the future and kind of many years 1178 01:04:08,890 --> 01:04:13,600 working on Reality Labs with the intent of owning the platform of the future. 1179 01:04:14,180 --> 01:04:18,600 And now this new modality comes along that feels 1180 01:04:18,690 --> 01:04:21,147 pretty clearly like how you do a lot of productive work and is 1181 01:04:21,230 --> 01:04:24,380 the platform of the future. And now Meta is really well set up. 1182 01:04:24,570 --> 01:04:28,860 Yeah, no, I mean, it's less about wanting to own a platform for its own sake. 1183 01:04:28,950 --> 01:04:31,300 It's about being able to build the best things. 1184 01:04:31,410 --> 01:04:36,117 And at some level, you know, if you look at like other social apps, 1185 01:04:36,200 --> 01:04:39,120 for example, I think part of the reason why Meta is so 1186 01:04:39,210 --> 01:04:42,087 much better is because we don't just build the app, right? 1187 01:04:42,170 --> 01:04:44,007 A lot of people like think about it, we're like this app company. 1188 01:04:44,090 --> 01:04:45,297 We're not an app company, right? 1189 01:04:45,380 --> 01:04:48,980 It's like we're like a deep technology company and we do this because we 1190 01:04:49,220 --> 01:04:52,307 are world-class at machine learning and like all the recommendation systems 1191 01:04:52,390 --> 01:04:55,687 are operating at a scale that basically few others are in the world. 1192 01:04:55,770 --> 01:04:58,670 I think it's basically like Just us and ByteDance on that, 1193 01:04:58,980 --> 01:05:03,730 and and then like okay we just kind of go all the way to the metal. 1194 01:05:04,150 --> 01:05:09,110 And to me, being able to also deliver go the last mile through the device, 1195 01:05:09,780 --> 01:05:14,907 it does come from a while of having these experiences where we couldn't build 1196 01:05:14,990 --> 01:05:17,040 the experiences that we wanted on phones. 1197 01:05:18,200 --> 01:05:23,207 But I just think for the future, the the models are going to be by far 1198 01:05:23,290 --> 01:05:25,837 the most important part of delivering the technology experience. 1199 01:05:25,920 --> 01:05:31,010 And I mean, this is something that I think we kind of bonded on and agreed on, like, 1200 01:05:31,260 --> 01:05:35,637 very early, is this notion that, like, in every generation of technology 1201 01:05:35,720 --> 01:05:39,390 companies, there's, like, no, there's usually not one company that 1202 01:05:39,480 --> 01:05:41,567 kind of is the— is, like, wins everything. 1203 01:05:41,650 --> 01:05:43,437 I don't think that that's how this is going to go either. 1204 01:05:43,520 --> 01:05:48,400 But with the internet, there were, like, okay, 5 or 6 very big companies and then 1205 01:05:48,490 --> 01:05:51,090 a whole ecosystem of other companies that are great. 1206 01:05:51,280 --> 01:05:56,287 The companies that did the best kind of had the core technology to be 1207 01:05:56,370 --> 01:05:59,330 able to serve people the best there. 1208 01:05:59,790 --> 01:06:03,167 And I think in this next era, there are going to probably be a small 1209 01:06:03,250 --> 01:06:07,127 handful of companies that have the ability to build frontier superintelligence. 1210 01:06:07,210 --> 01:06:11,047 And I think that those companies are going to be the ones that effectively drive 1211 01:06:11,130 --> 01:06:14,630 the agenda and can build most of the interesting, the most interesting 1212 01:06:15,050 --> 01:06:17,677 experiences for consumers, for enterprises, hardware, 1213 01:06:17,760 --> 01:06:19,137 these different kinds of things. 1214 01:06:19,220 --> 01:06:23,057 So, and I just look at it as, okay, like, is it even really a choice, right? 1215 01:06:23,140 --> 01:06:26,817 It's like, yeah, okay, Meta could, will probably exist into the future 1216 01:06:26,900 --> 01:06:28,447 regardless of whether we do this. 1217 01:06:28,530 --> 01:06:31,700 But if we want to be kind of an interesting company that is 1218 01:06:31,860 --> 01:06:35,370 like inventing new things that matter to people in the future, 1219 01:06:35,740 --> 01:06:39,827 I don't see how you can do that if you don't have the ability to build end-to-end 1220 01:06:39,910 --> 01:06:43,127 your own superintelligence and build them into the products that people use. 1221 01:06:43,210 --> 01:06:45,667 And I think that this is something that you just like believed from very 1222 01:06:45,750 --> 01:06:48,837 early on, Is basically, it's like, all right, there 1223 01:06:48,920 --> 01:06:50,297 are going to be like a handful of these. 1224 01:06:50,380 --> 01:06:54,430 Like, it's like there's a kind of personal and consumer-oriented version of this. 1225 01:06:54,760 --> 01:06:58,217 And like, this lane is open because everyone else seems to just be obsessed 1226 01:06:58,300 --> 01:07:01,727 so much with just automating knowledge work, which, I mean, isn't even that 1227 01:07:01,810 --> 01:07:03,767 charismatic version of the future, right? 1228 01:07:03,850 --> 01:07:06,437 Because it's like, all right, like, a big part of our belief too is 1229 01:07:06,520 --> 01:07:10,480 that the kind of highest use of superintelligence is invention 1230 01:07:10,650 --> 01:07:12,190 and creation, not automation. 1231 01:07:12,650 --> 01:07:15,117 And so, I don't know, it just seemed like there's like a big 1232 01:07:15,200 --> 01:07:17,900 place in there, but you kind of need to own the stack. 1233 01:07:21,290 --> 01:07:25,457 How is the Muse-mediated future going to shake 1234 01:07:25,540 --> 01:07:30,050 out with businesses need to— okay, 1235 01:07:30,550 --> 01:07:33,510 businesses variously need to build support 1236 01:07:34,130 --> 01:07:34,963 for 1237 01:07:38,640 --> 01:07:42,680 for Muse, where it might try to computer 1238 01:07:42,770 --> 01:07:45,640 use its way in, but that might be janky in certain ways. 1239 01:07:45,730 --> 01:07:48,437 So you kind of want everyone to have first-class support 1240 01:07:48,520 --> 01:07:50,067 for agents talking to things. Yeah. 1241 01:07:50,150 --> 01:07:54,777 Then the bigger question is they might actively try to block agents 1242 01:07:54,860 --> 01:07:56,237 from interacting with them. 1243 01:07:56,320 --> 01:07:58,787 And this might be because they have a misguided sense where, like, 1244 01:07:58,870 --> 01:08:01,747 it's actually in their interest to support agents, but they don't want to. 1245 01:08:01,830 --> 01:08:04,457 It might actually not be in their interest to support agents. 1246 01:08:04,540 --> 01:08:08,340 You can imagine where maybe they have a very thin moat and it all comes from 1247 01:08:09,090 --> 01:08:13,007 a business that only succeeds by consumers spending time on the site or only succeeds 1248 01:08:13,090 --> 01:08:15,927 by overcharging consumers, then they should rationally not want 1249 01:08:16,010 --> 01:08:17,637 to allow agents because the agents are like, why? 1250 01:08:17,720 --> 01:08:19,870 But those are going to get disrupted. Yeah. 1251 01:08:20,180 --> 01:08:22,137 So talk to me just through the second-order 1252 01:08:22,220 --> 01:08:24,060 effects of how the economy works. 1253 01:08:24,640 --> 01:08:27,437 Well, I think that it starts off with— I mean, we trained into the models 1254 01:08:27,520 --> 01:08:29,060 very good browser use, right? 1255 01:08:29,520 --> 01:08:32,897 So that way we want to make it so that no one No one has to do anything 1256 01:08:32,980 --> 01:08:34,360 to integrate with Muse. 1257 01:08:35,070 --> 01:08:38,360 And then, but then of course, for— we're kind of just going down 1258 01:08:38,450 --> 01:08:42,620 the list of the most important partners, and for people who want to work with us, 1259 01:08:43,080 --> 01:08:45,997 we'll just get direct API access, right? 1260 01:08:46,080 --> 01:08:48,537 And then it's like, okay, then it ends up being way more efficient 1261 01:08:48,620 --> 01:08:50,457 and faster for people to use Muse. 1262 01:08:50,540 --> 01:08:53,417 It's more efficient because now you don't have an agent crawling your website. 1263 01:08:53,500 --> 01:08:57,427 It's just going. So anyone who wants to do that, we eventually want to get there. 1264 01:08:57,510 --> 01:09:01,260 Um, and then I think the fullness of time is 1265 01:09:01,430 --> 01:09:04,720 that, you know, just like today every business has, 1266 01:09:05,390 --> 01:09:10,230 um, a website and social media accounts and email addresses and all these things, 1267 01:09:10,690 --> 01:09:12,227 every business is going to have agents too. 1268 01:09:12,310 --> 01:09:15,527 Mm-hmm. So, and we're also building that on the other side of it. 1269 01:09:15,610 --> 01:09:20,320 And, um, and so of course Muse agents will talk to other types of agents, but they'll 1270 01:09:20,410 --> 01:09:22,117 work really well with our agents, right? 1271 01:09:22,200 --> 01:09:25,077 Because I mean, we'll, we'll be able to integrate that Just, 1272 01:09:25,160 --> 01:09:28,077 I mean, so Muse using websites is just a moment in time thing. 1273 01:09:28,160 --> 01:09:29,247 Just to make it easy for everyone. 1274 01:09:29,330 --> 01:09:32,587 You guys want to have it as a backstop so it's reliable for people using it. 1275 01:09:32,670 --> 01:09:35,210 Yes, and easy for businesses to get bootstrapped. 1276 01:09:35,500 --> 01:09:40,340 But we have, we have way more businesses that are calling us asking to integrate 1277 01:09:40,430 --> 01:09:46,017 deeply, and very few really who have tried to block it. 1278 01:09:46,100 --> 01:09:49,227 And my guess is that the ones that try to block it, it will end up not being 1279 01:09:49,310 --> 01:09:50,817 the right strategic thing for them. 1280 01:09:50,900 --> 01:09:52,567 But we'll see. This will kind of play out over time. 1281 01:09:52,650 --> 01:09:54,727 It saves them a lot of time. Yeah. 1282 01:09:54,810 --> 01:09:55,937 It saves them a lot of money. 1283 01:09:56,020 --> 01:09:59,237 Like, I think these are, these are, like, clearly good things for the consumer. 1284 01:09:59,320 --> 01:10:03,487 Yeah. So in the long arc, you'd clearly expect that demand will 1285 01:10:03,570 --> 01:10:08,450 shift from current avenues more and more towards agentic demand. 1286 01:10:09,330 --> 01:10:12,457 And, you know, there always will be demand that lives elsewhere, 1287 01:10:12,540 --> 01:10:15,837 just in the same way that, like, there's still brick-and-mortar demand in commerce. 1288 01:10:15,920 --> 01:10:17,467 But there's a lot of ecommerce demand. 1289 01:10:17,550 --> 01:10:21,380 And kind of in the same way, like, I think agentic commerce demand 1290 01:10:21,470 --> 01:10:22,967 will grow as a portion of the pie. 1291 01:10:23,050 --> 01:10:25,467 The others will still exist, but will grow as a portion of the pie. 1292 01:10:25,550 --> 01:10:28,887 And I think there's sort of these questions for businesses in the long run, 1293 01:10:28,970 --> 01:10:32,230 like, do they want to participate in that slice 1294 01:10:32,480 --> 01:10:35,060 or that portion of the economy over time? 1295 01:10:35,690 --> 01:10:37,737 And I think it's an incredibly exciting area. 1296 01:10:37,820 --> 01:10:40,777 I think it's, like, very, very exciting to imagine what will 1297 01:10:40,860 --> 01:10:42,447 happen in this agentic commerce future. Yes. 1298 01:10:42,530 --> 01:10:45,160 I actually think Patrick wrote a great tweeticle. 1299 01:10:47,030 --> 01:10:48,367 Is that what we call it these days? I don't know. 1300 01:10:48,450 --> 01:10:50,167 Patrick has his own thing going on with tweets. 1301 01:10:50,250 --> 01:10:51,747 It's like they're buried amongst— 1302 01:10:51,830 --> 01:10:57,797 I noticed that Brother Karamazov was quite a good literary analysis. 1303 01:10:57,880 --> 01:11:01,470 But yeah, no, I think clearly 1304 01:11:02,050 --> 01:11:04,930 this is a discontinuous event for how 1305 01:11:06,100 --> 01:11:09,057 commerce will happen, broadly speaking, or just generally speaking, how 1306 01:11:09,140 --> 01:11:12,767 the entire how all the business pipes will work, maybe even in a broader sense. 1307 01:11:12,850 --> 01:11:16,310 But I think that, you know, you just go back to, this is clearly 1308 01:11:16,400 --> 01:11:17,817 a lot better for the consumers. 1309 01:11:17,900 --> 01:11:18,987 It's a lot better for the customers. 1310 01:11:19,070 --> 01:11:22,070 So there's a natural gravity towards that reality. 1311 01:11:22,400 --> 01:11:25,067 One of the exciting things, I'm curious what you guys see too, 1312 01:11:25,150 --> 01:11:29,617 but I think Muse is one of the first products where you're actually people are 1313 01:11:29,700 --> 01:11:32,370 fully delegating purchases to the agents. 1314 01:11:32,750 --> 01:11:35,837 And this is like a— this is kind of like a, you know, a step change, right? 1315 01:11:35,920 --> 01:11:39,667 Like, I think, you know, we've been talking a lot about agentic 1316 01:11:39,750 --> 01:11:42,637 shopping and like the possibility of it, but now it's like really happening. 1317 01:11:42,720 --> 01:11:45,047 And a lot of people are trusting the agents to shop for them. 1318 01:11:45,130 --> 01:11:48,767 And I think the agents are finding them, you know, in my case, like great teas, or, 1319 01:11:48,850 --> 01:11:52,437 you know, in whomever— whoever's case, like great products for them to purchase, 1320 01:11:52,520 --> 01:11:53,770 which is quite exciting. 1321 01:11:53,930 --> 01:11:58,267 And so I think that there's There's clearly an exciting thing happening here. 1322 01:11:58,350 --> 01:12:01,317 I think, I think part of the value prop— well, one, one part of the value prop 1323 01:12:01,400 --> 01:12:02,947 for the user is just reduced friction. 1324 01:12:03,030 --> 01:12:06,527 Like, hey, I don't feel like click on all these websites and do all this research 1325 01:12:06,610 --> 01:12:08,157 to try to find the exact products. 1326 01:12:08,240 --> 01:12:10,077 Like, that step is taken care of for me. 1327 01:12:10,160 --> 01:12:13,200 And also, like, criteria can just be automatically applied. 1328 01:12:13,330 --> 01:12:15,380 Like, hey, I need something by Wednesday. 1329 01:12:15,500 --> 01:12:18,087 So, you know, the agent will only look for things that could get to me by Wednesday. 1330 01:12:18,170 --> 01:12:19,167 But that's, I mean, keyword search. 1331 01:12:19,250 --> 01:12:20,667 It was insane that we ever relied on it because that's a very 1332 01:12:20,750 --> 01:12:22,337 reasonable constraint. You just want to shop by. 1333 01:12:22,420 --> 01:12:23,253 Yeah, exactly. 1334 01:12:24,340 --> 01:12:28,340 And so I think the like old way of like searching for stuff just feels so 1335 01:12:28,640 --> 01:12:31,140 obviously primitive now, now that you can do this. 1336 01:12:31,350 --> 01:12:34,727 Then there's this sort of like back part of the funnel, so to speak, 1337 01:12:34,810 --> 01:12:38,100 but just like the ease of friction reduction of just, yeah, 1338 01:12:38,230 --> 01:12:41,520 filling out the form, just like getting it ordered is quite nice. 1339 01:12:42,110 --> 01:12:45,570 And I think the long-term thing that we're quite excited about 1340 01:12:46,190 --> 01:12:52,197 is greater personalization. Ultimately, I think one of the exciting things about 1341 01:12:52,280 --> 01:12:58,747 Muse is that it has sort of like this holistic context into you as a person 1342 01:12:58,830 --> 01:13:00,417 and all of the aspects of your life. 1343 01:13:00,500 --> 01:13:04,337 So it sort of like can holistically understand things about your family or 1344 01:13:04,420 --> 01:13:06,637 things about your coworkers or things about your friends. 1345 01:13:06,720 --> 01:13:11,347 And then sort of like help you co-optimize across all of those to get you the exact 1346 01:13:11,430 --> 01:13:16,017 right gift for your, you know, best friend in the context of your friend 1347 01:13:16,100 --> 01:13:17,267 group based on what it knows about your friend group. 1348 01:13:17,350 --> 01:13:20,500 Like, I think there's just like a lot of exciting opportunities 1349 01:13:20,810 --> 01:13:24,647 that are somewhat new and novel when it comes to like these 1350 01:13:24,730 --> 01:13:25,817 personalizations, these recommendations. 1351 01:13:25,900 --> 01:13:29,050 Because historically, a lot of these recommendation systems are 1352 01:13:29,240 --> 01:13:31,700 quite siloed and quite isolated. Mm-hmm. 1353 01:13:32,070 --> 01:13:35,270 And so they're sort of like, they don't see like a holistic view 1354 01:13:35,700 --> 01:13:37,160 of you as a person. 1355 01:13:37,660 --> 01:13:41,417 And Muse has an opportunity to offer much greater levels of personalization. 1356 01:13:41,500 --> 01:13:46,460 Yeah, I think people are already seeing that, where if they ask, like with Muse, 1357 01:13:46,550 --> 01:13:49,887 if you ask it to buy something for you, quite frequently it'll add in some bit 1358 01:13:49,970 --> 01:13:52,677 of context that it pulled in from a different conversation. 1359 01:13:52,760 --> 01:13:54,610 You had a good example of this. Yeah. 1360 01:13:54,850 --> 01:13:59,980 Well, so my 3-year-old daughter loves baking. 1361 01:14:00,190 --> 01:14:03,397 She loves cooking. And so I'm like, all right, I'll start baking stuff. 1362 01:14:03,480 --> 01:14:08,440 It's like I'm normally into smoking meats, but I'll do some baking in order 1363 01:14:08,530 --> 01:14:10,700 to To make, to make Aurelia happy. 1364 01:14:11,530 --> 01:14:14,660 And so we've gotten into this really cute routine where, 1365 01:14:15,950 --> 01:14:19,330 um, you know, we, we, I've been playing with Muse voice mode, 1366 01:14:19,540 --> 01:14:22,890 which isn't fully rolled out yet, but we're, we're playing with it. 1367 01:14:23,000 --> 01:14:25,087 And, um, it does this great thing. 1368 01:14:25,170 --> 01:14:26,797 It segments the, the voices. 1369 01:14:26,880 --> 01:14:30,880 So it knows which one is me and it knows which one is not me based on that. 1370 01:14:31,510 --> 01:14:35,387 And we just like, you know, throughout the week we're talking about 1371 01:14:35,470 --> 01:14:37,097 what we'll want to bake that weekend. Mm-hmm. 1372 01:14:37,180 --> 01:14:42,267 And she'll like give it ideas or like she'll kind of come 1373 01:14:42,350 --> 01:14:48,230 up with crazy things that are not real recipes, but like bounce it off with Muse. 1374 01:14:48,360 --> 01:14:51,900 And then Muse will just make sure that we, 1375 01:14:52,610 --> 01:14:55,910 every Saturday when it's time to bake, 1376 01:14:56,700 --> 01:15:01,037 I get the ingredients from Instacart and they kind of show up and they're 1377 01:15:01,120 --> 01:15:04,087 on time, which I mean, the biggest thing is like if I were just 1378 01:15:04,170 --> 01:15:07,257 starting the activity on Saturday, you'd have to get everything 1379 01:15:07,340 --> 01:15:08,417 and it would take some time. 1380 01:15:08,500 --> 01:15:12,137 But now everything's ready whenever we're going to go sit down and do it. 1381 01:15:12,220 --> 01:15:15,057 This weekend it knew that watermelon was launching, so it threw 1382 01:15:15,140 --> 01:15:18,260 a watermelon in the bag. It was very, very sweet. 1383 01:15:18,350 --> 01:15:19,930 It was very heartwarming. 1384 01:15:21,060 --> 01:15:25,310 It deduced that the recipe did not have 1385 01:15:25,400 --> 01:15:29,777 enough— there was not enough, I guess, 1386 01:15:29,860 --> 01:15:33,277 expense in it to get the delivery, so it needed something else. 1387 01:15:33,360 --> 01:15:34,907 And it was like, you're going to— And it topped it up with a watermelon. 1388 01:15:34,990 --> 01:15:36,867 It was like, do you want this watermelon? I was like, yes. 1389 01:15:36,950 --> 01:15:38,780 I'm like, yes, that's very cute. 1390 01:15:38,870 --> 01:15:41,037 Which I think you, uh, did you— is that the watermelon you ate? 1391 01:15:41,120 --> 01:15:43,470 Yeah, that was the one I ate on the livestream. 1392 01:15:43,620 --> 01:15:45,167 Okay, but the watermelon on the livestream was— 1393 01:15:45,250 --> 01:15:47,037 That was just like, Muse just got it for me. 1394 01:15:47,120 --> 01:15:50,547 That was a Muse—yeah, those are just— You just got it for me. 1395 01:15:50,630 --> 01:15:52,420 But I think there's this, um— 1396 01:15:52,510 --> 01:15:56,047 That was a very memeable moment of that very memeable situation. 1397 01:15:56,130 --> 01:15:58,260 So we have Muse to thank for it. 1398 01:15:58,350 --> 01:16:01,937 So, but here's another interesting thing about the whole thing is I didn't 1399 01:16:02,020 --> 01:16:04,097 really use Instacart that much before. 1400 01:16:04,180 --> 01:16:08,230 You know, it's like every once in a while, maybe twice a year, I would like like, 1401 01:16:08,480 --> 01:16:10,567 use Priscilla's account or something. Yes. 1402 01:16:10,650 --> 01:16:15,900 I went and got an Instacart account to give Muse, and now I use it like once 1403 01:16:15,990 --> 01:16:19,320 or twice a week because like I'm talking with Aurelia with Muse, 1404 01:16:19,410 --> 01:16:22,457 and we're coming up with these recipes, and it's delivering them, and now 1405 01:16:22,540 --> 01:16:24,077 it's like a core part of my workflow. 1406 01:16:24,160 --> 01:16:28,497 So I actually think that the opportunity here isn't just, okay, it's going to be 1407 01:16:28,580 --> 01:16:30,790 20% more efficient on the conversion rate. 1408 01:16:31,040 --> 01:16:34,170 Like, I am a new Instacart like subscriber, and I just 1409 01:16:34,260 --> 01:16:35,467 wouldn't have done this before. Yes. 1410 01:16:35,550 --> 01:16:38,097 So, I do think that there's this question, and then there's this question, 1411 01:16:38,180 --> 01:16:42,220 which I think is these sort of interesting— like, 1412 01:16:42,760 --> 01:16:45,410 this is maybe the more interesting economic question, 1413 01:16:45,680 --> 01:16:48,850 is that Instacart account really for me, or is it for the Muse? 1414 01:16:48,940 --> 01:16:51,937 Because I think, like, over time, I think that the question is, 1415 01:16:52,020 --> 01:16:56,070 is the total addressable market for all of commerce just increasing dramatically? 1416 01:16:56,280 --> 01:16:58,700 Yes. Because it's not just the people anymore. 1417 01:16:58,820 --> 01:17:01,157 Like, now agents are effectively going to have accounts. 1418 01:17:01,240 --> 01:17:05,490 Yes. And I think that's where this starts to, like, become a really very exciting 1419 01:17:05,580 --> 01:17:08,537 opportunity where anyone who sits this out, I think, is going to end 1420 01:17:08,620 --> 01:17:12,880 up feeling sad over time. And why did you have to sign up for an Instacart account? 1421 01:17:13,340 --> 01:17:16,087 Like, we're solving this at Stripe Projects for developer 1422 01:17:16,170 --> 01:17:20,637 stuff where your coding agent— The Stripe integration, by the way, is great. 1423 01:17:20,720 --> 01:17:23,510 You guys did a great job with us on Muse. 1424 01:17:23,640 --> 01:17:25,470 I think we're very happy. 1425 01:17:25,640 --> 01:17:28,517 We have been very excited about agent-to-commerce for a long time, 1426 01:17:28,600 --> 01:17:31,147 obviously, because again, felt to us like clearly how people 1427 01:17:31,230 --> 01:17:33,320 are going to buy things in the future. 1428 01:17:34,610 --> 01:17:38,447 And that was both apparent because no one needs all that friction in their life 1429 01:17:38,530 --> 01:17:43,280 from the actual buying stage, and how early product research with AI had 1430 01:17:43,370 --> 01:17:46,450 product-market fit, even from kind of the earliest AIs. 1431 01:17:47,040 --> 01:17:50,917 It really feels like computer use was the last big unlock that we needed to make 1432 01:17:51,000 --> 01:17:54,750 it happen, because now you can ask your muse to buy anything, more or less. 1433 01:17:54,920 --> 01:17:57,217 And it'll just figure it out with computer use, right? 1434 01:17:57,300 --> 01:17:59,047 And so it's like the ultimate backwards compatibility layer. 1435 01:17:59,130 --> 01:18:01,557 Yeah, it's pretty— this is a set of things that Nat really obsessed over. 1436 01:18:01,640 --> 01:18:02,257 Yeah, yeah. 1437 01:18:02,340 --> 01:18:05,887 I think, I think there was a lot of, like, I think a lot of magic of Muse is 1438 01:18:05,970 --> 01:18:08,470 the fact that like a lot of this stuff just works. 1439 01:18:08,850 --> 01:18:12,437 And so like clearly if everything was just like a little bit 1440 01:18:12,520 --> 01:18:15,527 less reliable, it would feel like a total pain in the ass and nobody would use 1441 01:18:15,610 --> 01:18:16,897 it and people just throw away the app. 1442 01:18:16,980 --> 01:18:20,157 And so I think the fact that so much of this has gotten to the point 1443 01:18:20,240 --> 01:18:24,280 of reliability where You know, it just works is like a huge part of the magic. 1444 01:18:24,410 --> 01:18:26,287 Yeah. What were you saying Matt was obsessed with? 1445 01:18:26,370 --> 01:18:29,537 Oh, well, I mean, well, Matt's kind of obsessed with everything. 1446 01:18:29,620 --> 01:18:30,917 Yeah, well, that's generally a sexy character. 1447 01:18:31,000 --> 01:18:32,377 Yeah, no, it's great, it's great. 1448 01:18:32,460 --> 01:18:35,297 But this is, I mean, but yeah, I mean, this getting this experience to be good, 1449 01:18:35,380 --> 01:18:38,297 I think is something that he and a core group of people on that 1450 01:18:38,380 --> 01:18:41,130 team really focused on. It's kind of funny. 1451 01:18:41,220 --> 01:18:44,890 I think I had a dinner with Patrick, like, I want to say it was 2023, 1452 01:18:46,510 --> 01:18:48,477 where I was, he was like, what do you think we should be doing now? 1453 01:18:48,560 --> 01:18:51,600 I was Oh, you're like, you're already way behind the 8-ball 1454 01:18:51,690 --> 01:18:52,897 on this agentic commerce stuff. 1455 01:18:52,980 --> 01:18:54,517 Like, you got to figure this stuff out. 1456 01:18:54,600 --> 01:18:57,737 Like, agents are going to be buying things and like, you're not prepared for it. 1457 01:18:57,820 --> 01:18:59,277 So you were saying this to Patrick or Patrick saying this to you? 1458 01:18:59,360 --> 01:19:00,487 I was saying this to Patrick. 1459 01:19:00,570 --> 01:19:02,657 Well, because Patrick was very politely asking, what do you 1460 01:19:02,740 --> 01:19:06,327 think we should be doing? And I sort of had this like very like urgent spiel. 1461 01:19:06,410 --> 01:19:09,287 Yeah, and I think it's very, uh, it's funny how that was like— 1462 01:19:09,370 --> 01:19:10,247 Well, thank you for your provocation. 1463 01:19:10,330 --> 01:19:14,250 Well, it was directionally correct, but, uh, but like 3 years early. 1464 01:19:14,830 --> 01:19:18,547 2023, I think. And so, um, no, it's very interesting to look 1465 01:19:18,630 --> 01:19:22,590 at the agentic commerce stats at Stripe because 1466 01:19:23,050 --> 01:19:25,597 we started developing all this stuff and we had a whole bunch of announcements 1467 01:19:25,680 --> 01:19:29,720 at Sessions back in May, uh, and then it only as of September 1468 01:19:29,810 --> 01:19:31,470 basically started going vertical. 1469 01:19:34,310 --> 01:19:37,710 Meta is building increasingly capable models, including Muse Wonder, 1470 01:19:38,060 --> 01:19:41,810 and their models are available on OpenRouter, Alongside hundreds of others. 1471 01:19:42,070 --> 01:19:46,070 Stripe has agreed to acquire OpenRouter, who's the leading model gateway. 1472 01:19:46,780 --> 01:19:50,617 OpenRouter already routes more than 10 trillion tokens a day and handles much 1473 01:19:50,700 --> 01:19:53,167 of the complexity of working with different models through a single 1474 01:19:53,250 --> 01:19:57,540 API, making it easier to experiment, find competitive prices, and automatically 1475 01:19:57,630 --> 01:19:59,130 route around provider outages. 1476 01:19:59,630 --> 01:20:01,960 Try it for yourself at openrouter. ai. 1477 01:20:06,640 --> 01:20:11,970 Hey, could you order a watermelon to Stripe's office in South San Francisco? 1478 01:20:12,270 --> 01:20:14,570 I'm doing a podcast here. This would be funny. 1479 01:20:16,690 --> 01:20:17,850 Funny podcast idea. 1480 01:20:18,480 --> 01:20:20,480 I'll check that watermelon delivery now. 1481 01:20:22,360 --> 01:20:24,030 Great. We'll see how we get on. 1482 01:20:26,570 --> 01:20:29,420 Well, okay. This actually raises an interesting question. 1483 01:20:29,820 --> 01:20:34,450 People have been trying to make The HAL 9000. 1484 01:20:34,580 --> 01:20:35,413 It's working. 1485 01:20:36,750 --> 01:20:37,587 I really like this. 1486 01:20:37,670 --> 01:20:40,507 I think this is an awesome touch where it's like clacking away at its laptop. 1487 01:20:40,590 --> 01:20:42,300 I really like that in the app. 1488 01:20:43,420 --> 01:20:46,510 We'll just leave them there and we'll see what we get. 1489 01:20:46,720 --> 01:20:50,430 Yeah, it's tough to prop up. Yeah, there we go. 1490 01:20:52,220 --> 01:20:55,767 People have been trying to make the speak to a computer AI vision 1491 01:20:55,850 --> 01:20:57,390 happen for a really long time. 1492 01:20:57,980 --> 01:21:03,230 Where Apple launched Siri in 2014, 2015, something like that. 1493 01:21:03,770 --> 01:21:08,280 And obviously there's been Voice Mode and various AI products for a few years now. 1494 01:21:08,400 --> 01:21:10,197 Why is it so hard? Like, it's still not the case. 1495 01:21:10,280 --> 01:21:14,157 If you were driving your car and you wanted to check on the baking or move 1496 01:21:14,240 --> 01:21:16,117 a meeting or something like that, you actually probably 1497 01:21:16,200 --> 01:21:17,787 wouldn't do it with the voice. 1498 01:21:17,870 --> 01:21:19,837 I mean, maybe almost now with the latest things. 1499 01:21:19,920 --> 01:21:20,707 Yeah, well, there's a lot. 1500 01:21:20,790 --> 01:21:22,297 I mean, so— But you wouldn't develop Siri like you're doing. 1501 01:21:22,380 --> 01:21:24,797 Siri right now is actually a good example. As we're talking over each other. 1502 01:21:24,880 --> 01:21:26,177 And I mean, I don't know, when you're using different voices— 1503 01:21:26,260 --> 01:21:29,007 So is it just technical limitations? That's why we haven't done it? 1504 01:21:29,090 --> 01:21:30,847 Well, I think that there's a few things. 1505 01:21:30,930 --> 01:21:32,927 And one is you actually want it to go do stuff. 1506 01:21:33,010 --> 01:21:36,227 So there's a question of when you speak with it, even if it understands, 1507 01:21:36,310 --> 01:21:37,560 how's that going to work? 1508 01:21:38,020 --> 01:21:41,147 Then I think that there's a lot of stuff about the way that people 1509 01:21:41,230 --> 01:21:42,147 have conversations. 1510 01:21:42,230 --> 01:21:46,860 It isn't typically, I will give a prompt, and now I will wait, and you will speak. 1511 01:21:47,030 --> 01:21:49,080 And then, so it just goes back and forth. 1512 01:21:49,490 --> 01:21:51,657 Um, how many of your sessions when you're 1513 01:21:51,740 --> 01:21:56,037 talking to an agent in voice mode, do you basically, oh, 1514 01:21:56,120 --> 01:21:59,167 it starts saying something, but then like, I don't know, it's like, 1515 01:21:59,250 --> 01:22:02,167 I kind of feel like if I pause for 3 seconds, it's going to think 1516 01:22:02,250 --> 01:22:03,837 I'm done and want to speak. 1517 01:22:03,920 --> 01:22:07,257 And then if it starts speaking, but then I cut back in, is it going to 1518 01:22:07,340 --> 01:22:09,440 think that it's like a whole new question? 1519 01:22:09,930 --> 01:22:12,847 Um, I think there's a lot of the stuff about the conversational dynamics 1520 01:22:12,930 --> 01:22:14,280 that you want to get right. 1521 01:22:14,430 --> 01:22:20,560 But then there's this other issue, which is that For voice overall, 1522 01:22:21,770 --> 01:22:24,810 it's quite high bandwidth for you to communicate with voice, 1523 01:22:25,520 --> 01:22:30,067 but in terms of consuming information, having someone speak to you is very 1524 01:22:30,150 --> 01:22:32,197 inefficient compared to having it be visual. 1525 01:22:32,280 --> 01:22:35,870 Yeah. So I think like that's why, you know, partially when we do these, 1526 01:22:36,870 --> 01:22:39,747 these interfaces, I think it's, it's kind of working on its task. 1527 01:22:39,830 --> 01:22:40,790 I fired it off now. 1528 01:22:41,370 --> 01:22:45,090 So I think the ideal interface is probably not just strictly voice-voice. 1529 01:22:45,590 --> 01:22:49,800 Although I think voice output from you is pretty useful. 1530 01:22:50,300 --> 01:22:54,307 Yeah, I think, I think there very clearly already is an explosion of voice input. 1531 01:22:54,390 --> 01:22:57,040 Like, I think you even see this in coding where like— 1532 01:22:57,350 --> 01:22:59,097 Yeah, people are speaking to the— Yeah, yeah, yeah. 1533 01:22:59,180 --> 01:23:03,350 Most of the researchers just kind of like whisper into their keyboard because it is 1534 01:23:03,440 --> 01:23:07,520 like, as Mark's mentioning, it's like the highest bandwidth way to get your like 1535 01:23:07,610 --> 01:23:10,567 thoughts out, or usually the lowest friction way to get your thoughts out. 1536 01:23:10,650 --> 01:23:11,500 Yeah, yeah, yeah. 1537 01:23:11,650 --> 01:23:15,050 I think the most extreme version of this, by the way, is I think how 1538 01:23:16,200 --> 01:23:18,867 college students now use AI is maybe the most interesting, 1539 01:23:18,950 --> 01:23:22,290 where they just— I think a very common modality of how to use 1540 01:23:23,040 --> 01:23:28,920 AI is that they just will completely word vomit for 10, 15 minutes, 1541 01:23:29,170 --> 01:23:32,800 being full stream of consciousness, chain of thought of all the things. 1542 01:23:33,050 --> 01:23:35,057 And then have AI clean it up and infer. Yeah. 1543 01:23:35,140 --> 01:23:40,350 And have AI effectively do all of this organization of it and pulling things 1544 01:23:40,680 --> 01:23:42,397 together and structuring everything. 1545 01:23:42,480 --> 01:23:43,557 It's quite interesting. 1546 01:23:43,640 --> 01:23:46,770 It's a very interesting, I think, modality of tech usage. 1547 01:23:47,100 --> 01:23:49,227 Yeah. But yeah, I think voice input has been big. 1548 01:23:49,310 --> 01:23:54,027 I do think agents is like, that's a big step change, because I think 1549 01:23:54,110 --> 01:23:57,447 the mental model is just very different. 1550 01:23:57,530 --> 01:24:02,120 Like, if it's just like an informational thing that you're talking to, that like, 1551 01:24:02,540 --> 01:24:04,997 you will clearly get into situations where it just feels useless. 1552 01:24:05,080 --> 01:24:08,957 Whereas now that it can do things for you, it's much closer to the experience 1553 01:24:09,040 --> 01:24:10,217 of talking to a human. Totally. 1554 01:24:10,300 --> 01:24:11,133 Or to a person. 1555 01:24:11,550 --> 01:24:16,757 And so I think there's all these interesting uncanny valleys in interacting 1556 01:24:16,840 --> 01:24:22,470 with AIs that until they've been able to take actions have felt kind of strange. 1557 01:24:22,560 --> 01:24:28,147 And now that we have agents, I think one of the cool things about Muse 1558 01:24:28,230 --> 01:24:30,227 is this confluence of a number of things. 1559 01:24:30,310 --> 01:24:34,740 I think The fact that Muse is embodied and it like is this little guy 1560 01:24:35,320 --> 01:24:38,987 that like you feel like is working on your behalf is also only possible 1561 01:24:39,070 --> 01:24:44,160 and a coherent like representation and embodiment of the product because 1562 01:24:44,660 --> 01:24:48,580 it has persistent memory and it has the ability to do things for you 1563 01:24:48,670 --> 01:24:51,667 and the ability to take actions and has like a little bit of a personality. 1564 01:24:51,750 --> 01:24:54,797 Like, is the confluence of all these things coming together that make 1565 01:24:54,880 --> 01:24:56,580 this like embodiment concept— Yes. 1566 01:24:57,180 --> 01:24:58,630 Even work in the first place. 1567 01:24:59,180 --> 01:25:01,427 Then it turns out, okay, actually this is the real glasses, right? 1568 01:25:01,510 --> 01:25:05,227 It's like, okay, a lot of people want it even with just like no 1569 01:25:05,310 --> 01:25:06,910 display or a small display. Yes. 1570 01:25:07,770 --> 01:25:10,770 The camera and audio is enough of a useful product. 1571 01:25:11,110 --> 01:25:12,020 Yeah, yeah. 1572 01:25:12,110 --> 01:25:16,690 And I— and then we started shipping the display glasses that aren't the kind 1573 01:25:16,780 --> 01:25:19,777 of wide field of view full augmented reality, which, I mean, 1574 01:25:19,860 --> 01:25:25,117 I thought would be awesome too, but even the small display is pretty powerful. 1575 01:25:25,200 --> 01:25:29,000 So I think you'll start getting some interesting interactions there that are 1576 01:25:29,460 --> 01:25:32,000 kind of fully multimodal, right? 1577 01:25:32,130 --> 01:25:36,337 So it's voice in and out and It'll be able to see what you see, and it'll be 1578 01:25:36,420 --> 01:25:38,840 able to kind of paint the pixels. 1579 01:25:39,590 --> 01:25:42,240 And one of the— I mean, the other exciting thing that 1580 01:25:43,140 --> 01:25:45,590 we've done here is Muse Gadgets, which is really, 1581 01:25:45,890 --> 01:25:51,940 I think, unleashing the ecosystem to try cool device formats and form factors 1582 01:25:52,270 --> 01:25:53,397 that could be really exciting. 1583 01:25:53,480 --> 01:25:56,647 Like, since releasing that, we've gotten actually even a number 1584 01:25:56,730 --> 01:26:00,197 of phone manufacturers, you know, talk to us about excited You 1585 01:26:00,280 --> 01:26:02,407 know, we're like, we're just going to integrate Muse Gadgets 1586 01:26:02,490 --> 01:26:04,867 into our phone and see how well it works. And like, we're really excited about that. 1587 01:26:04,950 --> 01:26:07,997 Like, I think, um, I think part of what is cool about 1588 01:26:08,080 --> 01:26:11,997 the current moment of technology is that like, it's very clear 1589 01:26:12,080 --> 01:26:13,247 agents are really exciting. 1590 01:26:13,330 --> 01:26:16,500 It's really clear that like, this should unlock lots of things. 1591 01:26:16,710 --> 01:26:18,547 Will you guys come back to the household devices? 1592 01:26:18,630 --> 01:26:21,637 Like, this is obviously a personal device, but, um, we actually had 1593 01:26:21,720 --> 01:26:25,010 during COVID the, um, Portress— the Portal. 1594 01:26:25,100 --> 01:26:28,557 Portal. There we go. Yeah, I mean, like, he's finding the Portal inside. 1595 01:26:28,640 --> 01:26:30,977 One of the gadgets that we just shipped was the little HomeLink. 1596 01:26:31,060 --> 01:26:34,060 It's like the size of a little, like, nickel or something, 1597 01:26:34,230 --> 01:26:36,900 but with a USB, right? Yeah. 1598 01:26:37,070 --> 01:26:41,277 Um, uh, so you can basically connect that, and then your Muse can connect 1599 01:26:41,360 --> 01:26:42,157 to everything in your home. 1600 01:26:42,240 --> 01:26:47,280 This is actually one of the most fun things too is just having it connect to 1601 01:26:47,580 --> 01:26:49,577 your Sonos and connect to all your lights. 1602 01:26:49,660 --> 01:26:53,167 I mean, I have it basically like when it's time for me to start 1603 01:26:53,250 --> 01:26:56,457 winding down in the evening, I just have it like dim all the lights 1604 01:26:56,540 --> 01:26:59,630 across the house and it's nice. 1605 01:27:00,090 --> 01:27:01,967 How are we going to fund the AI buildout? 1606 01:27:02,050 --> 01:27:05,557 Everyone's worried about, you know, Google went free cash flow negative. 1607 01:27:05,640 --> 01:27:08,687 You know, you were talking about 1 gigawatt and then 5 gigawatts 1608 01:27:08,770 --> 01:27:09,977 and everything like this. 1609 01:27:10,060 --> 01:27:15,310 Meta's now free cash flow negative yet, but there's always more CapEx to fund. 1610 01:27:15,690 --> 01:27:18,480 And so what's the answer? Where is Susan Lee? 1611 01:27:18,570 --> 01:27:24,447 Exactly. No, I mean, look, I think that there are going 1612 01:27:24,530 --> 01:27:26,787 to be a lot of ways to create value from this. 1613 01:27:26,870 --> 01:27:32,000 So what we're trying to do with Muse specifically is make it so that it can be 1614 01:27:32,700 --> 01:27:35,037 widely used, because I think that there are going to be 1615 01:27:35,120 --> 01:27:39,797 some kind of valuable network effects at scale as people start being able 1616 01:27:39,880 --> 01:27:43,047 to interact with each other's, and that's going to be great. 1617 01:27:43,130 --> 01:27:45,927 So I think there are all these reasons why you want it to get to large scale. 1618 01:27:46,010 --> 01:27:47,387 Also, you know, we were talking about partners. 1619 01:27:47,470 --> 01:27:50,017 Partners are going to want to integrate more the more people are using it. 1620 01:27:50,100 --> 01:27:52,850 So it will get more valuable if we can get it to scale. 1621 01:27:53,060 --> 01:27:56,810 The best way to do that is to basically have a model where it is affordable 1622 01:27:56,980 --> 01:27:58,360 and free for people to use. 1623 01:27:58,940 --> 01:28:02,640 But there are going to be a lot of ways that I think we can scale revenue. 1624 01:28:03,650 --> 01:28:05,860 Subscriptions is going to be a big one. 1625 01:28:06,070 --> 01:28:09,830 You know, basically, just because the base version of it is 1626 01:28:09,950 --> 01:28:12,997 free and we want the free version to be excellent doesn't mean that there are 1627 01:28:13,080 --> 01:28:16,417 things that just naturally aren't going to be economical to include in that. 1628 01:28:16,500 --> 01:28:20,667 So, you know, if you want the kind of— you know, we talked about this earlier. 1629 01:28:20,750 --> 01:28:25,257 If you want to start using Muse Wonder in Muse tomorrow, then, you know, 1630 01:28:25,340 --> 01:28:28,307 the way that you'll do that is you'll pay for a subscription. 1631 01:28:28,390 --> 01:28:30,590 So I think that that will be a big business. 1632 01:28:31,060 --> 01:28:33,597 There are a bunch of other things around commerce that we're quite excited about 1633 01:28:33,680 --> 01:28:37,817 too, where we basically think— we believe that Muse is actually going to make people 1634 01:28:37,900 --> 01:28:42,187 money and save people money by using it, and that over time there are lots 1635 01:28:42,270 --> 01:28:46,360 of different ways to effectively take a small cut of what you are saving or 1636 01:28:46,450 --> 01:28:50,740 making for people, and that will end up being a very large revenue stream. 1637 01:28:51,280 --> 01:28:52,697 And then there's all the other things, right? 1638 01:28:52,780 --> 01:28:56,790 So, you think Muse will generally just cost negative money for people? 1639 01:28:57,040 --> 01:28:58,417 because it helps them out. 1640 01:28:58,500 --> 01:29:00,547 Well, there are already all these sites that I think are interesting where people 1641 01:29:00,630 --> 01:29:03,967 are like, go look through my emails, find a bunch of subscriptions I don't 1642 01:29:04,050 --> 01:29:06,257 actually use, cancel them, whatever. 1643 01:29:06,340 --> 01:29:10,967 Um, yeah, so yeah, I mean, that, um, I don't know if that's the thing that we 1644 01:29:11,050 --> 01:29:13,887 would monetize, but, but like, but you get, you get, you kind of get the 1645 01:29:13,970 --> 01:29:15,557 point and where it helps you make money. 1646 01:29:15,640 --> 01:29:18,437 I mean, one of the ideas that we kind of talk about, I mean, this is, 1647 01:29:18,520 --> 01:29:21,437 this is something that I think you've been kind of obsessed with, is making it like 1648 01:29:21,520 --> 01:29:23,570 a business in a box for small businesses. 1649 01:29:24,400 --> 01:29:26,657 So someone can just kind of— it can help them administer all these 1650 01:29:26,740 --> 01:29:27,567 parts of what they're doing. 1651 01:29:27,650 --> 01:29:30,867 And you can imagine that being very valuable and being very profitable, 1652 01:29:30,950 --> 01:29:33,160 both for the business owner and for Meta. 1653 01:29:35,080 --> 01:29:37,480 But I think one of the things that's interesting 1654 01:29:38,120 --> 01:29:43,587 is, like, OK, if we're building the models to build the personal agents 1655 01:29:43,670 --> 01:29:47,007 and the business agent, there is this large surplus of value 1656 01:29:47,090 --> 01:29:49,047 because now the model, it's very general, right? 1657 01:29:49,130 --> 01:29:50,887 So it doesn't just do those things. 1658 01:29:50,970 --> 01:29:53,270 We'll also have the API that'll be profitable. 1659 01:29:53,850 --> 01:29:58,180 A big part of why we want to start the Meta Enterprise Platform 1660 01:29:58,270 --> 01:30:01,687 and have CJ Desai join us to build this is we have this view, which is like, 1661 01:30:01,770 --> 01:30:05,720 all right, yeah, sure. So traditionally Meta hasn't been an enterprise company, 1662 01:30:05,940 --> 01:30:08,817 but we believe that a huge part of the challenge is building 1663 01:30:08,900 --> 01:30:10,867 the leading models in the first place. And they're very general. 1664 01:30:10,950 --> 01:30:16,790 And once we do that, if we make a significant investment in building 1665 01:30:17,240 --> 01:30:21,497 an enterprise group that can provide that in the way that enterprises need 1666 01:30:21,580 --> 01:30:24,087 to consume it, then I mean, I think a lot of enterprises are 1667 01:30:24,170 --> 01:30:25,467 going to want to use Muse Wonder. 1668 01:30:25,550 --> 01:30:27,837 It's going to be an excellent model and affordable. 1669 01:30:27,920 --> 01:30:31,797 And like, it's so, um, so I think that that is going to end 1670 01:30:31,880 --> 01:30:33,640 up being a very big business too. 1671 01:30:33,930 --> 01:30:38,470 And all of these things put together, coupled with the fact that we were already 1672 01:30:38,560 --> 01:30:41,230 starting off as a very profitable business that can, 1673 01:30:41,730 --> 01:30:45,937 that has good credit and can kind of underwrite these large investments, 1674 01:30:46,020 --> 01:30:50,237 I think puts us in a pretty good position as a lab to be able to build out 1675 01:30:50,320 --> 01:30:54,027 the compute that we need to build leading products, build up the network around 1676 01:30:54,110 --> 01:30:59,617 this, and position the company to really lead in the area for decades to come. 1677 01:30:59,700 --> 01:31:00,167 Okay. 1678 01:31:00,250 --> 01:31:02,577 So, you think ultimately there'll just be a lot of revenue from these products, 1679 01:31:02,660 --> 01:31:04,837 and so they'll be self-funding in fairly short order? 1680 01:31:04,920 --> 01:31:05,707 Yeah. 1681 01:31:05,790 --> 01:31:07,547 I mean, that's a much shorter answer than mine. 1682 01:31:07,630 --> 01:31:10,257 It's like— But you're saying people are underestimating— 1683 01:31:10,340 --> 01:31:13,640 The question was, Are we going to be able to make money from this? 1684 01:31:13,930 --> 01:31:15,180 The answer better be yes. 1685 01:31:16,100 --> 01:31:19,637 I think the history— Yes. The answer is yes. 1686 01:31:19,720 --> 01:31:21,977 But the history is very front-loaded expense, right? 1687 01:31:22,060 --> 01:31:25,397 Well, the history of AI as an industry, I think, to some degree is like, 1688 01:31:25,480 --> 01:31:28,937 you know, people have always sort of been hand-wringing about, you know, how are 1689 01:31:29,020 --> 01:31:30,527 these businesses going to make money? 1690 01:31:30,610 --> 01:31:34,400 And I think, you know, every year people are sort of generally 1691 01:31:34,490 --> 01:31:35,780 surprised at the upside. 1692 01:31:36,320 --> 01:31:39,270 That superintelligence is Is a useful, incredibly valuable— 1693 01:31:39,370 --> 01:31:42,077 Yeah, exactly. This is what— this is kind of Meta's whole playbook, right? 1694 01:31:42,160 --> 01:31:45,000 Is we build something, we get it to a billion people, 1695 01:31:45,210 --> 01:31:49,627 then we scale the revenue while growing our revenue primarily 1696 01:31:49,710 --> 01:31:53,220 in the meantime by continuing to scale the things that are already 1697 01:31:53,380 --> 01:31:54,967 kind of at scale and growing. Yeah. 1698 01:31:55,050 --> 01:31:58,220 I think this is also one of the dynamics, like, I think, 1699 01:31:59,310 --> 01:32:01,637 you know, I think Dario's talked about this as well, 1700 01:32:01,720 --> 01:32:04,307 but like, just in general, because it requires so much advanced 1701 01:32:04,390 --> 01:32:06,650 planning to build these models. 1702 01:32:07,020 --> 01:32:09,860 Like, if you look at the R&D expense at any given time, 1703 01:32:10,230 --> 01:32:13,777 that is like over— an overlapping R&D expense of multiple model 1704 01:32:13,860 --> 01:32:15,280 generations at once. Yes. 1705 01:32:15,650 --> 01:32:19,200 And, you know, these, they require a long time to work on. 1706 01:32:19,370 --> 01:32:22,247 And then ultimately, you know, you see like immense payoff 1707 01:32:22,330 --> 01:32:23,577 when, when you ultimately deliver them. 1708 01:32:23,660 --> 01:32:27,290 And this is kind of what we saw with the Muse Spark family of models. 1709 01:32:27,380 --> 01:32:30,730 Like, you know, again, we started working on these a long time ago. 1710 01:32:31,050 --> 01:32:34,547 And are seeing like incredible consumer adoption through the Muse product. 1711 01:32:34,630 --> 01:32:38,057 The other thing is that we use the models to accelerate our core business, right? 1712 01:32:38,140 --> 01:32:41,137 So I mean, you have this core business that's hundreds of billions of dollars 1713 01:32:41,220 --> 01:32:42,887 and still growing extremely quickly. 1714 01:32:42,970 --> 01:32:46,890 And now, like, if you can grow it a few percent faster by using these models 1715 01:32:46,980 --> 01:32:50,977 to accelerate these things, then that ends up funding a huge amount 1716 01:32:51,060 --> 01:32:52,910 of this too as you're scaling it out. 1717 01:32:53,190 --> 01:32:56,340 The Meta core business is a, is a like 8th wonder of the world. 1718 01:32:56,530 --> 01:32:58,067 It's extremely good business. It really is. 1719 01:32:58,150 --> 01:33:03,990 is quite incredible and is growing at sort of astonishing rates. 1720 01:33:05,080 --> 01:33:09,210 Yeah, you guys really cooked with that one. 1721 01:33:10,040 --> 01:33:13,167 Yeah, I think you're also getting at something, Alex, which is like there's 1722 01:33:13,250 --> 01:33:16,880 an interesting accounting artifact where if you go build a data center, 1723 01:33:17,340 --> 01:33:20,887 you have CapEx and you get to depreciate that over a useful life of many years. 1724 01:33:20,970 --> 01:33:24,187 And people debate what that useful life is, but the point is you do get 1725 01:33:24,270 --> 01:33:28,727 to depreciate it, whereas a lot of the AI costs are also just 1726 01:33:28,810 --> 01:33:31,910 in researcher salaries, and those salaries are being spent on, 1727 01:33:32,110 --> 01:33:35,780 you know, model training that also has a useful life of many years, 1728 01:33:36,110 --> 01:33:37,987 but you do not get to capitalize generally. 1729 01:33:38,070 --> 01:33:41,577 And that's also a financial advantage for us building out the data centers as 1730 01:33:41,660 --> 01:33:44,990 opposed to a bunch of the other companies that are paying clouds, 1731 01:33:45,200 --> 01:33:48,300 whereas they're paying the annual rate, but then at some point 1732 01:33:49,250 --> 01:33:52,797 the GPUs will be fully depreciated, but we're probably still going to be using 1733 01:33:52,880 --> 01:33:56,797 them for a while because they I mean, the value of even like an H100 at this 1734 01:33:56,880 --> 01:33:59,130 point is higher than it was last year, right? 1735 01:33:59,220 --> 01:34:01,820 Just because if compute continues being constrained. 1736 01:34:02,510 --> 01:34:06,140 So yeah, I think it's— yeah, I think— 1737 01:34:06,970 --> 01:34:08,727 Any advice for us with OpenRadar? 1738 01:34:08,810 --> 01:34:10,227 We have agreed to acquire them. 1739 01:34:10,310 --> 01:34:11,147 It has not closed yet. 1740 01:34:11,230 --> 01:34:17,440 Yeah, I think part of the exciting— part of what we're starting to see 1741 01:34:17,530 --> 01:34:20,567 in terms of the models, especially the frontier models that are 1742 01:34:20,650 --> 01:34:24,300 being developed, like Muse Wonder versus, let's say, a Fable or an Astra, 1743 01:34:24,450 --> 01:34:30,537 is that we actually are starting to focus in quite different directions 1744 01:34:30,620 --> 01:34:31,417 from the other models. 1745 01:34:31,500 --> 01:34:33,797 And I think there's a lot of intentionality that goes into the model 1746 01:34:33,880 --> 01:34:36,587 development process that we've discussed as part of this, you know, 1747 01:34:36,670 --> 01:34:39,920 in this conversation, that is genuinely different 1748 01:34:40,010 --> 01:34:41,507 from how other models are focusing. 1749 01:34:41,590 --> 01:34:43,807 And so, you know, when these models kind of get 1750 01:34:43,890 --> 01:34:48,347 collapsed to the same set of benchmarks, that elides all of the— 1751 01:34:48,430 --> 01:34:53,187 This is a good point. The interesting, um, sort of like, uh, texture that differs 1752 01:34:53,270 --> 01:34:54,267 between the various models. 1753 01:34:54,350 --> 01:34:58,357 And so I think one thing that is an exciting opportunity for OpenRadar, 1754 01:34:58,440 --> 01:35:02,860 generally speaking, is like, you know, um, helping to Helping developers 1755 01:35:02,990 --> 01:35:06,577 understand what these sort of subtle differences are so that they can make 1756 01:35:06,660 --> 01:35:10,110 the most informed decisions about what models ultimately they select. 1757 01:35:10,290 --> 01:35:13,390 And then also you can imagine the flip side of that, which is, 1758 01:35:14,500 --> 01:35:16,950 you have all these developers using these models. 1759 01:35:17,040 --> 01:35:20,587 And so those developers are great people to ask about what are the interesting 1760 01:35:20,670 --> 01:35:25,927 textual differences between these models that are more than just tweet length. 1761 01:35:26,010 --> 01:35:30,930 Like, this model's good, this model's bad, more than just that level of insight. 1762 01:35:31,020 --> 01:35:35,310 And I think this is like getting to the wine sommelier 1763 01:35:35,980 --> 01:35:39,277 version or form of understanding the models and their sort of relative 1764 01:35:39,360 --> 01:35:41,987 strengths and weaknesses, I think would be quite exciting. 1765 01:35:42,070 --> 01:35:45,117 Okay, so based on your experience, the OpenRadar view that there will be 1766 01:35:45,200 --> 01:35:49,037 a diversity of models in the future and they're optimized for different tasks and— 1767 01:35:49,120 --> 01:35:52,670 I agree with that. Exactly. That's just your guys' experience based on— 1768 01:35:53,200 --> 01:35:56,497 Yeah, we are focusing On different things. 1769 01:35:56,580 --> 01:36:01,210 We care a lot more about personal and professional agents, my observation, 1770 01:36:01,340 --> 01:36:04,930 than the other major labs, which is good. 1771 01:36:05,180 --> 01:36:06,927 There should be differentiation in the economy. 1772 01:36:07,010 --> 01:36:09,177 This is the point of capitalism to some extent. 1773 01:36:09,260 --> 01:36:10,727 Yeah. So you're going to say something? 1774 01:36:10,810 --> 01:36:12,397 No, I just really agree with that point. 1775 01:36:12,480 --> 01:36:16,517 I mean, I think that part of what I love about you guys working on OpenRouter 1776 01:36:16,600 --> 01:36:20,567 and being able to scale it is just the more infrastructure that can get 1777 01:36:20,650 --> 01:36:23,527 developed, that supports a diversity of players 1778 01:36:23,610 --> 01:36:25,710 in the ecosystem, I think is good overall. 1779 01:36:26,240 --> 01:36:30,697 I also, for the same reason, am quite a fan of some of the recent NVIDIA 1780 01:36:30,780 --> 01:36:33,580 moves to support the open source ecosystem. 1781 01:36:34,040 --> 01:36:38,120 I don't know, but this just kind of goes with our whole theory and philosophy 1782 01:36:38,210 --> 01:36:39,957 about, like, what does a good future look like? 1783 01:36:40,040 --> 01:36:43,260 Mm-hmm. We think that a good future is one where you achieve a, 1784 01:36:44,510 --> 01:36:47,430 like, you basically get prosperity by empowering people. 1785 01:36:47,640 --> 01:36:49,677 That's kind of the personal superintelligence thing that we're doing, 1786 01:36:49,760 --> 01:36:52,010 but there's the whole business version. Yeah. 1787 01:36:52,470 --> 01:36:55,270 But for a lot of the important safety topics, 1788 01:36:55,850 --> 01:36:59,690 like, our view is that the way to get there is having a diversity 1789 01:37:00,150 --> 01:37:03,567 and having checks and balances by kind of having a lot of different things out 1790 01:37:03,650 --> 01:37:06,697 there and accessible to people, not like just having a couple of good 1791 01:37:06,780 --> 01:37:08,867 models that are only in a couple companies' hands. 1792 01:37:08,950 --> 01:37:12,540 I think that that would be— from my perspective, I think that is 1793 01:37:13,370 --> 01:37:19,290 probably a riskier configuration for the future than lots of people 1794 01:37:19,380 --> 01:37:20,877 having access to different things. 1795 01:37:20,960 --> 01:37:22,417 Do you guys have a theory as to why that is? 1796 01:37:22,500 --> 01:37:28,170 Where I feel like maybe people predicted more of a winner-takes-all dynamic in AI, 1797 01:37:28,430 --> 01:37:32,260 or one person clearly— like, a story that people told back in 2023 is 1798 01:37:32,350 --> 01:37:36,270 that one AI lab will start to develop an advantage, 1799 01:37:36,600 --> 01:37:39,357 and that will allow them to get recursive self-improvement going, 1800 01:37:39,440 --> 01:37:42,977 And then they'll just pull further and further away from everyone else. 1801 01:37:43,060 --> 01:37:47,320 Whereas it has remained extremely competitive where 1802 01:37:47,610 --> 01:37:49,447 there's kind of multiple people at the frontier. 1803 01:37:49,530 --> 01:37:53,870 You guys with Wonder today are coming along with your frontier-class model. 1804 01:37:54,450 --> 01:37:58,750 Open source models have done an impressive job of keeping up, all told. 1805 01:37:59,330 --> 01:38:03,000 And so we had this story a few years ago that someone was just going 1806 01:38:03,090 --> 01:38:05,920 to run away with the prize. Why hasn't that happened? 1807 01:38:06,920 --> 01:38:11,930 I think there's a lot of underlying reasons and mechanics that drive this. 1808 01:38:12,430 --> 01:38:14,310 One certainly is just execution. 1809 01:38:14,600 --> 01:38:20,017 I think for any given frontier lab, there are periods of strong 1810 01:38:20,100 --> 01:38:24,020 execution and weak execution. And I think that is true across all the labs. 1811 01:38:24,110 --> 01:38:27,060 And so that sort of starts to average itself out over time. 1812 01:38:28,110 --> 01:38:29,710 I think another—Very diplomatic. 1813 01:38:31,200 --> 01:38:34,870 Sorry. Yeah. Is it that some of the Players in the lead screwed up? 1814 01:38:35,660 --> 01:38:37,617 Well, I think if all players, 1815 01:38:37,700 --> 01:38:39,867 when they were in the lead, executed perfectly, they would 1816 01:38:39,950 --> 01:38:41,870 probably be able to stay in the lead. 1817 01:38:42,370 --> 01:38:46,710 So I think that's one factor. I think the power and strength of distillation is 1818 01:38:46,800 --> 01:38:48,967 certainly another factor that's certainly benefiting the open source 1819 01:38:49,050 --> 01:38:51,507 ecosystem quite a bit. And I think this is just like a— 1820 01:38:51,590 --> 01:38:55,760 There's an anti-hegemonic force in the phenomenon of distilling a model. 1821 01:38:56,010 --> 01:38:59,350 Yeah. And this is just a quality of the technology. 1822 01:39:00,640 --> 01:39:04,017 It's not like anyone planned for distillation to work super duper well, 1823 01:39:04,100 --> 01:39:10,817 but it just turns out that you can get a lot of capabilities naturally 1824 01:39:10,900 --> 01:39:12,527 through some form of distillation. 1825 01:39:12,610 --> 01:39:15,197 And this is, broadly speaking, distillation is quite 1826 01:39:15,280 --> 01:39:20,080 a good force for the entire ecosystem because it means that you can pack very 1827 01:39:20,370 --> 01:39:22,917 strong intelligence capabilities into smaller and smaller models, 1828 01:39:23,000 --> 01:39:25,550 which enables us to serve them at much wider scale. 1829 01:39:25,880 --> 01:39:28,917 And that's certainly one of the very exciting stories is just 1830 01:39:29,000 --> 01:39:33,547 like really small models today are like, you know, much more powerful than the 1831 01:39:33,630 --> 01:39:35,680 biggest models a year ago or 2 years ago. 1832 01:39:36,800 --> 01:39:41,730 And then I think the last is that there's just sort of this— I think there's like, 1833 01:39:42,230 --> 01:39:48,270 I actually think a lot of people in the ecosystem want this future 1834 01:39:48,360 --> 01:39:51,610 where it is not just this like, you know, definitely. 1835 01:39:51,900 --> 01:39:56,907 One final company in all of capitalism kind of kind of end outcome. 1836 01:39:56,990 --> 01:39:58,367 Like, I think this is one of the reasons— 1837 01:39:58,450 --> 01:40:01,617 Not just individuals, but companies putting dollars against things. 1838 01:40:01,700 --> 01:40:05,707 Yeah. And I think, like, you know, this is actually one of the reasons why 1839 01:40:05,790 --> 01:40:10,127 a lot of people were excited to join Meta and Meta Superintelligence Labs 1840 01:40:10,210 --> 01:40:12,217 in the first place, was that they were just sort of like, 1841 01:40:12,300 --> 01:40:15,047 you know, I want to work at a place that is focused 1842 01:40:15,130 --> 01:40:16,967 on giving this technology to billions of people. 1843 01:40:17,050 --> 01:40:19,470 That is, like, not the stated goal necessarily 1844 01:40:19,680 --> 01:40:20,807 of all the other frontier labs. 1845 01:40:20,890 --> 01:40:23,437 And so, I think there's, like, a lot of sort of just, like, 1846 01:40:23,520 --> 01:40:26,890 natural forces where many people in the industry are sort of, 1847 01:40:26,980 --> 01:40:31,317 like, grappling with, okay, what is the actual world state 1848 01:40:31,400 --> 01:40:32,947 that I want to live in in the future? 1849 01:40:33,030 --> 01:40:37,237 And there's lots of sort of, like, decisions that people make at all levels 1850 01:40:37,320 --> 01:40:40,697 in terms of where they work or who they do business with or what business deals they 1851 01:40:40,780 --> 01:40:45,250 do that sort of, like, drive towards a more equalized outcome. 1852 01:40:46,040 --> 01:40:51,800 You get into the Some of the MSL startup, which I also wanted to ask about, which is 1853 01:40:52,840 --> 01:40:55,880 it just seems fundamentally very hard still to— 1854 01:40:57,050 --> 01:41:01,140 again, we talked about what goes into developing a frontier model, but also 1855 01:41:01,640 --> 01:41:04,890 from a standing start, like when was the standing start for MSL? 1856 01:41:05,930 --> 01:41:08,770 Well, we launched Llama 4 1857 01:41:09,270 --> 01:41:12,980 on April 4th, I think it was, 2025. 1858 01:41:13,650 --> 01:41:19,240 Right. And then I think, what was it, April 20th 1859 01:41:20,410 --> 01:41:22,490 or 21st was, I think, when I was like, 1860 01:41:22,580 --> 01:41:29,420 hey, when I started calling people for advice on what to do. 1861 01:41:29,790 --> 01:41:34,377 So, I remember I was in a staff meeting and I got, like, the WhatsApp 1862 01:41:34,460 --> 01:41:35,927 notification, like, have time for a call? 1863 01:41:36,010 --> 01:41:41,260 I was like, huh, interesting. Okay, so MSL is a little over 2 years old. 1864 01:41:41,760 --> 01:41:45,390 No, no, no, 1 year. 1 year. Is it '25? Oh, it's '26. 1865 01:41:45,640 --> 01:41:47,057 Oh, sorry, I'm living in the future. 1866 01:41:47,140 --> 01:41:50,190 Yeah, you are in the future. MSL is a little over 1 year old. 1867 01:41:50,560 --> 01:41:55,940 Um, yeah, what are your reflections a little over a year in on what was 1868 01:41:56,730 --> 01:42:00,110 load-bearing, as the IAs would say, in how it came together? 1869 01:42:00,490 --> 01:42:04,830 Well, I mean, I think that there was this whole thing early on which is just like, 1870 01:42:05,410 --> 01:42:07,367 first we had to pull together this core group. 1871 01:42:07,450 --> 01:42:07,577 Right? 1872 01:42:07,660 --> 01:42:14,460 So, um, so Nat and Alex and I, we were like talking a bit up front, um, 1873 01:42:15,300 --> 01:42:19,300 and, you know, at first we were, you know, brainstorming. 1874 01:42:19,720 --> 01:42:23,717 Yeah, like I was talking to Nat about, could we— is there a way that we can like 1875 01:42:23,800 --> 01:42:27,057 structure this so that way it's like a win for Alex's shareholders and like 1876 01:42:27,140 --> 01:42:28,640 we can like get him over here? 1877 01:42:29,060 --> 01:42:32,730 And then, so then we were talking about, uh, like what would the plan be? 1878 01:42:33,230 --> 01:42:37,777 And, and then basically I was just, I was in this mode where I was just like 1879 01:42:37,860 --> 01:42:40,107 asking a lot of different people what we should do. 1880 01:42:40,190 --> 01:42:43,780 And like every time that, that we would talk, 1881 01:42:44,820 --> 01:42:48,617 I'd come away with like 3 more things of like, oh yeah, this is what we should do. 1882 01:42:48,700 --> 01:42:52,200 It's like very like high signal, very like a lot of strategic clarity. 1883 01:42:52,670 --> 01:42:56,800 And so we were brainstorming and working on this for a bit. 1884 01:42:57,210 --> 01:42:59,967 And then Nat and Daniel joined and we pulled together a few 1885 01:43:00,050 --> 01:43:01,930 of the the initial researchers. 1886 01:43:02,050 --> 01:43:05,510 But like, in the beginning, it was sort of like the Supernova thing 1887 01:43:05,640 --> 01:43:08,437 where we just kind of like jammed a bunch of people together. 1888 01:43:08,520 --> 01:43:10,977 And I don't know, it was like, I don't think either of us 1889 01:43:11,060 --> 01:43:13,017 slept that much for the first couple of months. 1890 01:43:13,100 --> 01:43:16,690 No. I like, don't even, I kind of have this mental block almost 1891 01:43:16,900 --> 01:43:19,570 where I like, I kind of almost 1892 01:43:20,940 --> 01:43:23,947 cleared my calendar for like a month or 2 1893 01:43:24,030 --> 01:43:28,280 and was just meeting researchers and talking to them. 1894 01:43:28,450 --> 01:43:32,910 And I remember even before we announced the deal where you were going to come 1895 01:43:33,000 --> 01:43:35,877 over, we were basically like, you were recruiting people and people 1896 01:43:35,960 --> 01:43:38,910 were like, what is going on? Like, it's like, what is this? 1897 01:43:39,050 --> 01:43:42,927 But, um, but then there was the whole question, which was a really 1898 01:43:43,010 --> 01:43:45,590 hands-on data labeler. 1899 01:43:46,800 --> 01:43:48,927 But, but that was, that was this whole question 1900 01:43:49,010 --> 01:43:50,557 too, where it's like, all right, well, like, 1901 01:43:50,640 --> 01:43:52,807 okay, so if we're not happy with the trajectory of the current 1902 01:43:52,890 --> 01:43:58,270 program, Like, who were the right set of people to lead the program going forward? 1903 01:43:58,730 --> 01:44:02,237 And Alex and I had spent a bunch of time talking over the years. 1904 01:44:02,320 --> 01:44:05,737 I mean, maybe like, whatever, once every few months or something we'd talk. 1905 01:44:05,820 --> 01:44:09,920 And I was always just very impressed by how he thought about things strategically. 1906 01:44:10,080 --> 01:44:14,870 And I kind of had this view of like, all right, yeah, currently he's doing 1907 01:44:15,620 --> 01:44:18,000 this very operational data business. 1908 01:44:19,960 --> 01:44:26,257 But it always felt to me like he was the sharpest entrepreneur of his 1909 01:44:26,340 --> 01:44:29,930 generation, of the ones who I had had a chance to meet. 1910 01:44:32,310 --> 01:44:34,937 And like, it always just struck me, it's like, all right, like, 1911 01:44:35,020 --> 01:44:37,977 this is a guy who, yeah, he's like running this like kind of very 1912 01:44:38,060 --> 01:44:40,610 grindy operational business, but like— 1913 01:44:40,730 --> 01:44:42,527 Lovely business still, by the way. What's up? 1914 01:44:42,610 --> 01:44:48,777 We're big investors. It's not a— it's a— there's more Commentary on the difficulty 1915 01:44:48,860 --> 01:44:49,737 of running the business. 1916 01:44:49,820 --> 01:44:52,907 When you run the 8th wonder of the world, Alex, a lot of businesses look 1917 01:44:52,990 --> 01:44:54,740 like a grindy operational business. 1918 01:44:56,330 --> 01:44:58,170 It's just like my view of things. 1919 01:44:58,290 --> 01:45:02,920 It's like grindy, grindy. No, no, no. 1920 01:45:04,840 --> 01:45:08,467 But no, so it's, so I just kind of had this view. 1921 01:45:08,550 --> 01:45:11,930 I'm like, all right, like, even though this is sort of 1922 01:45:13,220 --> 01:45:16,270 I think maybe when we got started, a lot of people were like, 1923 01:45:16,560 --> 01:45:18,067 okay, like, how is this going to work? 1924 01:45:18,150 --> 01:45:22,437 It's like the kind of like the social media guy is working 1925 01:45:22,520 --> 01:45:23,940 with the like data guy. 1926 01:45:24,190 --> 01:45:27,157 Like, is that like, are we going to build a frontier lab here or what? 1927 01:45:27,240 --> 01:45:30,157 But, but I mean, but my, but I mean, my like really strong 1928 01:45:30,240 --> 01:45:33,447 conviction, because I talked to like all the leaders of all the labs, like all the 1929 01:45:33,530 --> 01:45:35,367 heads of research, like all this stuff. 1930 01:45:35,450 --> 01:45:38,537 And like, I was like, no, Alex is the guy. Like, he's the fucking guy. 1931 01:45:38,620 --> 01:45:40,820 And like, and we're going to, and basically, 1932 01:45:40,920 --> 01:45:46,260 like, we're going to basically be able to, to, to go pull together a team and recruit 1933 01:45:46,720 --> 01:45:48,547 the people that we need to go do this. 1934 01:45:48,630 --> 01:45:54,387 And like, we need this kind of very, this kind of like strategic clarity 1935 01:45:54,470 --> 01:45:58,020 that he has around the different ways that we want to approach this, 1936 01:45:58,140 --> 01:46:01,060 the kind of extreme focus on talent density, 1937 01:46:01,770 --> 01:46:06,357 on kind of like the clarity of communicating the different vision, 1938 01:46:06,440 --> 01:46:09,090 which we kind of fashioned together over this period. 1939 01:46:10,610 --> 01:46:13,450 And a bunch of things that I think were 1940 01:46:14,240 --> 01:46:17,950 kind of extreme but have basically worked, 1941 01:46:18,200 --> 01:46:23,290 like having this, this like very flat culture was something 1942 01:46:23,380 --> 01:46:25,877 that you were kind of really obsessed with, right? 1943 01:46:25,960 --> 01:46:28,757 It's, I mean, you have like, was it almost 200 direct reports? 1944 01:46:28,840 --> 01:46:30,927 That's like, we have, it's not, it doesn't, it's not quite 1945 01:46:31,010 --> 01:46:32,137 that extreme in practice. 1946 01:46:32,220 --> 01:46:35,137 I think that there's like, like team leads and all, and all this stuff. 1947 01:46:35,220 --> 01:46:36,437 My muse helps me manage that. 1948 01:46:36,520 --> 01:46:41,017 Yeah, well, that's— But like, but there were these series of things 1949 01:46:41,100 --> 01:46:45,980 that I think quite early we, you kind of articulated, and I was like, 1950 01:46:46,770 --> 01:46:53,280 okay, this is like kind of extreme, but it seems like directionally right, 1951 01:46:53,740 --> 01:46:55,077 and like we should go do this. 1952 01:46:55,160 --> 01:46:57,457 And, you know, I went and like talked to other people who were more like 1953 01:46:57,540 --> 01:47:01,667 the people you'd conventionally think of, and it's like, it was just very much like, 1954 01:47:01,750 --> 01:47:04,417 I don't know what you'd expect expect a big company to operate. 1955 01:47:04,500 --> 01:47:07,007 And I was like, no, we want to do something that's— 1956 01:47:07,090 --> 01:47:11,590 As you describe this, I'll note 4 things that I think even other 1957 01:47:12,300 --> 01:47:15,300 opinionated, effective founder CEOs would not have done. 1958 01:47:16,140 --> 01:47:20,640 One, you just called the recent prior effort by 1959 01:47:21,770 --> 01:47:26,187 a bunch of people at the company a failure and not good enough. 1960 01:47:26,270 --> 01:47:29,027 I mean, you were more diplomatic than that, but the act of standing up off 1961 01:47:29,110 --> 01:47:32,527 the trajectory— Yeah, but the act of standing up So, you know, 1962 01:47:32,610 --> 01:47:34,237 people can read between the lines. 1963 01:47:34,320 --> 01:47:36,577 So sometimes, yeah, sometimes you have to reboot a team. 1964 01:47:36,660 --> 01:47:37,493 That's one. 1965 01:47:37,830 --> 01:47:43,540 Thing 2 is you are willing to pay ungodly amounts of money for effectively talent. 1966 01:47:43,830 --> 01:47:47,337 Like, you're paying a lot of money to bring Alex and some other people, but, 1967 01:47:47,420 --> 01:47:50,460 you know, a lot of money for Alex in particular over. 1968 01:47:52,720 --> 01:47:57,387 Thing 3— And 49% of this very good grindy business. 1969 01:47:57,470 --> 01:48:01,020 Yeah, exactly. But let's keep the story straight. 1970 01:48:01,520 --> 01:48:04,190 So there's that. 1971 01:48:04,690 --> 01:48:08,980 Thing 3 is, I'll note, Alex had never trained a frontier model before. 1972 01:48:09,360 --> 01:48:13,527 And so you were taking some amount of a bet on Alex's potential to do 1973 01:48:13,610 --> 01:48:17,320 something where he had done stuff in AI, but no offense, 1974 01:48:17,410 --> 01:48:20,657 he just had not done the specific thing that you were being hired to do. 1975 01:48:20,740 --> 01:48:23,537 And then 4th, you were saying you cleared your calendar 1976 01:48:23,620 --> 01:48:25,327 and just only did this for 2 months. 1977 01:48:25,410 --> 01:48:28,837 I think a lot of CEOs would've had the business review and would have had 1978 01:48:28,920 --> 01:48:32,920 the board thing and they would have had the customer stuff and things like that. 1979 01:48:33,130 --> 01:48:37,387 Those 4 seem to me pretty interesting where I think a lot 1980 01:48:37,470 --> 01:48:40,017 of CEOs would not have done those. Yeah, that's interesting. 1981 01:48:40,100 --> 01:48:42,637 I definitely think— I mean, one thing you may not appreciate this 1982 01:48:42,720 --> 01:48:46,390 about yourself, but I think one thing that is quite special about Marc is 1983 01:48:47,730 --> 01:48:51,357 the degree to which, like, once he concludes that something is 1984 01:48:51,440 --> 01:48:56,150 the right thing or something makes sense, he just like fully shifts everything 1985 01:48:56,240 --> 01:48:58,140 to be consistent with that belief set. 1986 01:48:58,610 --> 01:49:04,540 And it is quite like, to your point, I think like the inertia of most, even 1987 01:49:05,000 --> 01:49:08,417 very successful founders who like are opinionated and edgy or whatnot, 1988 01:49:08,500 --> 01:49:11,750 would just be to like, oh, I still have like my whole calendar. 1989 01:49:11,920 --> 01:49:15,920 And like, so I'll just like talk about doing this thing and hope that somebody 1990 01:49:16,050 --> 01:49:21,097 does it kind of situation. Whereas I think, No, I think when Marc concludes 1991 01:49:21,180 --> 01:49:23,887 that something is the thing he has to do or makes a lot 1992 01:49:23,970 --> 01:49:29,147 of sense, then he really does, like, he has, like, maybe very high 1993 01:49:29,230 --> 01:49:30,227 learning rate in that sense. 1994 01:49:30,310 --> 01:49:34,260 Like, he's very willing to, like, pivot entire things to be consistent with it. 1995 01:49:34,610 --> 01:49:39,947 And I think that, like, you know, one very underappreciated thing 1996 01:49:40,030 --> 01:49:43,447 is the degree to which so much of the company had to pivot 1997 01:49:43,530 --> 01:49:50,207 to support MSL as well. Like, I think that while it was a kind of like this more 1998 01:49:50,290 --> 01:49:55,710 startup-y effort within a larger company, you know, the needs of MSL 1999 01:49:55,800 --> 01:49:58,967 were still quite large. Like, we needed a huge amount of help on infrastructure. 2000 01:49:59,050 --> 01:50:01,347 We needed these large-scale data centers to be built. 2001 01:50:01,430 --> 01:50:05,060 We needed like lots of support from many other parts of the company. 2002 01:50:06,140 --> 01:50:09,850 And especially as Muse has been successful, we've also needed lots of— 2003 01:50:09,980 --> 01:50:12,357 Yeah. Deep collaboration with the rest of the company. 2004 01:50:12,440 --> 01:50:18,530 And this is one thing that I think kind of only is possible because Mark is 2005 01:50:18,940 --> 01:50:23,237 the founder CEO, and he can ensure that we have kind 2006 01:50:23,320 --> 01:50:29,540 of like a global optimization towards this sort of like AI maximalist version of Meta 2007 01:50:30,120 --> 01:50:33,040 versus the natural inertia of an organization towards 2008 01:50:34,040 --> 01:50:37,130 just local optimization ad infinitum. 2009 01:50:37,800 --> 01:50:41,930 But it's also a lot of how we work together is it's important to me to be 2010 01:50:42,390 --> 01:50:46,807 in the details of this work so that we can get everything around the company 2011 01:50:46,890 --> 01:50:48,850 to go do what we need to do. Right. 2012 01:50:48,980 --> 01:50:51,477 And so it's like, I mean, yeah, I'll have opinions about this stuff, but 2013 01:50:51,560 --> 01:50:53,977 I'm not like, I'm not the lead researcher. Right. 2014 01:50:54,060 --> 01:50:58,820 It's like, it's, you know, it's, but I think that's kind of a lot of, 2015 01:50:58,940 --> 01:51:02,907 I think how we work together well is that, yeah, it's, I mean, we're, 2016 01:51:02,990 --> 01:51:05,697 we're kind of like, quite in the details of all of this stuff, 2017 01:51:05,780 --> 01:51:08,660 but running the lab is a big thing. 2018 01:51:08,990 --> 01:51:11,667 Running the rest of the company in order to support this is like 2019 01:51:11,750 --> 01:51:13,250 the effort is a big thing. 2020 01:51:13,670 --> 01:51:16,797 You almost like, you can't really get this done, I think, without 2021 01:51:16,880 --> 01:51:18,000 both of those pieces. 2022 01:51:18,710 --> 01:51:22,427 And one thing that I think was quite important about when we set 2023 01:51:22,510 --> 01:51:27,260 the whole thing up is Meta, the company, invested a huge amount 2024 01:51:27,350 --> 01:51:31,140 to stand up MSL, and then we very clearly told everyone 2025 01:51:31,230 --> 01:51:33,140 like, hey, there's no deadlines. 2026 01:51:33,400 --> 01:51:37,520 We are going to take things at the right pace, which I think on the one hand could 2027 01:51:37,610 --> 01:51:39,777 be like, you know, absolutely terrifying. 2028 01:51:39,860 --> 01:51:42,947 It's like you've just like stood up this extremely expensive unit and you're 2029 01:51:43,030 --> 01:51:46,407 telling everyone like, hey— Yeah, and you're worried inside a larger company 2030 01:51:46,490 --> 01:51:49,367 that like, okay, 7 years from now maybe something will ship, right? 2031 01:51:49,450 --> 01:51:50,283 Yeah, totally. 2032 01:51:50,410 --> 01:51:54,917 And so, but I think that like, first, I think one of the cool things 2033 01:51:55,000 --> 01:51:56,957 about, you know, focusing on talent density and hiring all 2034 01:51:57,040 --> 01:52:01,087 these very brilliant people who have done so much already, that's very impressive, 2035 01:52:01,170 --> 01:52:05,180 is, you know, everybody had like a lot of motivation to ship 2036 01:52:05,640 --> 01:52:09,677 and everybody wanted to sort of like leave their mark, so to speak, on the field. 2037 01:52:09,760 --> 01:52:13,520 And so everybody was like heavily, highly self-motivated 2038 01:52:14,020 --> 01:52:15,570 to move as quickly as possible. 2039 01:52:15,850 --> 01:52:20,230 And one of the things that's been funny and I've had to continue reminding 2040 01:52:20,320 --> 01:52:25,900 the team of is like, you know, it is a team of a lot of insecure overachievers. 2041 01:52:26,200 --> 01:52:29,617 So like at any one point, if I ask them like, hey, 2042 01:52:29,700 --> 01:52:32,850 how good do you think— where do you think we'll be in 3 months? 2043 01:52:33,160 --> 01:52:36,127 I always get the most like— Everyone always says it's awful. It's awful. 2044 01:52:36,210 --> 01:52:38,507 We're screwed. Yeah. These like very pessimistic answers. 2045 01:52:38,590 --> 01:52:42,210 And then I record it and I like 3 months later I'm like, hey, 2046 01:52:42,760 --> 01:52:43,757 you know, we just shipped this model. 2047 01:52:43,840 --> 01:52:47,677 Let me just show you what all of you said, where all of you said we 2048 01:52:47,760 --> 01:52:51,560 would be in 3 3 months ago. And, uh, and this is— 2049 01:52:51,770 --> 01:52:53,897 And has that caused them to get any more optimistic or no? 2050 01:52:53,980 --> 01:52:56,227 No, they're still— they're very self-critical. 2051 01:52:56,310 --> 01:52:59,510 I think it's, it's not, it's not that they're pessimistic about— 2052 01:53:00,110 --> 01:53:04,277 I think that the average person seems quite optimistic about MSL and very 2053 01:53:04,360 --> 01:53:06,777 self-critical about their own project. Yeah. 2054 01:53:06,860 --> 01:53:09,990 Which I think is actually quite a healthy, a healthy dynamic. 2055 01:53:10,780 --> 01:53:12,950 Um, but, but the trend continues. 2056 01:53:13,040 --> 01:53:16,707 Like, I think, I think, and, and this is, I mean, in many ways Muse Wonder is like 2057 01:53:16,790 --> 01:53:21,170 this incredible milestone for everyone because it is a— 2058 01:53:21,290 --> 01:53:23,797 The team was very, very critical. Yeah. 2059 01:53:23,880 --> 01:53:25,920 That was working on it along the way. 2060 01:53:26,130 --> 01:53:28,840 When did you guys know that Muse Wanderer was good? 2061 01:53:31,600 --> 01:53:33,220 Um, that's a good question. 2062 01:53:33,390 --> 01:53:37,140 I think that we, uh, so there's all these intermediate drawings. 2063 01:53:37,230 --> 01:53:39,147 You're peeking in the oven and it's like, I don't know, it seems 2064 01:53:39,230 --> 01:53:40,187 kind of gooey still. 2065 01:53:40,270 --> 01:53:42,987 And yeah, yeah, going back to it, I mean, there are all these intermediate 2066 01:53:43,070 --> 01:53:46,657 intermediate signals that you can get along the way that things are good or bad. 2067 01:53:46,740 --> 01:53:48,860 And what are the intermediate signals? 2068 01:53:49,530 --> 01:53:53,240 So in pre-training, you have a lot of perplexity-based evaluation. 2069 01:53:53,370 --> 01:53:57,660 So you have a different regime of evals that can help you in pre-training. 2070 01:53:57,750 --> 01:54:01,840 And then you have the more traditional benchmarks that can help you towards RL. 2071 01:54:03,250 --> 01:54:06,007 But no, I mean, this model, 2072 01:54:06,090 --> 01:54:08,847 kind of as you mentioned, has been cooking for quite a while. 2073 01:54:08,930 --> 01:54:12,637 And You know, a lot of— I think one of the funniest 2074 01:54:12,720 --> 01:54:18,730 things about AI research is that, you know, there's a lot of times where 2075 01:54:18,850 --> 01:54:22,397 people just sort of like are kind of squinting at a bunch of dots and being 2076 01:54:22,480 --> 01:54:27,357 like, yeah, looks like it's good, or, uh, looks a little bit off. 2077 01:54:27,440 --> 01:54:31,860 Like, one of the funniest things is the people have protractors, 2078 01:54:33,490 --> 01:54:38,827 like analog protractors that they put against you know, a set of lines on their 2079 01:54:38,910 --> 01:54:40,870 monitor to like measure an angle. 2080 01:54:41,210 --> 01:54:45,250 And then they, um, so he's not come for this job yet. 2081 01:54:45,340 --> 01:54:47,507 So I think there's, I think there's like a huge amount 2082 01:54:47,590 --> 01:54:52,677 of like, uh, like it's very, um, there's more vibes, um, driving like, okay. 2083 01:54:52,760 --> 01:54:56,477 But when did we have this conversation where, when did you know it was good? 2084 01:54:56,560 --> 01:55:01,980 Um, well, we had this debate after it was done pre-training where it was okay. 2085 01:55:03,690 --> 01:55:09,820 How long will it take in RL before it is better than Muse Spark? 2086 01:55:10,650 --> 01:55:13,450 And I remember the folks on the pre-training team 2087 01:55:14,070 --> 01:55:16,200 were just trying to set expectations. 2088 01:55:16,330 --> 01:55:18,830 They're like, look, sometimes this takes a while. 2089 01:55:19,370 --> 01:55:20,770 And then it was immediately. 2090 01:55:21,250 --> 01:55:24,540 And so we were like, okay, I think we're on a good path. 2091 01:55:25,170 --> 01:55:29,170 That to me was sort of—Yeah. I think when when people got— yeah. 2092 01:55:29,260 --> 01:55:30,420 We've been excited. 2093 01:55:30,670 --> 01:55:32,717 I think we've been pretty excited about it for a few months now. 2094 01:55:32,800 --> 01:55:34,600 Yeah. But there's a lot to do there. 2095 01:55:35,470 --> 01:55:39,930 But going back to the beginning, I think one of the things that was very 2096 01:55:41,270 --> 01:55:44,370 interesting and challenging in standing the thing up was like, 2097 01:55:44,560 --> 01:55:47,940 you really just wanted this like very high talent density effort. 2098 01:55:49,020 --> 01:55:53,360 And we were just kind of pulling some of the best people from Meta in and 2099 01:55:53,650 --> 01:55:55,327 recruiting a lot of the best people. 2100 01:55:55,410 --> 01:55:57,737 And I mean, you were asking, it's like, all right, well, like, okay, 2101 01:55:57,820 --> 01:56:00,990 for people who, like, if you were asking before, it's like, 2102 01:56:01,080 --> 01:56:04,580 well, Alex hadn't necessarily trained models like this. 2103 01:56:04,750 --> 01:56:08,290 But to me, one of the big things was like all of the best people were 2104 01:56:08,380 --> 01:56:09,880 very excited to work with him. 2105 01:56:10,460 --> 01:56:12,297 And I think like, it's, I mean, you, 2106 01:56:12,380 --> 01:56:16,217 I'm sure see this in all of, in a lot of the things that you do, 2107 01:56:16,300 --> 01:56:19,807 which is like, you know, it's like you, most people 2108 01:56:19,890 --> 01:56:22,390 are not kind of one-dimensional, right? 2109 01:56:22,980 --> 01:56:25,100 Yeah, maybe you do a specific thing, 2110 01:56:27,310 --> 01:56:30,690 but your ability to grow and expand is 2111 01:56:30,900 --> 01:56:36,947 kind of largely based on how effectively can you just do people who are excellent 2112 01:56:37,030 --> 01:56:40,530 in different domains want to come work with you on different things. 2113 01:56:40,740 --> 01:56:42,497 Yeah, and the great people have followership. 2114 01:56:42,580 --> 01:56:46,040 Yeah. And so I think that there was a little bit of that, 2115 01:56:47,830 --> 01:56:52,550 but also just It's even for the people that don't know you yet, 2116 01:56:53,130 --> 01:56:56,880 or like, or aren't, you know, don't know what you're about. 2117 01:56:56,970 --> 01:57:00,177 Like, are you like an impressive person and a compelling 2118 01:57:00,260 --> 01:57:02,360 person that people want to come work with? 2119 01:57:02,890 --> 01:57:06,810 Do people feel comfortable that you're going to create this environment? 2120 01:57:06,980 --> 01:57:10,897 And I think you were like very thoughtful about this early on. 2121 01:57:10,980 --> 01:57:14,737 We were kind of— Some of the stuff I think you kind of had 2122 01:57:14,820 --> 01:57:17,870 fully formed as beliefs of how you thought we should do this. 2123 01:57:18,240 --> 01:57:21,657 But I also remember when we were having the initial recruiting 2124 01:57:21,740 --> 01:57:24,410 conversations, we were sort of in real time 2125 01:57:25,080 --> 01:57:28,457 shaping— it's like, okay, well, this construct that we're talking about 2126 01:57:28,540 --> 01:57:32,167 with this person isn't quite going to work, but like, they seem good, 2127 01:57:32,250 --> 01:57:33,127 and like, they have this thought. 2128 01:57:33,210 --> 01:57:36,050 And it was just like, like, really kind of coalescing 2129 01:57:36,510 --> 01:57:38,637 this vision about what the shape of it should be in real time. 2130 01:57:38,720 --> 01:57:40,467 I mean, I don't know, I mean, I'm sure you remember that. 2131 01:57:40,550 --> 01:57:44,520 this, where it was like there were some of the initial concepts of like, 2132 01:57:45,100 --> 01:57:50,400 I mean, I think probably the most extreme thing was I think you had the concept of 2133 01:57:50,900 --> 01:57:54,150 the talent density and all and that kind of piece. 2134 01:57:54,690 --> 01:57:58,150 But I think we weren't sure upfront that it was going to have 2135 01:57:58,240 --> 01:58:00,110 to be a full reboot, right? 2136 01:58:00,280 --> 01:58:04,790 It's like that basically started when we were out talking to 2137 01:58:05,410 --> 01:58:07,330 a lot of the kind of best people. 2138 01:58:07,540 --> 01:58:11,247 And we realized that, hey, like, in order to make this as attractive 2139 01:58:11,330 --> 01:58:14,300 as possible, it should feel like, okay, we're starting this 2140 01:58:14,460 --> 01:58:15,710 thing again from scratch. 2141 01:58:15,800 --> 01:58:18,757 And yeah, we'll pull in a lot of the great people from the Llama effort 2142 01:58:18,840 --> 01:58:20,340 and FAIR and all of that too. 2143 01:58:21,010 --> 01:58:25,307 But that was a very intentional thing that we basically figured out and that you 2144 01:58:25,390 --> 01:58:28,897 figured out as we were out talking to the initial set of people of like 2145 01:58:28,980 --> 01:58:30,727 what was going to make this compelling. 2146 01:58:30,810 --> 01:58:34,477 And it ended up being really smart and like making a huge difference 2147 01:58:34,560 --> 01:58:37,737 in the culture of the lab and how it got and how it got started. 2148 01:58:37,820 --> 01:58:40,527 But I don't know, it's like being able to be that dynamic 2149 01:58:40,610 --> 01:58:43,860 about figuring out, like, what does the shape of this need to be? 2150 01:58:44,280 --> 01:58:47,290 It's not like that. I think that's a very uncommon trait. 2151 01:58:47,450 --> 01:58:49,117 It's a very entrepreneurial trait, right? 2152 01:58:49,200 --> 01:58:52,620 So, when I was thinking about, like, all right, like, this field is 2153 01:58:52,710 --> 01:58:54,257 going to keep on evolving, right? 2154 01:58:54,340 --> 01:58:56,877 So, what is the thing that's going to be important for someone who's 2155 01:58:56,960 --> 01:59:01,967 going to run a lab well? It's not like, are they an expert in whatever the latest 2156 01:59:02,050 --> 01:59:06,850 modality is of, like, or kind of of training technique for a lab. 2157 01:59:07,390 --> 01:59:09,600 It's, is the person a clear 2158 01:59:11,140 --> 01:59:14,440 thinker and able to kind of dynamically 2159 01:59:14,810 --> 01:59:18,777 update and move in, like, as kind of an entrepreneur would? 2160 01:59:18,860 --> 01:59:21,907 I think this is just like an inherently very entrepreneurial space. 2161 01:59:21,990 --> 01:59:26,530 Has this experience changed your view on the value of 2162 01:59:30,200 --> 01:59:34,080 going external, where both your reboot, 2163 01:59:34,460 --> 01:59:38,050 a bunch of the people you brought in were external, like Alex and 2164 01:59:38,500 --> 01:59:39,840 Nat and Daniel and folks. 2165 01:59:40,340 --> 01:59:44,340 But then also you're describing deciding whether you needed a reboot 2166 01:59:44,970 --> 01:59:47,970 based on going to the outside 2167 01:59:49,140 --> 01:59:52,020 intelligence of the aggregate views 2168 01:59:52,520 --> 01:59:55,647 of the researchers in the industry and getting a sense for the— But just Do 2169 01:59:55,730 --> 02:00:01,317 you feel like you'll carry forward that approach for future unrelated projects? 2170 02:00:01,400 --> 02:00:04,067 Because obviously the risk of a large company is that everyone's like, 2171 02:00:04,150 --> 02:00:08,100 oh, we got a great approach. Yeah, let me show you our 100-page document on it. 2172 02:00:08,330 --> 02:00:14,870 I mean, obviously you can't— a lot of what we had to do here was 2173 02:00:15,120 --> 02:00:16,667 rotate a lot of the rest of the company. 2174 02:00:16,750 --> 02:00:19,627 And like for things like the infrastructure team that now, 2175 02:00:19,710 --> 02:00:22,007 you know, it's like building these huge clusters to support this. 2176 02:00:22,090 --> 02:00:23,990 I mean, that's obviously a huge thing. 2177 02:00:25,130 --> 02:00:28,260 So huge parts of the company kind of had to pivot to do this. 2178 02:00:28,430 --> 02:00:32,557 And obviously you can't do too many of those at the same time, right? 2179 02:00:32,640 --> 02:00:34,840 Only one thing can be the top priority. Yes. 2180 02:00:35,190 --> 02:00:39,060 But I think on balance, it has made me somewhat more optimistic. 2181 02:00:39,150 --> 02:00:43,107 I mean, we brought in this guy Kunal Shah, who, you know, it's like, 2182 02:00:43,190 --> 02:00:45,990 I think he's like one of the top entrepreneurs in India, 2183 02:00:46,150 --> 02:00:48,987 building this great company CRED, and And now he's running WhatsApp, 2184 02:00:49,070 --> 02:00:51,570 which I always thought should do more in commerce, 2185 02:00:51,700 --> 02:00:54,787 and we were sort of underserved there, and India is the biggest 2186 02:00:54,870 --> 02:00:56,210 country that we're in. 2187 02:00:57,670 --> 02:00:59,797 But, I mean, that's— it's a bit of a different 2188 02:00:59,880 --> 02:01:02,837 situation, because we're not, like, pivoting 2189 02:01:02,920 --> 02:01:04,507 the whole company towards WhatsApp. 2190 02:01:04,590 --> 02:01:07,387 We just wanted to bring more entrepreneurial energy there. 2191 02:01:07,470 --> 02:01:09,677 This was, like, a level 10 out of 10 difficulty thing. 2192 02:01:09,760 --> 02:01:12,137 I don't want to talk about it in the past tense, as if it's done, because it's still 2193 02:01:12,220 --> 02:01:14,057 a level 10 out of 10 difficulty thing. Yeah, yeah, sure. 2194 02:01:14,140 --> 02:01:16,890 But I think 2195 02:01:18,100 --> 02:01:21,817 the number of pieces that kind of had 2196 02:01:21,900 --> 02:01:27,570 to work, it was quite all-encompassing. 2197 02:01:27,820 --> 02:01:30,697 And I think that it's both difficult for the people doing it, 2198 02:01:30,780 --> 02:01:33,700 but I do think that that also puts some level of strain on 2199 02:01:34,490 --> 02:01:37,037 the rest of the company, on the financial model of the company. 2200 02:01:37,120 --> 02:01:40,320 I think like you— so I wouldn't want to conflate those 2 pieces. 2201 02:01:40,540 --> 02:01:43,587 Right? It's like I think there's yeah, it's good to have new blood, and it's good 2202 02:01:43,670 --> 02:01:46,547 to have people who understand the context. 2203 02:01:46,630 --> 02:01:50,637 And you know, you were talking about people who've been around Meta for a while 2204 02:01:50,720 --> 02:01:52,057 and understand the ups and downs. 2205 02:01:52,140 --> 02:01:54,477 And it's like I don't know if you've worked at Meta for 10 years, 2206 02:01:54,560 --> 02:01:55,897 you've seen some shit, right? 2207 02:01:55,980 --> 02:02:00,897 It's like it's like we've gone through a lot of hard times, and and like I think 2208 02:02:00,980 --> 02:02:03,080 that that's kind of important context too. 2209 02:02:03,190 --> 02:02:07,070 But but no, I I think a lot of this is just. 2210 02:02:08,240 --> 02:02:11,907 trying to, you know, you're— whenever you're building 2211 02:02:11,990 --> 02:02:16,617 a team, you're— you inherently, you know, it's like Alex had a theory for what 2212 02:02:16,700 --> 02:02:20,667 the team shape needed to be, and like, I guess I had a theory for what 2213 02:02:20,750 --> 02:02:23,670 like the shape of the leaders needed to be for it. 2214 02:02:23,960 --> 02:02:29,180 And, um, and, you know, it's like a year plus in, it like seems okay. 2215 02:02:29,340 --> 02:02:32,217 Like, it's like, first of all, it was just good to try something different, right? 2216 02:02:32,300 --> 02:02:34,177 Mm-hmm. And then different than what everyone else was doing. 2217 02:02:34,260 --> 02:02:41,187 Yes. And yeah, I mean, that's— 2218 02:02:41,270 --> 02:02:45,777 I think there's some other dimensions here that are worth mentioning too. 2219 02:02:45,860 --> 02:02:49,150 Like, one of the— like, when deciding we had to do the full 2220 02:02:49,240 --> 02:02:54,197 reboot, like, you know, a lot of it was, you know, one of the main inputs was 2221 02:02:54,280 --> 02:02:56,747 external, but also one of the inputs— I don't think I've ever— 2222 02:02:56,830 --> 02:02:59,667 actually, I can't remember if I told you this story, but one of the first meetings 2223 02:02:59,750 --> 02:03:05,587 I had When I joined, I was in this meeting with some of the leads 2224 02:03:05,670 --> 02:03:06,820 of the existing effort. 2225 02:03:07,210 --> 02:03:10,297 And I was like, okay, I think the first thing we need to do is 2226 02:03:10,380 --> 02:03:15,140 train a really good reasoning model by some timeframe. 2227 02:03:15,260 --> 02:03:18,770 I don't know when, but this is clearly the most important milestone. 2228 02:03:19,270 --> 02:03:23,140 And one of the key leads in the group 2229 02:03:24,060 --> 02:03:29,650 who no longer works at Meta, but was like, oh, we don't need a reasoning model. 2230 02:03:29,820 --> 02:03:32,570 Do any of our consumers need a reasoning model? 2231 02:03:33,320 --> 02:03:34,697 I feel like we don't need to do a reasoning model. 2232 02:03:34,780 --> 02:03:36,117 We could just do a normal model. 2233 02:03:36,200 --> 02:03:39,370 And I had this heart-sinking moment. 2234 02:03:39,700 --> 02:03:41,790 Oh, no, I've joined the wrong company. 2235 02:03:42,370 --> 02:03:45,460 I was like, oh, I've got to take 10 steps back. 2236 02:03:46,040 --> 02:03:49,170 And I think this is kind of a funny anecdote. 2237 02:03:49,300 --> 02:03:50,837 I remember calling Nat after this meeting. 2238 02:03:50,920 --> 02:03:55,180 I was like, Nat, like, this is what they just said in the meeting. 2239 02:03:55,340 --> 02:04:00,680 But I think it was specifically an indicator of something that I think, 2240 02:04:01,100 --> 02:04:04,397 like, Mark, when he talked to people, like, certainly also started to realize, 2241 02:04:04,480 --> 02:04:08,067 which is like, we needed leadership in the company to be 2242 02:04:08,150 --> 02:04:11,607 much more AI-pilled and much more sort of like forward-leaning 2243 02:04:11,690 --> 02:04:15,527 and front-leaning on like where the— how quickly the technology's moving, 2244 02:04:15,610 --> 02:04:18,077 where it's going to go to, what the possibilities are. 2245 02:04:18,160 --> 02:04:21,580 And effectively dramatically more AI-pilled. 2246 02:04:22,080 --> 02:04:27,957 And I think as Marc started pivoting and shifting 2247 02:04:28,040 --> 02:04:32,140 the whole company towards this direction, and it was very clear that this is like, 2248 02:04:33,010 --> 02:04:36,340 I think Marc got AI-pilled, and then everyone saw that he was 2249 02:04:36,430 --> 02:04:39,847 AI-pilled and everyone realized, like, oh, this is where the company's going. 2250 02:04:39,930 --> 02:04:45,230 Then I think you had a lot of the existing leaders like very quickly 2251 02:04:45,730 --> 02:04:50,230 pick it up and change. And like Adam Mosseri, for example, who runs Instagram, 2252 02:04:50,570 --> 02:04:56,317 has one of the most like aesthetic but also like effective kind 2253 02:04:56,400 --> 02:05:01,077 of like personal vibe code and personal mini app setup of anyone that I've seen. 2254 02:05:01,160 --> 02:05:04,870 Like, I think it's like he's like really gone quite far in the direction. 2255 02:05:05,330 --> 02:05:09,167 Mm-hmm. And I think a lot of the leaders at Meta who have been at Meta a long 2256 02:05:09,250 --> 02:05:11,667 time are, like, all very adaptive. 2257 02:05:11,750 --> 02:05:15,007 Like, I think, you know, kind of to the prior point, they've been in— Yeah. 2258 02:05:15,090 --> 02:05:18,180 A highly turbulent, like, period of technology. 2259 02:05:18,390 --> 02:05:22,220 Now, AI is, like, maybe, you know, 5 or 10 times more turbulent 2260 02:05:22,310 --> 02:05:24,307 and faster moving than those past waves. 2261 02:05:24,390 --> 02:05:28,100 But I think they're, like, they have adapted to new trends many times. 2262 02:05:28,190 --> 02:05:31,647 And so I think there's, like, I've been very impressed, actually, 2263 02:05:31,730 --> 02:05:35,780 by seeing how quickly a lot of Meta leadership 2264 02:05:36,990 --> 02:05:40,287 If I compare the kind of conversations I would have with them when I first joined 2265 02:05:40,370 --> 02:05:43,077 versus the kind of conversations I have now— They're now super AI-built. Yeah. 2266 02:05:43,160 --> 02:05:47,620 And it's very impressive, honestly, for that to happen at the large scale 2267 02:05:47,870 --> 02:05:50,870 of Meta, which is one of the largest companies in the world. 2268 02:05:51,670 --> 02:05:53,297 Yeah, we got them there pretty quickly. 2269 02:05:53,380 --> 02:05:56,177 It was, yeah, but I mean, like you're saying, it's like if 2270 02:05:56,260 --> 02:06:00,180 you start coding, then it's helpful. 2271 02:06:01,760 --> 02:06:05,680 But I think a big part of what, a big part of the culture that we were 2272 02:06:05,770 --> 02:06:11,850 trying to create is also, you know, when we were talking about how you have 2273 02:06:11,940 --> 02:06:15,570 all these direct reports, the point of it isn't that 2274 02:06:16,650 --> 02:06:20,360 you have the direct reports, it's that there aren't, it's the absence 2275 02:06:20,450 --> 02:06:22,737 of non-technical managers, right? 2276 02:06:22,820 --> 02:06:25,157 So, so I don't think you would say that you're 2277 02:06:25,240 --> 02:06:28,660 like, you know, that you're providing a ton of managerial support 2278 02:06:28,750 --> 02:06:31,497 to 200 people at once. I mean, that's not a feature of the thing. 2279 02:06:31,580 --> 02:06:36,010 Alex, well known for— Career chats and emotional, how are you doing, checking. 2280 02:06:36,510 --> 02:06:39,840 But I think it's just well known among researchers that, you know, 2281 02:06:39,930 --> 02:06:40,967 there's this common pattern. 2282 02:06:41,050 --> 02:06:45,600 It's like you go from being very technical to you go to a role where you're, 2283 02:06:46,020 --> 02:06:50,310 you're not like in the details anymore, and then you very quickly become not 2284 02:06:50,440 --> 02:06:54,860 technical or don't kind of understand it much faster than you probably realize. 2285 02:06:55,020 --> 02:06:58,470 And it's quite dangerous. It's such a fast-moving domain, especially. 2286 02:06:59,110 --> 02:07:01,360 I think that, like, this whole thing of just, 2287 02:07:01,450 --> 02:07:04,327 like, you have to stay technical, people have to use the agents, 2288 02:07:04,410 --> 02:07:06,330 people should be writing code. Yeah. 2289 02:07:06,780 --> 02:07:10,537 I think that that's— that has been a really important part of this whole thing. 2290 02:07:10,620 --> 02:07:14,120 Have you had any very funny disagreements between the two of you guys? 2291 02:07:14,250 --> 02:07:15,547 We have a lot of disagreements. 2292 02:07:15,630 --> 02:07:18,177 Yeah, I mean, we're working on naming up until very recently. 2293 02:07:18,260 --> 02:07:21,717 Naming is a good bike. It's one of the things I appreciate about Alex. 2294 02:07:21,800 --> 02:07:23,967 It's like one of his— I mean, what this is like, 2295 02:07:24,050 --> 02:07:26,930 I think it goes back to your your philosophy, 2296 02:07:27,310 --> 02:07:29,937 you're talking about this for recruiting back at Scale, is like hire 2297 02:07:30,020 --> 02:07:31,397 people who really care, right? 2298 02:07:31,480 --> 02:07:34,150 I think there's some expletives in it, but it's fine. 2299 02:07:34,400 --> 02:07:36,150 But I— 2300 02:07:36,770 --> 02:07:38,980 But Alex really cares, it turns out. 2301 02:07:39,070 --> 02:07:42,777 I think that's like one of the things that I thought was, you know, I think 2302 02:07:42,860 --> 02:07:44,407 that that's like really important, right? 2303 02:07:44,490 --> 02:07:49,367 It's, so I think part of what you kind of get when you like have 2304 02:07:49,450 --> 02:07:53,917 the supernova of like all these different talented but very strong-willed people 2305 02:07:54,000 --> 02:07:57,090 together is like, you're going to have like fun fights, um, 2306 02:07:57,380 --> 02:08:00,337 and not fun fights, but like, but it's like to some degree I'd rather 2307 02:08:00,420 --> 02:08:04,630 have people push for things that they believe in and like argue over things 2308 02:08:04,760 --> 02:08:07,387 than, you know, just have people think that they're like blindly 2309 02:08:07,470 --> 02:08:08,767 executing something, you know. 2310 02:08:08,850 --> 02:08:11,147 It's like people need to feel ownership. So yeah. 2311 02:08:11,230 --> 02:08:13,580 How is Mark a different boss than you expected? 2312 02:08:14,560 --> 02:08:16,897 Um, well, I don't have a huge comparison set. 2313 02:08:16,980 --> 02:08:18,817 I think I can leave now. Yeah, exactly. 2314 02:08:18,900 --> 02:08:23,030 Yeah, no, it's No, I think that like the— 2315 02:08:23,900 --> 02:08:26,530 I think it has been very— 2316 02:08:28,950 --> 02:08:33,040 one of the very cool things about like working with Mark over the past year 2317 02:08:33,410 --> 02:08:38,290 has just been watching like the true degree of his boldness 2318 02:08:38,540 --> 02:08:41,877 and like, and this trait that we were talking about before of like, you know, 2319 02:08:41,960 --> 02:08:48,637 once he internalizes certain beliefs, like the— speed and magnitude of willing 2320 02:08:48,720 --> 02:08:55,350 to change prior actions and pivot towards the worldview that is 2321 02:08:55,440 --> 02:08:56,767 consistent with those beliefs. 2322 02:08:56,850 --> 02:09:02,070 I think these are pretty unusual traits, generally speaking, even 2323 02:09:02,780 --> 02:09:04,737 among this base of founders. 2324 02:09:04,820 --> 02:09:08,277 Mark actually ingests new information and fully updates his worldview and then 2325 02:09:08,360 --> 02:09:10,747 acts in accordance with the newly installed worldview. 2326 02:09:10,830 --> 02:09:14,000 Very Bayesian. And then willing to take— Very Bayesian. 2327 02:09:14,200 --> 02:09:17,000 Very Bayesian. I've never gotten such a nice compliment. 2328 02:09:17,540 --> 02:09:20,340 What lovely priors you have. 2329 02:09:21,000 --> 02:09:25,587 Yeah. Posterior. You're not allowed to talk about posteriors anymore after 2330 02:09:25,670 --> 02:09:27,050 the whole incident. Yeah. 2331 02:09:27,680 --> 02:09:31,560 No, but then I think also just the degree of boldness. 2332 02:09:31,760 --> 02:09:35,267 And I think that, like, the— this is— it was, like, 2333 02:09:35,350 --> 02:09:38,727 actually quite puzzling to me at first, the degree to which you were, like, 2334 02:09:38,810 --> 02:09:40,357 willing to make these massive bets. 2335 02:09:40,440 --> 02:09:43,147 It was something actually Nat and I would often, like, talk about. 2336 02:09:43,230 --> 02:09:45,860 I was like, Wow, we're really going big here. 2337 02:09:46,860 --> 02:09:49,460 And both of us would be like, would you go this big? 2338 02:09:49,660 --> 02:09:53,410 Would you go this big? And so I think that this, I think it's like, I mean, 2339 02:09:53,790 --> 02:09:58,120 that is quite a notable trait and something I really admire. 2340 02:09:58,870 --> 02:10:03,380 And I think is very, I actually, if I 2341 02:10:04,090 --> 02:10:06,420 look back at the history of Meta and 2342 02:10:07,010 --> 02:10:11,390 I imagine what the scenario would've been in all these pivotal moments, 2343 02:10:11,800 --> 02:10:15,220 I can see how like this is kind of the movie, so to speak. 2344 02:10:15,310 --> 02:10:19,017 Yeah. Or like this is kind of how a lot of those situations go where it's 2345 02:10:19,100 --> 02:10:22,520 like there's— I feel like probably 2346 02:10:23,860 --> 02:10:26,190 the way I imagine is like there clearly is 2347 02:10:26,280 --> 02:10:29,130 moments where like something very, very big is happening. 2348 02:10:30,360 --> 02:10:34,697 And it may not even really be showing in the metrics or it may not be impacting 2349 02:10:34,780 --> 02:10:37,630 user counts or impacting revenue or whatever it might be. 2350 02:10:37,750 --> 02:10:39,247 It's not really impacting things. 2351 02:10:39,330 --> 02:10:41,997 But clearly, if you believe that thing is going to happen, 2352 02:10:42,080 --> 02:10:45,330 then it has massive implications for the company and the product. 2353 02:10:47,340 --> 02:10:52,257 And I think my post hoc retelling of Meta's history is 2354 02:10:52,340 --> 02:10:56,220 that I think Meta is a company where every time that happens, Meta takes 2355 02:10:56,310 --> 02:10:58,010 it extremely, extremely seriously. 2356 02:10:58,270 --> 02:11:01,810 And that has enabled the company to innovate and continue evolving 2357 02:11:02,600 --> 02:11:05,450 with a very fast-moving and fast-changing tech landscape. 2358 02:11:06,520 --> 02:11:08,990 Mark is very Bayesian. Yeah, very Bayesian. 2359 02:11:09,190 --> 02:11:12,610 So Bayesian. Life is short, just do the right thing. 2360 02:11:13,200 --> 02:11:16,990 Yeah. And I think AI is, I mean, I think AI is maybe that, like, 2361 02:11:17,160 --> 02:11:18,207 you know, times 100 or whatnot. 2362 02:11:18,290 --> 02:11:23,540 But, like, I think the, you know, a lot of CEOs 2363 02:11:24,210 --> 02:11:27,710 would've told you last, you know, last summer, oh yeah, I think AI is 2364 02:11:27,800 --> 02:11:29,127 the most important thing for my business. 2365 02:11:29,210 --> 02:11:32,757 Like, I think maybe every CEO would say that, But then you look at, like, 2366 02:11:32,840 --> 02:11:34,887 what they're actually doing and how consistent are their 2367 02:11:34,970 --> 02:11:36,220 actions with that belief. 2368 02:11:36,470 --> 02:11:41,980 And I think this is, like, a, like, quite, quite special thing about Meta and Mark. 2369 02:11:42,980 --> 02:11:44,687 Last question, because we're past all of our bedtimes. 2370 02:11:44,770 --> 02:11:46,060 I'm so flattered. Yeah. 2371 02:11:47,150 --> 02:11:50,250 What do you make of all of Alex's deranged posting on Twitter? 2372 02:11:50,900 --> 02:11:56,570 I mean, I think most of it is great, but, but I, I guess my main view— 2373 02:11:56,780 --> 02:12:00,540 But it doesn't matter how How much you fail, it matters how win— 2374 02:12:00,660 --> 02:12:01,827 how big you win when you win. 2375 02:12:01,910 --> 02:12:06,457 I think with— so I think with communication, I think it's 2376 02:12:06,540 --> 02:12:08,340 most important to just be authentic. 2377 02:12:08,710 --> 02:12:11,717 And I think when you're trying to be authentic, some of the things 2378 02:12:11,800 --> 02:12:13,347 don't come out the way you want. 2379 02:12:13,430 --> 02:12:18,100 And I think that that is a— that it is better to make the trade in that direction 2380 02:12:18,560 --> 02:12:22,850 than it is to be overly scripted and not be authentic. 2381 02:12:22,940 --> 02:12:25,390 But I mean, most of it, I just feel is hilarious. 2382 02:12:25,560 --> 02:12:29,520 So, yeah, what do you think about my deranged tweets? 2383 02:12:30,150 --> 02:12:32,900 I'm not a heavy X user, but— 2384 02:12:33,570 --> 02:12:36,700 But you are back on X now. I'm back to posting. 2385 02:12:36,820 --> 02:12:39,287 I mean, it's like a bunch of the researchers are there and we want 2386 02:12:39,370 --> 02:12:42,820 to be over there, but everything I post there, I also post on Threads 2387 02:12:43,580 --> 02:12:45,370 and it's all good. 2388 02:12:45,460 --> 02:12:50,380 But every time I load X, I have like 4 Alex posts and it just like cracks me up. 2389 02:12:50,500 --> 02:12:52,377 It's like some of these, Like, it's like, where do you— we're 2390 02:12:52,460 --> 02:12:53,547 just talking about this. 2391 02:12:53,630 --> 02:12:55,967 Well, you're now a clearinghouse, I presume, where people send you good stuff. 2392 02:12:56,050 --> 02:12:57,887 I was like, where do you come up with this stuff? 2393 02:12:57,970 --> 02:13:00,850 It's like, yeah, this is—yeah. How does it work? 2394 02:13:01,600 --> 02:13:04,727 Um, most of the— like, uh, we were, we were talking about this with my partner 2395 02:13:04,810 --> 02:13:09,357 yesterday, and he was, you know, I think one observation was just, um, 2396 02:13:09,440 --> 02:13:13,987 there was actually probably still to this moment, I— my, my, uh, thought pattern 2397 02:13:14,070 --> 02:13:15,777 oftentimes is, oh, that's funny. 2398 02:13:15,860 --> 02:13:17,907 I wonder what Jolly would think about that. 2399 02:13:17,990 --> 02:13:21,030 And then, you know, you get good memes out of that. 2400 02:13:21,200 --> 02:13:25,870 That being said, I have experienced the sort of like brutal 2401 02:13:26,410 --> 02:13:30,170 metabolism of the internet and how quickly it like metabolizes memes. 2402 02:13:30,540 --> 02:13:34,630 And so the sort of like, you know, you really have to stay 2403 02:13:35,840 --> 02:13:40,390 on the cutting edge of memeing to— It's easy to fall behind. Yeah. 2404 02:13:40,640 --> 02:13:43,190 Which is maybe a good summary of this conversation. 2405 02:13:44,310 --> 02:13:47,057 It's easy to fall behind, you know, you gotta stay at the frontier. 2406 02:13:47,140 --> 02:13:50,600 Stay in the details. Yeah, stay technical and stay at the frontier. 2407 02:13:50,690 --> 02:13:52,937 Yeah. Yeah. Alrighty, we'll leave it there. All right. 2408 02:13:53,020 --> 02:13:57,450 Awesome. Thanks, guys. Thank you. Okay, wait, so what is everyone drinking? 2409 02:13:57,610 --> 02:14:03,410 Well, that. What is this? That's a keg of Guinness that I brought you. 2410 02:14:03,950 --> 02:14:05,787 This is Guinness from? Dublin. 2411 02:14:05,870 --> 02:14:08,707 Oh, wow. You were just there? Yeah, I was just there this week. Oh, awesome. 2412 02:14:08,790 --> 02:14:11,507 Yeah. Well, thank you. We got the local stuff. Yeah, there you go. 2413 02:14:11,590 --> 02:14:15,217 Yeah, no, I figured this would be a good occasion to Get the Guinness going. 2414 02:14:15,300 --> 02:14:16,133 Yeah.