WEBVTT 00:00:00.080 --> 00:00:03.517 I think we don't yet know the shape and form of the ultimate products. Just one 00:00:03.600 --> 00:00:07.117 just obvious historical analogy is you know the personal computer from sort of 00:00:07.200 --> 00:00:09.597 invention in 1975 through to you know 00:00:09.680 --> 00:00:14.797 basically 1992 was a text prompt system 17 years in you know the whole industry 00:00:14.880 --> 00:00:17.597 took a left turn into gooies and never looked back and then by the way you know 00:00:17.680 --> 00:00:20.477 5 years after that the industry took a left turn into web browsers and never 00:00:20.560 --> 00:00:22.957 looked back right and you know look I'm I'm sure there will be chat bots 20 00:00:23.040 --> 00:00:26.477 years from now but I'm pretty confident that both the current chatbot companies 00:00:26.560 --> 00:00:29.357 and many new companies are going to figure out many kinds of user 00:00:29.440 --> 00:00:33.680 experiences that are radically different that we don't we don't even know 00:00:34.160 --> 00:00:36.557 Please join me in welcoming Mark Andre 00:00:36.640 --> 00:00:38.477 and Ben Horowitz with general partner 00:00:38.560 --> 00:00:43.597 Eric Torberg. Follow me into our solo. Get in the flow 00:00:43.680 --> 00:00:45.437 and you can pitch it like a photo. Music 00:00:45.520 --> 00:00:47.517 makes me maintain and make melodies for 00:00:47.600 --> 00:00:52.797 MC's motivation breaks. I'm never >> Thank you for the rock Kim. Who did 00:00:52.880 --> 00:00:57.117 that? >> Ben picks the Ben picked the music. 00:00:57.200 --> 00:00:59.277 Mark, there's been a lot of talk lately 00:00:59.360 --> 00:01:02.317 about the limitations of LLMs that they 00:01:02.400 --> 00:01:04.957 they can't do true invention of say new 00:01:05.040 --> 00:01:09.437 science that they can't do true creative genius that it's just combining or 00:01:09.520 --> 00:01:13.117 packaging. You have thoughts here. What say you? 00:01:13.200 --> 00:01:15.917 >> Yeah. So, so for me, yes, you get all these questions and yeah, they usually 00:01:16.000 --> 00:01:20.557 come in either, you know, sort of are are language models intelligent in the 00:01:20.640 --> 00:01:23.117 sense of like can they actually um you know, can they actually process 00:01:23.200 --> 00:01:26.877 information and have sort of conceptual breakthroughs? the way that people can. 00:01:26.960 --> 00:01:30.077 And then there's are language models or or video models creative, you know, can 00:01:30.160 --> 00:01:33.037 they can they create new art? Um actually have genuine creative 00:01:33.120 --> 00:01:35.517 breakthroughs. And of course my my my answer to both of those is well can 00:01:35.600 --> 00:01:39.677 people do those things? Um and um I think there's two two two questions 00:01:39.760 --> 00:01:43.597 there which is like okay even if some people are quote unquote intelligent as 00:01:43.680 --> 00:01:46.637 in having original uh conceptual breakthroughs and not just let's just 00:01:46.720 --> 00:01:51.437 say regurgitating the training set um uh or following scripts um uh how you know 00:01:51.520 --> 00:01:55.117 what percentage of people can actually do that and I say I've only met a few 00:01:55.200 --> 00:01:59.197 some of them are here in the room um but uh you know not that many most people 00:01:59.280 --> 00:02:01.677 never do and then creativity I mean how many people are actually genuinely 00:02:01.760 --> 00:02:04.957 creative right and so you you kind of point to a Beethoven or you know Van 00:02:05.040 --> 00:02:07.357 Gogh or something like that you're like Okay, that's creativity and yeah, that's 00:02:07.440 --> 00:02:10.637 creativity and then how many Beethovenas and Van Go are there? Obviously, not 00:02:10.720 --> 00:02:14.557 very many. So, so one is just like okay, like you know, if it's if it if if if 00:02:14.640 --> 00:02:18.637 these things clear the bar of, you know, 99.99% of humanity, you know, then 00:02:18.720 --> 00:02:22.317 that's pretty interesting just in and of itself. But then I you dig into it 00:02:22.400 --> 00:02:26.237 further and you're like, okay, like how many actual real conceptual 00:02:26.320 --> 00:02:29.437 breakthroughs have there ever been actually ever in human history as 00:02:29.520 --> 00:02:34.317 compared to sort of remixing remixing ideas. Um and you know like if you look 00:02:34.400 --> 00:02:37.197 at the history of technology it's almost always the case that the big 00:02:37.280 --> 00:02:41.357 breakthroughs are the result of you know usually at least 40 years of sort of 00:02:41.440 --> 00:02:44.797 work ahead of time you know four decades right in fact language models themselves 00:02:44.880 --> 00:02:48.557 are the culmination of eight decades right of previous work and so there's 00:02:48.640 --> 00:02:51.357 remixing and then in the arts it's the exact same thing you know novels and 00:02:51.440 --> 00:02:54.557 music and everything like you know there are clearly creative leaps but you know 00:02:54.640 --> 00:02:57.597 there's just tremendous amounts of influence that came in from from people 00:02:57.680 --> 00:03:00.877 who came before and even if you think about like somebody with the creativity 00:03:00.960 --> 00:03:03.997 of a Beethoven like there There's a lot of Beethoven in Mozart and Heiden and in 00:03:04.080 --> 00:03:07.677 the composers that came before and so there's just tremendous amounts of of of 00:03:07.760 --> 00:03:11.357 remixing and combination. And so it's it's it's a little bit of an angel's 00:03:11.440 --> 00:03:14.317 dancing on the head of a pin question which is like if you can get if you can 00:03:14.400 --> 00:03:19.197 get you know within you know I don't know one you know 001% of of kind of 00:03:19.280 --> 00:03:22.477 worldbeating uh you know generational creativity intelligence like you're 00:03:22.560 --> 00:03:26.397 you're you're probably all the way there. So I so emotionally I want to 00:03:26.480 --> 00:03:30.077 like hold out hope that there is you know still something special about human 00:03:30.160 --> 00:03:33.197 creativity and I I certainly believe that and I and I and I very much want to 00:03:33.280 --> 00:03:37.037 believe that but um I don't know when I use these things I'm like wow they seem 00:03:37.120 --> 00:03:40.077 to be awfully smart and awfully creative. So I'm I'm I'm I'm pretty 00:03:40.160 --> 00:03:42.637 convinced that they're going to clear the bar. 00:03:42.720 --> 00:03:45.597 >> Yeah. the I think that seems to be a common 00:03:45.680 --> 00:03:48.957 theme in your analysis when when people talk about the limitations of LMS, you 00:03:49.040 --> 00:03:52.317 know, can they do transfer learning or or just learning in general? You seem to 00:03:52.400 --> 00:03:55.517 ask, can people do this? >> Yes. Can people do these things? Well, 00:03:55.600 --> 00:03:58.237 it's like lateral thinking, right? So, yeah, so it's like reasoning in or out 00:03:58.320 --> 00:04:00.477 of distribution, right? And so it's like, okay, I know a lot of people who 00:04:00.560 --> 00:04:02.797 are very good at reasoning inside distribution. How many people do I 00:04:02.880 --> 00:04:05.517 actually know who are good at reasoning outside of distribution and doing 00:04:05.600 --> 00:04:08.557 transfer learning? And and the answer is like I know a handful. Like I I know a 00:04:08.640 --> 00:04:12.077 few I know a few people um where whenever you ask them a question, you 00:04:12.160 --> 00:04:15.517 get an extremely original answer. And usually that answer involves bringing in 00:04:15.600 --> 00:04:19.197 some idea from some adjacent space and basically being able to bridge domains. 00:04:19.280 --> 00:04:21.837 Um and so you know you'll ask them a question about I don't know you know 00:04:21.920 --> 00:04:24.637 finance and they'll they'll bring you an answer from psychology or you ask them a 00:04:24.720 --> 00:04:27.277 question about psychology and they'll bring you an answer from biology right 00:04:27.360 --> 00:04:31.437 or whatever it is. And so I I know you know I know I don't know sitting here 00:04:31.520 --> 00:04:35.197 today probably three I probably know three people who can do that reliably 00:04:35.280 --> 00:04:39.277 out of the you know I I've got you know I've got 10,000 in my address book. Um 00:04:39.360 --> 00:04:42.797 and so three out of 10,000 >> Yeah. is not is not that high a 00:04:42.880 --> 00:04:46.557 percentage. By by the way, I find this very encouraging. Like I I yeah, 00:04:46.640 --> 00:04:50.797 immediately the mood in the room has gone completely to hell. Um I find this 00:04:50.880 --> 00:04:53.437 very encouraging. I find this very encouraging because look at what 00:04:53.520 --> 00:04:55.437 humanity has been able to build, right? 00:04:55.520 --> 00:04:57.437 Despite all of our limitations, right? 00:04:57.520 --> 00:04:59.997 And and look at all the creativity that we've been able to exhibit and all the 00:05:00.080 --> 00:05:02.877 amazing art and all the amazing movies and all the amazing novels and all the 00:05:02.960 --> 00:05:05.917 amazing technical inventions and scientific breakthroughs. And so we, you 00:05:06.000 --> 00:05:08.157 know, we've been able to do, you know, everything we've been able to do with 00:05:08.240 --> 00:05:12.317 the limitations that we have. Um, and so, you know, I think that, you know, 00:05:12.400 --> 00:05:15.597 like, you know, do you need to get to the thing where you are 100% positive 00:05:15.680 --> 00:05:19.037 that's actually doing, you know, original thinking? I don't think so. Uh, 00:05:19.120 --> 00:05:21.437 I I think it'd be great if you did, and I think ultimately we'll probably 00:05:21.520 --> 00:05:22.957 conclude that that's what's happening. 00:05:23.040 --> 00:05:26.717 Um, but like I it's not necessary for like just tremendous amounts of 00:05:26.800 --> 00:05:30.237 improvement. >> Ben, we were just celebrating some some 00:05:30.320 --> 00:05:34.717 hip-hop legends at your paid in full uh event last week and so you think a lot 00:05:34.800 --> 00:05:38.157 about creative genius. How do you think about this question? 00:05:38.240 --> 00:05:41.437 Yeah, I mean I think that uh I agree 00:05:41.520 --> 00:05:43.997 with Mark that it's 00:05:44.080 --> 00:05:46.397 whatever it is, it's very useful. um 00:05:46.480 --> 00:05:48.797 even if it isn't all the way that level 00:05:48.880 --> 00:05:51.757 uh I think that 00:05:51.840 --> 00:05:54.637 you know there's something about the 00:05:54.720 --> 00:05:57.277 actual like real time human experience 00:05:57.360 --> 00:06:00.957 that humans are very into um at least in 00:06:01.040 --> 00:06:03.997 art where 00:06:04.080 --> 00:06:06.077 you know with the current state of the 00:06:06.160 --> 00:06:08.717 technology kind of the the pre-training 00:06:08.800 --> 00:06:11.757 doesn't have quite the the right data to 00:06:11.840 --> 00:06:15.837 to to get to um what you really want to 00:06:15.920 --> 00:06:20.397 Uh but it, you know, it's pretty good. 00:06:20.480 --> 00:06:23.997 >> It is pretty good. >> How many So how many true conceptual So 00:06:24.080 --> 00:06:27.197 Ben Ben Ben Ben's Ben, one of Ben's nonprofit activities is something called 00:06:27.280 --> 00:06:29.997 the Pain and Fall Foundation, which is honoring and actually providing 00:06:30.080 --> 00:06:33.197 essentially a pension a pension for uh you know, for sort of, you know, the 00:06:33.280 --> 00:06:37.917 great innovators in in in rap and hip-hop. Um and and so he has he knows 00:06:38.000 --> 00:06:40.797 and has many of we were just at the event and he you know has many of the 00:06:40.880 --> 00:06:43.357 kind of leading lights of that field for the last 50 years you know perform and 00:06:43.440 --> 00:06:47.117 and it's really fun to meet them and talk to them. Um but like how many 00:06:47.200 --> 00:06:49.917 people in that entire field over the course of the last 50 years would you 00:06:50.000 --> 00:06:53.357 classify as like a true conceptual innovator? 00:06:53.440 --> 00:06:56.397 >> Yeah. Well, you know it's interesting. 00:06:56.480 --> 00:07:00.957 Well, it depends how broadly you define it but 00:07:01.040 --> 00:07:05.917 >> you know there were several of them there last you know on Saturday. Rock, I 00:07:06.000 --> 00:07:08.957 think. Yeah, Rockm, you'd certainly put in that category. Dr. Dre, you'd 00:07:09.040 --> 00:07:12.237 certainly put in that category. George Clinton, you'd certainly put in that 00:07:12.320 --> 00:07:14.877 category. Um, 00:07:14.960 --> 00:07:17.917 >> you know, in a narrower sense, like Cool 00:07:18.000 --> 00:07:22.077 G Rap certainly had a new idea. Um, 00:07:22.160 --> 00:07:26.317 but you know, it depends like a 00:07:26.400 --> 00:07:28.317 fundamental kind of musical 00:07:28.400 --> 00:07:33.837 breakthrough, you probably just say like Rock Kim and George Clinton. Um, are 00:07:33.920 --> 00:07:38.877 they excited? >> So, so two out of >> Well, I mean those of the guys who were 00:07:38.960 --> 00:07:41.997 there. >> Oh, yeah. Yeah. >> Yeah. But yeah, it's a tiny percentage. 00:07:42.080 --> 00:07:45.997 Tiny tiny tiny tiny tiny. >> We had Jared at the fireside last night 00:07:46.080 --> 00:07:49.757 with Jared Leto. He was talking about how many people in Hollywood are are 00:07:49.840 --> 00:07:54.077 really scared or against this um what's happening here is is what do you see in 00:07:54.160 --> 00:07:57.757 you know when you talk to the Dr. JS the nas the Kanye's are they excited? Are 00:07:57.840 --> 00:07:59.277 they using it? Are they 00:07:59.360 --> 00:08:02.797 >> Yeah. No, I the so everybody uh who I 00:08:02.880 --> 00:08:07.117 speak to there are definitely people who are scared in music but like there are a 00:08:07.200 --> 00:08:09.677 lot of people who are very very interested in it and particularly the 00:08:09.760 --> 00:08:13.357 hip-hop guys are interested because um 00:08:13.440 --> 00:08:15.917 it's almost like a replay of what they 00:08:16.000 --> 00:08:18.557 did right that they just took other 00:08:18.640 --> 00:08:20.797 music and they kind of built new music 00:08:20.880 --> 00:08:23.997 out of it and I think that you know AI 00:08:24.080 --> 00:08:26.957 is uh a fantastic creative tool for 00:08:27.040 --> 00:08:29.037 them. like way opens up the pallet and 00:08:29.120 --> 00:08:32.477 then for you know a lot of what hip-hop 00:08:32.560 --> 00:08:35.437 is is it's um kind of telling a very 00:08:35.520 --> 00:08:38.797 specific story of a specific time and 00:08:38.880 --> 00:08:42.477 place which um having intimate knowledge 00:08:42.560 --> 00:08:44.877 and being trained just on that thing is 00:08:44.960 --> 00:08:46.637 is actually an advantage as opposed 00:08:46.720 --> 00:08:49.517 being like a generally smart uh music 00:08:49.600 --> 00:08:54.397 model. Um, people also use the same logic of, hey, whatever is more 00:08:54.480 --> 00:08:58.157 intelligent will rule whatever is less intelligent. And and Mark, you recently 00:08:58.240 --> 00:09:02.077 uh not not said by anybody who owns a cat. 00:09:02.160 --> 00:09:06.077 >> Yeah, exactly. Mark, you recently tweeted, "A supreme 00:09:06.160 --> 00:09:08.477 shape rotator can only rotate shapes, 00:09:08.560 --> 00:09:10.557 but a supreme word cell can rotate shape 00:09:10.640 --> 00:09:13.197 rotators." And and also, 00:09:13.280 --> 00:09:19.117 >> someone's clapping here. And also, high IQ experts work for midiq generalists. 00:09:19.200 --> 00:09:22.637 What means? >> Yeah. What what means? Uh yeah. So it's 00:09:22.720 --> 00:09:24.797 the PhDs all work for MBAs, right? So 00:09:24.880 --> 00:09:30.397 it's like, you know, okay. So yeah, like so yeah. Well, I just, you know, just 00:09:30.480 --> 00:09:32.797 take it up a level. It's just like when you look at the world today, do you 00:09:32.880 --> 00:09:37.917 think we're being ruled by the smart ones, right? Like is that is that your big 00:09:38.000 --> 00:09:41.277 conclusion from like current events, current affairs, 00:09:41.360 --> 00:09:43.197 right? Like okay, we put the geniuses in charge. Like 00:09:43.280 --> 00:09:45.197 >> you mean Kamla and Trump aren't the best? 00:09:45.280 --> 00:09:48.317 >> Well, now let's not even be specific towards the US. let's just look all over 00:09:48.400 --> 00:09:51.837 the world. >> Um, you know, yeah. And so like it's 00:09:51.920 --> 00:09:56.317 just like there's this thing. So I think two two 00:09:56.400 --> 00:09:59.437 things are true. One is we we probably all kind of underwrite the importance of 00:09:59.520 --> 00:10:03.117 intelligence. Um, and actually there's a whole kind of backstory here of like 00:10:03.200 --> 00:10:06.397 intelligence actually turns out to be this like incredibly inflammatory, you 00:10:06.480 --> 00:10:10.157 know, kind of topic for lots of reasons over the last hundred years. um uh which 00:10:10.240 --> 00:10:13.757 which we could we could talk about in great detail but like it it and you know 00:10:13.840 --> 00:10:16.557 it's and the even the just the very idea that like some people are smarter than 00:10:16.640 --> 00:10:19.117 other people you just like really freaks people out and people people don't like 00:10:19.200 --> 00:10:22.797 to talk about it we really struggle with that as a society and so like and then 00:10:22.880 --> 00:10:26.077 it is true that intelligence is like in humans intelligence is correlated to 00:10:26.160 --> 00:10:30.237 almost every kind of positive life outcome right and so u intelligence 00:10:30.320 --> 00:10:32.957 generally in the social sciences what they'll tell you is what they call fluid 00:10:33.040 --> 00:10:35.517 intelligence or or the G factor or IQ is 00:10:35.600 --> 00:10:40.237 sort of it's sort of 0.4 four correlated to basically everything. Um, and so it's 00:10:40.320 --> 00:10:44.317 0.4 correlation to like educational outcomes and like you know professional 00:10:44.400 --> 00:10:48.717 outcomes and and you know income and by the way also like life satisfaction and 00:10:48.800 --> 00:10:51.837 by the way non-violence you know being able to solve problems without physical 00:10:51.920 --> 00:10:54.637 violence and so forth. And so like on the one hand like we probably all 00:10:54.720 --> 00:10:58.237 underrate intelligence. Um on the other hand the people who are in the fields 00:10:58.320 --> 00:11:01.917 that involve intelligence probably overrate intelligence. Um, and you might 00:11:02.000 --> 00:11:05.117 even you might even coin a term like maybe like intelligence supremacist or 00:11:05.200 --> 00:11:08.477 something like that where it's just like oh like intelligence is very important 00:11:08.560 --> 00:11:12.077 and so therefore maybe it's like the most important thing or the only thing 00:11:12.160 --> 00:11:14.797 but but then you look at reality and you're like okay that's clearly not the 00:11:14.880 --> 00:11:17.197 case. >> Yeah. It's still zero only 0.4 right. 00:11:17.280 --> 00:11:19.277 Yeah. >> Well so to start with it's only 0.4 and 00:11:19.360 --> 00:11:23.117 you know in the social sciences 0.4 is a giant correlation factor right like most 00:11:23.200 --> 00:11:26.317 most things that where you can correlate whether it's you know genes or observed 00:11:26.400 --> 00:11:28.877 behavior whatever to anything in the social sciences the correlations are 00:11:28.960 --> 00:11:32.797 much smaller than that. So 0.4 is is tiny, but it's still only 0.4. So, even 00:11:32.880 --> 00:11:35.997 if you're like a fullon if you even if you're like a full-on genetic 00:11:36.080 --> 00:11:39.197 determinist and you're just like, you know, genetic IQ just like drives all 00:11:39.280 --> 00:11:43.997 these outcomes, like it still doesn't explain, you know, 6 uh of the 00:11:44.080 --> 00:11:46.957 correlation and so that leaves it but but that's just on the individual level. 00:11:47.040 --> 00:11:49.517 Then you just look at the collective level. Well, you just look at the 00:11:49.600 --> 00:11:52.797 collective level and it's like a famous famous observation is you take a b you 00:11:52.880 --> 00:11:55.677 take a bunch of you take any group of people, you put them in a mob and the 00:11:55.760 --> 00:11:59.437 mob is dumber, right, than the average and and and you put a bunch of smart 00:11:59.520 --> 00:12:02.797 people in a mob and they definitely turn dumber like and you see that all the 00:12:02.880 --> 00:12:07.597 time, right? Um uh and so you put people in groups and they they they behave very 00:12:07.680 --> 00:12:11.357 differently and then you you create and then you create these you know questions 00:12:11.440 --> 00:12:13.917 around like who's in charge whether who's in charge at a at a company or 00:12:14.000 --> 00:12:17.277 who's in charge of a of a country and like it it's whatever the filtration 00:12:17.360 --> 00:12:21.277 process it's clearly not it's not it's it's not it's certainly not only on IQ 00:12:21.360 --> 00:12:24.957 and it may not even be primarily on IQ and so so therefore it's just like this 00:12:25.040 --> 00:12:27.757 assumption that you kind of hear in some of the AI circles uh which is like 00:12:27.840 --> 00:12:31.197 inevitably the smart you know kind of thing is going to govern the dumb thing 00:12:31.280 --> 00:12:34.397 like I I just think that's like very easily uh it's just sort of very easily 00:12:34.480 --> 00:12:37.677 and obviously falsified like intelligence isn't sufficient and then 00:12:37.760 --> 00:12:39.277 you just you just you just convey it. 00:12:39.360 --> 00:12:41.677 You know, we're all in this room lucky enough to know a lot of smart people and 00:12:41.760 --> 00:12:44.637 you you just kind of observe smart people and like some smart people, you 00:12:44.720 --> 00:12:46.877 know, really figure out how to have their stuff together and become very 00:12:46.960 --> 00:12:51.117 successful and a lot of smart people never do. Um and so there there's there 00:12:51.200 --> 00:12:54.877 there must be there obviously are and there and there in fact must be many 00:12:54.960 --> 00:12:57.997 other factors that have to do with success um and have to do with like 00:12:58.080 --> 00:13:01.677 who's in charge than just raw intelligence. It it begs the the 00:13:01.760 --> 00:13:05.277 follow-up question of what are what are some examples of what that might be you 00:13:05.360 --> 00:13:07.837 know skills sort of outside of intelligence and more particularly 00:13:07.920 --> 00:13:11.037 specifically why couldn't AI systems you know learn them? 00:13:11.120 --> 00:13:14.237 >> Yeah. So Ben like what what what other than intelligence what what in your 00:13:14.320 --> 00:13:16.717 experience determines for example success in leadership or in 00:13:16.800 --> 00:13:21.677 entrepreneurship or in in solving complex problems or organizing people? 00:13:21.760 --> 00:13:25.357 >> Yeah. There there are many things um you know 00:13:25.440 --> 00:13:30.557 like a lot of it is uh being able to have a confrontation in the correct way 00:13:30.640 --> 00:13:34.557 and like there's some intelligence in that but a lot of it is just under 00:13:34.640 --> 00:13:37.517 really understanding who you're talking to 00:13:37.600 --> 00:13:40.797 you know being able to interpret 00:13:40.880 --> 00:13:45.037 everything about how they're thinking about it and just kind of generally 00:13:45.120 --> 00:13:47.277 seeing decisions through the eyes of the 00:13:47.360 --> 00:13:49.117 people working in the company not 00:13:49.200 --> 00:13:54.397 through your eyes is the skill that you you know you develop by 00:13:54.480 --> 00:13:56.957 talking to people all the time, understanding what they're saying, so 00:13:57.040 --> 00:14:01.517 forth, these kinds of things. And it's just um 00:14:01.600 --> 00:14:05.517 you know, it's certainly not an IQ thing. And not that like I I could 00:14:05.600 --> 00:14:08.477 imagine an AI training on any individual 00:14:08.560 --> 00:14:11.197 and like figuring it all out and knowing 00:14:11.280 --> 00:14:14.237 what to say and so forth. Um but then 00:14:14.320 --> 00:14:17.437 you also need that integrated with uh 00:14:17.520 --> 00:14:21.277 you know like whatever the business ought to be doing. So you're not you're 00:14:21.360 --> 00:14:26.477 not trying to do what's popular. You're trying to get people to do what's 00:14:26.560 --> 00:14:30.797 correct even if they don't like it. And you know that's a lot of management. So 00:14:30.880 --> 00:14:34.877 uh it's not a problem anybody's working on 00:14:34.960 --> 00:14:36.317 currently but maybe they will. 00:14:36.400 --> 00:14:37.997 >> It's some right some combination of like 00:14:38.080 --> 00:14:40.957 courage some combination of motivation 00:14:41.040 --> 00:14:45.517 some combination of um of emotional uh understanding theory of mind. 00:14:45.600 --> 00:14:47.517 >> Yeah. what you know what do people want 00:14:47.600 --> 00:14:50.957 like you know married to you know what 00:14:51.040 --> 00:14:56.077 needs to be done and then like how talented are they like which ones can 00:14:56.160 --> 00:14:59.357 you afford like if they jump out the window it's fine you know which one's 00:14:59.440 --> 00:15:01.357 not fine you know this kind of thing 00:15:01.440 --> 00:15:06.477 it's a there's a lot of like weird subtleties to it uh and it's very 00:15:06.560 --> 00:15:08.157 situational I think the hardest thing 00:15:08.240 --> 00:15:10.637 about it uh and why management books are 00:15:10.720 --> 00:15:13.997 so bad is because it's situational um 00:15:14.080 --> 00:15:15.677 you know like your company your product, 00:15:15.760 --> 00:15:18.557 your people, your org chart is very very 00:15:18.640 --> 00:15:23.117 different than you know, yeah, here are the five steps to building a strategy. 00:15:23.200 --> 00:15:27.117 It's like, well, that's the most useless thing I ever read because it has 00:15:27.200 --> 00:15:30.237 nothing to do with you. >> So, one of the interesting things on 00:15:30.320 --> 00:15:33.277 this like on this is right the the concept of theory of mind is really 00:15:33.360 --> 00:15:36.157 important, right? So, theory of mind is can you in your head model what's 00:15:36.240 --> 00:15:38.557 happening in the other person's head, right? And and and you would think that 00:15:38.640 --> 00:15:41.037 maybe that you know maybe obviously people who are smarter should be better 00:15:41.120 --> 00:15:44.477 at that. It turns out that that may not be true and I'll the reason to believe 00:15:44.560 --> 00:15:48.397 that that's not true which which is as follows. So the so the the US military 00:15:48.480 --> 00:15:50.957 is was the early adopter and has continued to be sort of the leading 00:15:51.040 --> 00:15:55.597 adopter in in US society of of of actually IQ testing and they uh they 00:15:55.680 --> 00:15:58.237 they they basically launder it through something called the ASVAB which is 00:15:58.320 --> 00:16:01.597 their they call vocational aptitude battery test but it's basically an I 00:16:01.680 --> 00:16:05.917 it's essentially an IQ test. Um and uh so they they they they still use 00:16:06.000 --> 00:16:09.117 basically explicit IQ tests and they they slot people uh into different 00:16:09.200 --> 00:16:11.197 specialties and roles uh you know in 00:16:11.280 --> 00:16:16.157 part with according IQ um including into leadership roles. Um and and so they 00:16:16.240 --> 00:16:18.637 they know what everybody's IQ is and they they kind of organize organize 00:16:18.720 --> 00:16:21.677 around that. And one of the things that they found over the years is if the 00:16:21.760 --> 00:16:24.797 leader is more than one standard deviation of IQ away from the followers, 00:16:24.880 --> 00:16:28.957 it's a real problem. Um and and and that's true in both directions, right? 00:16:29.040 --> 00:16:33.997 Um if if the leader is not smart enough to be able to right manage the you know 00:16:34.080 --> 00:16:37.917 is to be able to mo you know for somebody who is less smart to model the 00:16:38.000 --> 00:16:41.117 mental behavior of somebody who's more smart is inher of course inherently very 00:16:41.200 --> 00:16:44.477 challenging and maybe impossible. But it turns out the reverse is also true which 00:16:44.560 --> 00:16:47.677 is if the leader is two standard deviations above the norm of the 00:16:47.760 --> 00:16:51.677 organization that he's running. He also loses theory of mind. Right? it it's 00:16:51.760 --> 00:16:56.317 it's actually very hard for very smart people to um to to model uh the internal 00:16:56.400 --> 00:16:59.997 thought processes of even moderately smart people. Um and so there there's 00:17:00.080 --> 00:17:03.597 actually there's actually a real there's actually a real need to have a level of 00:17:03.680 --> 00:17:07.597 connection there that's not just right and therefore by inference if you had a 00:17:07.680 --> 00:17:10.717 person or a or a machine that had you know a thousand IQ or something like it 00:17:10.800 --> 00:17:14.397 it may just be it would be so alien its understanding of reality would be so 00:17:14.480 --> 00:17:17.997 alien to the people or the things that it was managing that it wouldn't it 00:17:18.080 --> 00:17:20.717 wouldn't even be able to connect in any sort any sort of realistic way. So again 00:17:20.800 --> 00:17:23.677 this is a very good argument that like it yeah this is the world is going to be 00:17:23.760 --> 00:17:27.677 far from organized by IQ for yeah for centuries to come. 00:17:27.760 --> 00:17:29.517 >> Yeah and Zuckerberg had a great line 00:17:29.600 --> 00:17:32.397 which is intelligence is not life and 00:17:32.480 --> 00:17:36.717 life life has a lot of dimensionality to it that is independent of intelligence. 00:17:36.800 --> 00:17:40.477 I think that you know if you spend all your time 00:17:40.560 --> 00:17:45.357 working on intelligence you lose track of that. We sometimes say about uh some 00:17:45.440 --> 00:17:47.197 specific people that they're too smart 00:17:47.280 --> 00:17:52.397 to properly model or or you know um too they s sort of assume too much 00:17:52.480 --> 00:17:55.917 rationality on other people or they just o overthink things or overrationalize 00:17:56.000 --> 00:17:59.517 them. Um yeah just to your point that it's it's on everything. 00:17:59.600 --> 00:18:03.997 >> Yeah. Yeah. Yeah. People often people seldom do what's in their best interest 00:18:04.080 --> 00:18:06.557 I should say. >> You know I also suspect this kind of 00:18:06.640 --> 00:18:09.357 gets more into the biology side of things. I you know there's more and more 00:18:09.440 --> 00:18:14.477 scientific evidence that basically also that like human human cognition human 00:18:14.560 --> 00:18:17.117 cognition or human I don't whatever you want to call it self-awareness 00:18:17.200 --> 00:18:19.357 information processing decision-m sort 00:18:19.440 --> 00:18:24.477 of experience um uh is is not purely a brain like the basically the d the sort 00:18:24.560 --> 00:18:27.997 of mind famous mindbody dualism is just not correct like and again this is an 00:18:28.080 --> 00:18:31.917 argument against sort of IQ supremacism or intelligent supremacism is it's not 00:18:32.000 --> 00:18:34.957 you know we we human beings didn't experience existence just through the 00:18:35.040 --> 00:18:38.077 rational thought u and and and specifically not through just the 00:18:38.160 --> 00:18:40.877 rational thought of the brain, but rather it's a whole body experience, 00:18:40.960 --> 00:18:43.277 right? And there's there's there's aspects of our nervous system and 00:18:43.360 --> 00:18:46.557 there's aspects of everything from our gut biome to, you know, to to you know, 00:18:46.640 --> 00:18:50.797 to to smells, you know, to alactory senses and and um you know, and hormones 00:18:50.880 --> 00:18:55.837 and like all all kinds of like biochemical kind of aspects to life. Um 00:18:55.920 --> 00:18:58.237 I sus if you just kind of track the research, I suspect we're going to find 00:18:58.320 --> 00:19:01.837 is human cognition is a full body experience. Uh much much more than much 00:19:01.920 --> 00:19:06.397 more than people thought. Um and and so therefore to actually and this you know 00:19:06.480 --> 00:19:08.717 this is like a and this is you know one of the kind of big fundamental 00:19:08.800 --> 00:19:12.157 challenges in the AI AI field right now which is you know the form of AI that we 00:19:12.240 --> 00:19:15.917 have working is is the is the fully mind body dual version of it which is it's 00:19:16.000 --> 00:19:20.077 just a disembodi you know like a disembodied brain you know the robotics 00:19:20.160 --> 00:19:23.517 revolution for sure is coming when that happens when we put AI in in physical 00:19:23.600 --> 00:19:26.317 objects that move around the world you know you're going to be able to get 00:19:26.400 --> 00:19:29.997 closer to having that kind of you know inte integrated intellectual physical 00:19:30.080 --> 00:19:31.197 you know experience you're going to have 00:19:31.280 --> 00:19:37.720 sensors in the robots are going to be a lot more data. 00:19:45.040 --> 00:19:48.397 But it is just to me at least reading the research like that all those ideas 00:19:48.480 --> 00:19:51.757 feel very nent and we have a lot of work to do to try to figure that out. 00:19:51.840 --> 00:19:56.957 >> Do you have a sense for how they are at theory of mind today? Um or do you have 00:19:57.040 --> 00:20:00.157 a where the limitations are? you like to talk to them a lot. Are there any 00:20:00.240 --> 00:20:02.877 particular things that are particularly surprising to you as you do? 00:20:02.960 --> 00:20:05.757 >> Yeah, I would say generally they're really good. Um, yeah. And so like one 00:20:05.840 --> 00:20:08.717 of the one one of the more I find one of the more fascinating ways, you know, to 00:20:08.800 --> 00:20:12.637 to work with language models is actually have them create personas. Um, and uh 00:20:12.720 --> 00:20:16.397 and then you know, basically have well actually so I like I like I like 00:20:16.480 --> 00:20:19.117 basically I like Socratic dialogues. I like when things are argued out and like 00:20:19.200 --> 00:20:23.117 a Socratic dialogue. And so you you know tell a tell any advanced LLM today to 00:20:23.200 --> 00:20:25.757 create a Socratic dialogue and it'll either make up the personas, you can 00:20:25.840 --> 00:20:29.037 tell what it is, it does a good job. it has this very very annoying property 00:20:29.120 --> 00:20:31.197 which is it wants everybody to be happy. 00:20:31.280 --> 00:20:35.437 Um and so it wants all of its personas to agree. Um and so by default it will 00:20:35.520 --> 00:20:39.117 have a uh it will have a briefly interesting discussion and then it will 00:20:39.200 --> 00:20:42.477 sort of figure out you know basically it like you're watching I don't know PBS 00:20:42.560 --> 00:20:44.397 special or something. It'll it'll kind of figure out how to bring everybody 00:20:44.480 --> 00:20:46.797 into agreement. Everybody's happy at the end of the discussion. And of course I 00:20:46.880 --> 00:20:50.877 hate that. Like it drives me nuts. I don't want that. So instead I I 00:20:50.960 --> 00:20:52.637 tell it I'm like make the conversation 00:20:52.720 --> 00:20:57.997 more tense, right? And like fraught with like anger and like you know people, you 00:20:58.080 --> 00:21:01.357 know, going to get like increasingly upset throughout the conversation. And 00:21:01.440 --> 00:21:04.237 then it starts to get really interesting. Um and then I and then I 00:21:04.320 --> 00:21:07.997 tell it, you know, bring it, you know, use introduce a lot more cursing. Um you 00:21:08.080 --> 00:21:10.797 know, really have them go at it like all the gloves come off, they're going for 00:21:10.880 --> 00:21:13.357 pull full, you know, you know, reputational destruction of each other. 00:21:13.440 --> 00:21:14.637 >> You do a lot of these skits. 00:21:14.720 --> 00:21:16.797 >> Yeah, these skits. And then I get carried away and then I'm like it turns 00:21:16.880 --> 00:21:19.197 out they're all like secret ninjas and then they all start fighting and you've 00:21:19.280 --> 00:21:22.717 got Einstein you know you know you know hitting you know Neils Bore with 00:21:22.800 --> 00:21:26.637 nunchucks and it and by the way it's happy to do that too. Um so you do have 00:21:26.720 --> 00:21:29.597 to you have to you have to control yourself but it it is very good a theory 00:21:29.680 --> 00:21:32.557 of mind and then I'll give you another example. There's a there's a startup 00:21:32.640 --> 00:21:36.637 actually in the UK uh in uh in in the world of politics and and what what they 00:21:36.720 --> 00:21:40.797 found is that um they found that language models now are good enough so 00:21:40.880 --> 00:21:44.477 specifically for for politics which is sort of a sub subcategory where this 00:21:44.560 --> 00:21:48.477 this idea matters. Um so you know in politics people do focus group you do 00:21:48.560 --> 00:21:51.757 focus groups of voters all the time and and by the way many businesses also do 00:21:51.840 --> 00:21:54.477 that. Um, you know, so you get a bunch of people together from different 00:21:54.560 --> 00:21:56.797 backgrounds in a room and you kind of guide them through a discussion and try 00:21:56.880 --> 00:21:59.517 to get their their points of view on things. And and focus groups are often 00:21:59.600 --> 00:22:02.637 surprising. Like politicians who if you talk to politicians who do focus groups, 00:22:02.720 --> 00:22:05.117 they're often surprising. They're often surprised by the things that they 00:22:05.200 --> 00:22:07.597 thought voters cared about is actually not the things that voters care about. 00:22:07.680 --> 00:22:10.797 And so you can actually learn a lot by doing this. But focus groups are very 00:22:10.880 --> 00:22:13.197 expensive to run. And then there's a long lag time because they have to be 00:22:13.280 --> 00:22:15.997 actually physically organized and you have to recruit people and vet people 00:22:16.080 --> 00:22:19.597 and and so forth. Um, and so it turns out that the the the the 00:22:19.680 --> 00:22:23.037 state-of-the-art models now are are good enough at this so they can actually they 00:22:23.120 --> 00:22:27.277 can they can correctly um accurately reproduce a focus group of real people 00:22:27.360 --> 00:22:30.957 um inside the model. Um so so so they're good enough to clear that bar. In other 00:22:31.040 --> 00:22:33.837 words, you you can basically have a focus group actually happening in the 00:22:33.920 --> 00:22:36.877 model where you create personas in the model and then it actually accurately 00:22:36.960 --> 00:22:40.157 represents, you know, a college student from, you know, Kentucky is contrasted 00:22:40.240 --> 00:22:42.797 to a housewife from Tennessee is contrasted to a, you know, whatever 00:22:42.880 --> 00:22:44.797 whatever you you just like specify this. 00:22:44.880 --> 00:22:47.197 And so, you know, they're good enough to clear they're good enough to clear that 00:22:47.280 --> 00:22:50.157 bar and, you know, we'll we'll see how far they get. 00:22:50.240 --> 00:22:52.797 >> I want to segue to the bubble 00:22:52.880 --> 00:22:56.157 conversation. Uh, Amin and G2, Jensen 00:22:56.240 --> 00:22:58.557 and Matt spoke about the enormous scale 00:22:58.640 --> 00:23:00.397 of physical infrastructure being built 00:23:00.480 --> 00:23:04.237 out. AI capex is 1% of GDP. How should 00:23:04.320 --> 00:23:07.437 we understand and think about this bubble question? 00:23:07.520 --> 00:23:10.797 >> Well, I think the fact that it's a question means we're not in a bubble. 00:23:10.880 --> 00:23:14.557 That's the first thing to understand. I mean, a bubble is a psychological 00:23:14.640 --> 00:23:19.437 phenomenon um as much as anything. And in order to get to a bubble, everybody 00:23:19.520 --> 00:23:23.757 has to believe it's not a bubble. That that's sort of the the core mechanic of 00:23:23.840 --> 00:23:28.157 it. And that, you know, we call that capitulation. Everybody just gives up. 00:23:28.240 --> 00:23:31.997 Like, okay, I'm not going to short these stocks anymore. I'm tired of losing all 00:23:32.080 --> 00:23:37.277 my money. I'm going to go long. Uh and we saw that actually in, 00:23:37.360 --> 00:23:38.877 you know, and a little bit of question 00:23:38.960 --> 00:23:41.757 like really what was the tech bubble? Um 00:23:41.840 --> 00:23:45.597 but in the kind of.com era right as the 00:23:45.680 --> 00:23:49.437 prices went through the roof Warren Buffett started inviting investing in 00:23:49.520 --> 00:23:52.717 tech. So like and he swore he would never invest in tech because you didn't 00:23:52.800 --> 00:23:57.117 understand it. And so if he capitulated nobody was saying it was a bubble when 00:23:57.200 --> 00:23:59.357 it became like a quote unquote bubble. 00:23:59.440 --> 00:24:02.957 Now if you look at that phenomenon 00:24:03.040 --> 00:24:07.197 um the internet clearly was not a bubble. Uh you know it was a real thing. 00:24:07.280 --> 00:24:09.837 It was in the short term there was a 00:24:09.920 --> 00:24:12.797 kind of price dislocation that happened 00:24:12.880 --> 00:24:17.037 because uh the the market um you know 00:24:17.120 --> 00:24:21.197 there were just not enough people on the network to make those products go at the 00:24:21.280 --> 00:24:26.317 time uh and then the prices kind of outran the market. 00:24:26.400 --> 00:24:29.837 you know, in AI, it's much harder to see that because there's so much demand in 00:24:29.920 --> 00:24:34.237 the short term, right? Like we don't have a demand problem right now. And 00:24:34.320 --> 00:24:35.917 like the idea that we're going to have a 00:24:36.000 --> 00:24:37.997 demand problem five years from now to me 00:24:38.080 --> 00:24:41.357 seems quite absurd. Uh, you know, could 00:24:41.440 --> 00:24:46.717 there be like weird bottlenecks that that appear, you know, like we just at 00:24:46.800 --> 00:24:50.397 some point we just don't have enough cooling or something like that? Yeah, 00:24:50.480 --> 00:24:53.677 you know, maybe. But like like right now 00:24:53.760 --> 00:24:57.437 if you look at demand and supply and 00:24:57.520 --> 00:25:02.557 what's going on and multiples uh against growth, 00:25:02.640 --> 00:25:06.637 it doesn't look like a bubble at all to me. Um but 00:25:06.720 --> 00:25:08.477 I don't know. Do you think it's a bubble, Mark? 00:25:08.560 --> 00:25:10.237 >> Yeah. Look, I I would just say this. 00:25:10.320 --> 00:25:13.517 Yeah. Like nobody know So nobody knows in the sense of like the experts like if 00:25:13.600 --> 00:25:15.597 you're talking to anybody at like a hedge fun or a bank or whatever, like 00:25:15.680 --> 00:25:20.237 they definitely don't know. Um uh generally the CEOs don't know. So it the 00:25:20.320 --> 00:25:24.397 >> by the way a lot of VCs don't know that they just get upset like VCs get like 00:25:24.480 --> 00:25:26.637 emotionally upset when you guys have 00:25:26.720 --> 00:25:29.357 higher valuations like it it makes them 00:25:29.440 --> 00:25:32.797 like like angry. Uh and you know and I I 00:25:32.880 --> 00:25:34.877 get it all the time and I'm like what 00:25:34.960 --> 00:25:37.277 are you mad about? Like the is 00:25:37.360 --> 00:25:40.797 working man be happy. Come on. But so so 00:25:40.880 --> 00:25:45.837 like there's a lot of emotion around like people wanting it to be be a bubble. 00:25:45.920 --> 00:25:48.397 >> Yeah. No, nothing's worse than passing on a deal and then having the company 00:25:48.480 --> 00:25:50.717 become a great success. Like it's just I was just just put 00:25:50.800 --> 00:25:52.477 >> that that valuation is outrageous. 00:25:52.560 --> 00:25:55.117 >> You can be furious about that for 30 years in our business. It's it's 00:25:55.200 --> 00:25:58.637 amazing. Um and you can find Yeah. You come up with all kinds of reasons to 00:25:58.720 --> 00:26:01.917 cope and and explain why it wasn't your mistake. But it's you know it's the 00:26:02.000 --> 00:26:05.997 world it's the world that's wrong, not me. Right. So there there's a lot of 00:26:06.080 --> 00:26:09.037 that. Yeah. Uh yeah. So I I just I would just I would just say like I would 00:26:09.120 --> 00:26:12.157 always say bring the conversation back to ground truth fundamentals. And the 00:26:12.240 --> 00:26:15.517 the two big ground truth fundamentals are number one, does the technology 00:26:15.600 --> 00:26:18.637 actually work? Um, you know, can it deliver on its promise? And then number 00:26:18.720 --> 00:26:21.997 two, is are customers paying for it? Um, and if those if those two things are 00:26:22.080 --> 00:26:25.597 true, then it's very hard to it's very hard to uh like as long as those two 00:26:25.680 --> 00:26:28.957 things stay grounded. Um, you know, gen generally generally things are going to 00:26:29.040 --> 00:26:31.037 I think are going to be on track. 00:26:31.120 --> 00:26:34.477 >> Yeah. When Gavin was up here with DG, he said 00:26:34.560 --> 00:26:38.157 chatbt was a Pearl Harbor moment for Google, the moment when the giant wakes 00:26:38.240 --> 00:26:41.917 up. when when we look at history and and platform shifts, what determine whether 00:26:42.000 --> 00:26:45.117 the incumbent actually wins the next wave versus versus new entrance or how 00:26:45.200 --> 00:26:48.637 should we think about that in in AMP? 00:26:48.720 --> 00:26:51.597 >> Well, you know, reacting to it is 00:26:51.680 --> 00:26:55.357 important. Um, but that doesn't mean 00:26:55.440 --> 00:26:58.477 like I it's a Pearl Harbor moment. I 00:26:58.560 --> 00:26:59.997 think 00:27:00.080 --> 00:27:02.797 Google got their head out of their ass. 00:27:02.880 --> 00:27:06.600 It was the sound of it. 00:27:06.960 --> 00:27:11.837 Uh so you know they're not going to get completely run over but 00:27:11.920 --> 00:27:15.597 nonetheless like I don't think open AI is going away. So like they they 00:27:15.680 --> 00:27:20.797 definitely let that happened. Um yeah some of it to speed and then just look 00:27:20.880 --> 00:27:22.557 it's execution over a long period of 00:27:22.640 --> 00:27:25.197 time and uh you know some of these very 00:27:25.280 --> 00:27:28.077 large companies to varying degrees have 00:27:28.160 --> 00:27:31.997 lost their ability to execute. And so if 00:27:32.080 --> 00:27:35.837 you're talking about a brand new platform and you're talking about, you 00:27:35.920 --> 00:27:38.957 know, kind of building for a long time, it's like, you know, Microsoft got 00:27:39.040 --> 00:27:41.197 caught with their pants down on Google. 00:27:41.280 --> 00:27:44.317 Um, Microsoft's still like very strong, 00:27:44.400 --> 00:27:47.277 but they missed that whole opportunity. 00:27:47.360 --> 00:27:52.477 They also missed the opportunity. You know, Apple was nothing and Microsoft 00:27:52.560 --> 00:27:55.437 fully believed that they were going to own mobile computing. They completely 00:27:55.520 --> 00:27:58.797 missed that one. But they were still so big from their Windows monopoly they 00:27:58.880 --> 00:28:01.917 could build into other things. So you 00:28:02.000 --> 00:28:03.997 know I think generally the new companies 00:28:04.080 --> 00:28:07.277 have won the new markets. Uh 00:28:07.360 --> 00:28:11.197 and that doesn't mean the big company the biggest companies the biggest 00:28:11.280 --> 00:28:15.277 monopolies from the prior generation just last a long time is the way I would 00:28:15.360 --> 00:28:18.237 look at it. >> Yeah. I I also think we don't quite know 00:28:18.320 --> 00:28:22.157 like it's all happened so fast we we actually don't I think we don't yet know 00:28:22.240 --> 00:28:24.557 the shape and form of the ultimate products. 00:28:24.640 --> 00:28:27.517 >> Yeah. >> Um Right. And so and so like because 00:28:27.600 --> 00:28:29.917 it's it's tempting and this is kind of what what always happens. It's kind of 00:28:30.000 --> 00:28:32.397 it's kind of tempting to look I'm not saying what that's what these guys did 00:28:32.480 --> 00:28:35.037 on stage but it's kind of tempting to look you sometimes you hear the kind of 00:28:35.120 --> 00:28:37.437 reductive version of this which is basically it's like oh there's either 00:28:37.520 --> 00:28:40.717 going to be a chatbot or a search engine right the competition is between a 00:28:40.800 --> 00:28:44.237 chatbot and a search engine and the problem Google has is the classic 00:28:44.320 --> 00:28:47.037 problem of dis you know disruption. are you going to disrupt the 10 blue links 00:28:47.120 --> 00:28:50.797 model and swap in you know at you know sort of uh AI answers and you know 00:28:50.880 --> 00:28:53.757 potentially disrupt the advertising model and then the problem OpenAI has is 00:28:53.840 --> 00:28:56.477 they have the the full you know the full chat product but you know they don't 00:28:56.560 --> 00:28:59.437 have the advertising yet and they don't have the distribution Google scale 00:28:59.520 --> 00:29:02.877 distribution and so you know you kind of say okay that's a fairly that's a fairly 00:29:02.960 --> 00:29:05.917 cl like that'd be straight out of a like you know the innovator's dilemma you 00:29:06.000 --> 00:29:10.237 know business textbook like this is just a very clear you know one one versus one 00:29:10.320 --> 00:29:14.397 you know kind of dynamic but that assumes that you know the mistake that 00:29:14.480 --> 00:29:17.437 you could make in thinking way is that assumes that the forms of the product in 00:29:17.520 --> 00:29:21.117 5 10 15 20 years that that are going to be the main things that people use are 00:29:21.200 --> 00:29:25.277 going to be either a search engine or a chatbot, right? Um and and you know the 00:29:25.360 --> 00:29:28.797 just there's you know there's just obvious historical analogies. One just 00:29:28.880 --> 00:29:32.477 obvious historical analogy is, you know, the the personal computer from sort of 00:29:32.560 --> 00:29:37.117 invention in 1975 through to, you know, basically 1992, 00:29:37.200 --> 00:29:40.797 you know, was was a was a text prompt system, right? Um, you know, and at the 00:29:40.880 --> 00:29:43.757 time, by the way, an interactive text prompt was a big advance over the 00:29:43.840 --> 00:29:46.957 previous generation of like punch card systems, time sharing systems. Uh, and 00:29:47.040 --> 00:29:51.357 then, you know, it was, you know, 1992, so was what, seven, 17 years in, you 00:29:51.440 --> 00:29:54.317 know, the whole industry took a left turn into GUIEs and never looked back, 00:29:54.400 --> 00:29:56.957 you know. And then by the way, you know, 5 years after that, the industry took a 00:29:57.040 --> 00:30:00.557 left turn in web browsers and never looked back, right? And so the very 00:30:00.640 --> 00:30:04.077 shape and form and nature of the user experience and how it and how it fits 00:30:04.160 --> 00:30:08.077 into our lives, uh, you know, is is is I think still unformed. And so like and 00:30:08.160 --> 00:30:10.637 you know, look, I'm I'm sure there will be chat bots 20 years from now, but I 00:30:10.720 --> 00:30:14.797 I'm I'm pretty confident that, um, you know, both the current chatbot companies 00:30:14.880 --> 00:30:17.677 and many new companies are going to figure out many kinds of user 00:30:17.760 --> 00:30:20.957 experiences that are radically different that we don't we don't even know yet. 00:30:21.040 --> 00:30:23.277 Yeah. And by and by the way, that's one of the things of course that keeps the 00:30:23.360 --> 00:30:26.077 tech industry fun, which is it, you know, especially on the especially on 00:30:26.160 --> 00:30:29.197 the software side, you know, is it's not it's not it's not obvious what the shape 00:30:29.280 --> 00:30:31.197 and form of the products are. And there's just I think there's just 00:30:31.280 --> 00:30:32.557 tremendous headroom for invention. 00:30:32.640 --> 00:30:35.597 >> As as you're coaching entrepreneurs and the entrepreneurs in this room, what 00:30:35.680 --> 00:30:40.237 what else feels different about this era or or or other advice that you find 00:30:40.320 --> 00:30:43.997 yourself whether it's around uh sort of the talent wars that are going on or 00:30:44.080 --> 00:30:47.437 other aspects that feel unique to this era? What what other advice do you want 00:30:47.520 --> 00:30:50.717 to be leaving our entrepreneurs with? 00:30:50.800 --> 00:30:54.157 that's unique to this era. Well, like I I I actually think you said the right 00:30:54.240 --> 00:30:57.117 thing, which is this is a unique era. 00:30:57.200 --> 00:30:59.277 And so 00:30:59.360 --> 00:31:01.757 trying to 00:31:01.840 --> 00:31:05.117 learn the organizational design lessons 00:31:05.200 --> 00:31:08.717 of the past or trying to learn um kind 00:31:08.800 --> 00:31:13.997 of too much from the last generation is can be deceptive 00:31:14.080 --> 00:31:15.437 because things really are different. 00:31:15.520 --> 00:31:17.357 like the way these you know the way your 00:31:17.440 --> 00:31:19.517 companies are getting built is is quite 00:31:19.600 --> 00:31:23.837 different in in many aspects. Uh and you 00:31:23.920 --> 00:31:26.397 know the types of 00:31:26.480 --> 00:31:29.677 you know what the just like our observ 00:31:29.760 --> 00:31:33.837 observation on like PhD AI researchers 00:31:33.920 --> 00:31:36.317 is just very different than like a 00:31:36.400 --> 00:31:39.357 traditional um engineer full stack 00:31:39.440 --> 00:31:42.397 engineer or something like that. So you 00:31:42.480 --> 00:31:46.397 know I I think you do have to think through a lot of things from first 00:31:46.480 --> 00:31:50.797 principles uh because it is different and like you know observing from the 00:31:50.880 --> 00:31:52.877 outside it's really different. 00:31:52.960 --> 00:31:54.717 >> Yeah. >> Yeah. And I would just offer like I I do 00:31:54.800 --> 00:31:56.797 think things are going to change. So I already talked about I think the shape 00:31:56.880 --> 00:31:58.477 and form of products is going to change. 00:31:58.560 --> 00:32:00.637 Um uh and so like I think there's still 00:32:00.720 --> 00:32:02.397 a lot of creativity there. I also think 00:32:02.480 --> 00:32:06.157 and I I I let's say I think that um like 00:32:06.240 --> 00:32:09.757 in a in a world of supply and demand the thing that creates gluts is shortages. 00:32:09.840 --> 00:32:13.197 Um right so like when something becomes too scarce there becomes a massive 00:32:13.280 --> 00:32:16.797 economic incentive to figure out how to unlock new supply and so the the the 00:32:16.880 --> 00:32:19.917 current generation of AI companies are really struggling with uh particular 00:32:20.000 --> 00:32:23.357 shortages of of the really talented AI researchers and engineers and then 00:32:23.440 --> 00:32:27.277 they're really you know challenged with shortage of of infrastructure capacity 00:32:27.360 --> 00:32:31.597 chips and and data centers and power. Um I I don't want to call timing on this. 00:32:31.680 --> 00:32:35.677 There will come a time when both of those things become gluts. Um and so you 00:32:35.760 --> 00:32:39.037 I don't know I don't know that we can plan for that. Um although I I would 00:32:39.120 --> 00:32:43.437 just say the following. Number one, um the the the researcher engineer side of 00:32:43.520 --> 00:32:47.517 things, it is striking. It is striking to the degree to which um there are 00:32:47.600 --> 00:32:51.197 excellent, you know, outstanding models coming out of China now. Um you know, 00:32:51.280 --> 00:32:54.397 and and in a m from multiple companies and you know, specifically, you know, uh 00:32:54.480 --> 00:32:58.397 Deepseek and and Quinn and Kimmy. Um it is striking how the teams that are 00:32:58.480 --> 00:33:02.157 making those are not, you know, the name brand, you know, for the most part, 00:33:02.240 --> 00:33:05.597 these are not like the name brand people with their names on all the papers. Um 00:33:05.680 --> 00:33:09.277 and and so like China is successfully decoding how to like basically take 00:33:09.360 --> 00:33:10.877 young people and train them up in the field. 00:33:10.960 --> 00:33:13.197 >> Well, and XAI to a large extent too. 00:33:13.280 --> 00:33:17.357 >> Yeah. Yeah. And so I I think that I think there's going to be and look it 00:33:17.440 --> 00:33:20.317 makes sense up until it it makes sense that for a while it's going to be this 00:33:20.400 --> 00:33:23.037 super esoteric skill set and people are going to pay through the nose for it. 00:33:23.120 --> 00:33:26.157 But like you know there's no question the information is right being 00:33:26.240 --> 00:33:28.957 transferred into the environment. People are learning how to do this. Um you know 00:33:29.040 --> 00:33:32.957 college kids are figuring it out. Um, and so, um, you know, there's there's 00:33:33.040 --> 00:33:35.597 and I don't know that there's ever going to be a talent glut per se, but like I 00:33:35.680 --> 00:33:37.517 think for sure there's gonna there's gonna be a lot more people in the future 00:33:37.600 --> 00:33:40.717 who of course know know how to build these things. Um, and then and then by 00:33:40.800 --> 00:33:43.437 the way also of course you know AI building AI, right? So that the the the 00:33:43.520 --> 00:33:46.797 the tools themselves are going to be better better at at contributing to 00:33:46.880 --> 00:33:50.237 that. And so and and I think that I think this is good because I think that 00:33:50.320 --> 00:33:52.957 you know the current level of of shortage of of engineers and researchers 00:33:53.040 --> 00:33:55.837 is is is too constraining. And then and then on the chip side I don't I don't 00:33:55.920 --> 00:33:58.237 want to I'm not a chip guy and I don't want to call call it specifically but 00:33:58.320 --> 00:34:00.957 like it it's never been the case. It's never been the case in the ship industry 00:34:01.040 --> 00:34:04.237 that there's ever, you know, every every shortage in the ship industry has always 00:34:04.320 --> 00:34:08.397 resulted in a glut uh because the the profit the profit pool of a shortage, 00:34:08.480 --> 00:34:11.517 the margins get too big, the incentive uh for other people to come in and 00:34:11.600 --> 00:34:14.157 figure out how to commoditize the function get too big. And so, you know, 00:34:14.240 --> 00:34:16.877 Nvidia has like, you know, the best position probably anybody's ever had in 00:34:16.960 --> 00:34:19.917 chips. But notwithstanding that, I I find it hard to believe that there's 00:34:20.000 --> 00:34:22.237 going to be this level of pressure on infrastructure in 5 years. 00:34:22.320 --> 00:34:24.477 >> Yeah. And even if the bottleneck within 00:34:24.560 --> 00:34:29.277 the infrastructure moves, so if if it becomes power, if it becomes cooling or 00:34:29.360 --> 00:34:32.957 or or anything else, then you'll have a chip glut for sure. Yeah. 00:34:33.040 --> 00:34:36.157 >> So So I I think over the I I would just say this, it's likely the challenges 00:34:36.240 --> 00:34:38.477 that we that we all have in five years from now are going to be different 00:34:38.560 --> 00:34:42.237 challenges. >> Yeah. Yeah. Yeah. Like don't 00:34:42.320 --> 00:34:46.477 don't definitely this industry of all industries don't look at us as static 00:34:46.560 --> 00:34:51.437 like you know the positions uh could change very very fast. 00:34:51.520 --> 00:34:55.757 Let's actually close on a more of this macro note. Mark, you mentioned China. 00:34:55.840 --> 00:34:59.597 Last month, we were in DC and what one of the big questions the senator has is 00:34:59.680 --> 00:35:03.037 how should we make sense of sort of the state of the AI race visa v China. Do 00:35:03.120 --> 00:35:06.797 you you want to share just the the high level um summary of what what you shared 00:35:06.880 --> 00:35:09.597 with them? >> Yeah. So my sense of things and I and I 00:35:09.680 --> 00:35:12.637 think the current I think the current if you just observe currently specifically 00:35:12.720 --> 00:35:14.637 like deep sea quan and and these models 00:35:14.720 --> 00:35:19.997 coming out of China I my my sense basically is like I would say the US 00:35:20.080 --> 00:35:22.957 specifically in the west generally but you know more and more specifically the 00:35:23.040 --> 00:35:26.557 US is like the conceptual innovations are you know have been you know coming 00:35:26.640 --> 00:35:29.197 out coming out of coming out of the US coming out of the west you know kind of 00:35:29.280 --> 00:35:33.837 the the big kind of conceptual breakthroughs um uh China is extremely 00:35:33.920 --> 00:35:36.637 good at picking up ideas and implementing them and scaling them and 00:35:36.720 --> 00:35:39.437 commoditizing them and and you know that's that they do that obviously 00:35:39.520 --> 00:35:43.117 throughout the manufacturing world. Um and and they're doing it now very I 00:35:43.200 --> 00:35:47.357 think successfully uh sort of in AI. Um and so I would say that they're running 00:35:47.440 --> 00:35:50.717 they're running the catch-up game like really well. Um you know and then 00:35:50.800 --> 00:35:53.517 there's there's sort of always this question of like how much of that is 00:35:53.600 --> 00:35:57.197 like being done let's just say like authentically um uh you know through 00:35:57.280 --> 00:36:00.237 hard work um and smart people and then how much is being done with maybe a 00:36:00.320 --> 00:36:03.117 little bit of help um maybe a little USB 00:36:03.200 --> 00:36:07.997 stick uh in the middle of the night uh you know kind of help um Okay. 00:36:08.080 --> 00:36:10.157 >> So, uh, you know, there's always a little bit of a question, but like 00:36:10.240 --> 00:36:13.677 either either way, uh, you know, they're they're doing a great job. Uh, obviously 00:36:13.760 --> 00:36:17.197 they they aspire to, you know, more than that. Um, and there are many very smart 00:36:17.280 --> 00:36:19.997 and creative people in China. And so, you know, it will be interesting now to 00:36:20.080 --> 00:36:22.797 see, you know, the level to which the conceptual breakthroughs start to come 00:36:22.880 --> 00:36:26.877 from there and whether they whether they pull ahead. Um, and so, but like I would 00:36:26.960 --> 00:36:29.437 say like what we tell people in Washington is like look this this is a 00:36:29.520 --> 00:36:31.277 foot this is now this is a full-on race. 00:36:31.360 --> 00:36:33.517 It's a foot race. It's a game of inches. 00:36:33.600 --> 00:36:36.317 Like we're not going to have a 5year lead. we're going to have like maybe a 00:36:36.400 --> 00:36:39.757 six-month lead. Like we have to run fast. We have to win. Like we we have to 00:36:39.840 --> 00:36:42.717 we have to do this. We we can't and then we can't put constraints on our 00:36:42.800 --> 00:36:45.597 companies that the the the Chinese government isn't putting on their own 00:36:45.680 --> 00:36:49.677 companies. And so um you know we'll just lose and you know do do you really want 00:36:49.760 --> 00:36:51.677 do you really want to wake up in the morning and live in a world you know 00:36:51.760 --> 00:36:53.837 really controlled and run by Chinese AI? 00:36:53.920 --> 00:36:55.277 Most of us would say no we don't want to 00:36:55.360 --> 00:36:59.117 live in that world. Um and so um so so 00:36:59.200 --> 00:37:02.317 that so there's that and I would say I feel moderately good about that just 00:37:02.400 --> 00:37:05.837 because I think that I think we're we're really good at software. um you know the 00:37:05.920 --> 00:37:09.197 minute this goes into you know embodied AI in the form of robotics I think 00:37:09.280 --> 00:37:12.317 things get a lot scarier and you know this is the thing I'm now spending time 00:37:12.400 --> 00:37:16.637 in DC trying to really educate people on which is you know the ch because the US 00:37:16.720 --> 00:37:19.597 and the west have chosen to de-industrialize to the extent that we 00:37:19.680 --> 00:37:24.157 have over the last 40 years um you know China China specifically now has this 00:37:24.240 --> 00:37:28.717 giant industrial ecosystem for building you know sort of mechanical electrical 00:37:28.800 --> 00:37:33.357 and um uh and semiconductor and now software you know devices of all kinds 00:37:33.440 --> 00:37:39.037 including phones and drones phones and cars um and robots. Um and so uh you 00:37:39.120 --> 00:37:41.757 know there's going to be a phase two to the AI revolution. It's going to be 00:37:41.840 --> 00:37:45.917 robotics. It's going to happen you know pretty quickly here I think. Um and when 00:37:46.000 --> 00:37:49.437 it does like even if the US stays ahead in software like the robots's got to get 00:37:49.520 --> 00:37:53.197 built and that's not an easy thing. And it's not just like a company that does 00:37:53.280 --> 00:37:55.997 that. It's got to be an entire ecosystem. Um and it's you know it's 00:37:56.080 --> 00:37:59.037 going to be you know it's going to you know like I mean you know the car 00:37:59.120 --> 00:38:01.837 industry was not three car companies. It was thousands and thousands of of of 00:38:01.920 --> 00:38:04.557 component suppliers building all the parts. And it's been the same thing for 00:38:04.640 --> 00:38:07.437 airplanes and the same thing for computers and everything else. Um it's 00:38:07.520 --> 00:38:08.797 going to be the same thing for robotics. 00:38:08.880 --> 00:38:12.317 Um and you know by default sitting here today that's all going to happen in 00:38:12.400 --> 00:38:15.597 China. And so even if they never quite catch us in software they might just 00:38:15.680 --> 00:38:19.917 like lap us in hardware and and that'll be that. Um you know the good news is I 00:38:20.000 --> 00:38:23.277 I think there's a growing awareness in there's a growing awareness I would say 00:38:23.360 --> 00:38:26.797 across the political political spectrum in the US that like de-industrialization 00:38:26.880 --> 00:38:29.597 went too far. Uh and there's a growing desire to kind of figure out how to 00:38:29.680 --> 00:38:33.757 reverse that. Um, and um, you know, I say I'm guardedly optimistic that we'll 00:38:33.840 --> 00:38:37.357 be making progress on that, but I think there's a lot of work to be done 00:38:37.440 --> 00:38:39.837 >> on that call to arms. Uh, let's wrap. 00:38:39.920 --> 00:38:43.997 Uh, thank you Mark and Ben. To to wrap up, I'd like to welcome back. 00:38:44.080 --> 00:38:51.080 >> Thank you, everybody.