Microsoft CEO Satya Nadella on How Intelligence Will Reshape Our World

Opening and the token factory

00:00Especially in technology business, we collectively as an industry have to just mature and mature at a very fast rate because the impact of digital technology in our lives and in our society is so deep. Now the first thing that we have been doing is building the system all the way from power to the DC to the rack to the accelerators to the network. You have to create magic.

00:35Welcome to the stage Satia Nadella. >> Hey Satya, thank you so much for doing this. I know last minute schedule changes. You can be here in person. >> No, not at all. It's my pleasure Anel. It's great to be with you. >> Yeah. Right before this, we uh had an opportunity to hear from Bob Mechaf about buildouts throughout the 80s and 90s. So you've got a tall order. I know you admire Bob as much as I do. U so so may maybe one place to start is you know there's so much buildout happening now. What do you think will make or break the next decade of this infrastructure buildout that's happening now? >> Yeah. I mean the way I think um analis each of these buildouts um of infrastructure always comes down to two

01:22sets of things right. how efficient is the buildout for what is the new platform uh and then what's the way it gets to ubiquity in some sense it's the app server right I've always thought about at least from a a platform perspective what's the core primitives of the platform and then how does the platform have a developer interface so that developers can use it right I mean at the end of the day you could say that's what operating systems are right so the operating system book at least I read in school was that tenant bomb thing which is you abstract the hardware and expose an app model and that's a good way to I think think about the infrastructure so in that context the first thing is how do you make things efficient this time

Efficiency and the agent factory

02:05around it's that tokens per dollar per watt right so the token factory um has to be super efficient so there's a lot of work that goes on I think is going on is at the systems level uh all the way to power uh right uh and so that's one side of it. Then on the app server side, it's more like an agent factory. So if you have a token factory, so think of it just as throughput's got to be most efficient. Uh but then you got to harness that to build very efficient effective agents and that is an entire different ball of wax, right? which is you got to have a set of application services so that any agent that you're building uh delivers on some eval metric that matters in the world and so those are the two at least I

02:56think as the infrastructure pieces. >> Yeah, I think you have such a engineer point of view of this and I I remember you mentioning the sort of complete thought from silicon to operating systems to applications. um how do you think that cascades down to the core part of how networks are designed in all this buildout and across the WAN and the edge and how do you think about that? >> Yeah, I mean it's it's fascinating. I mean obviously what you all are doing uh in fact the first time I remember you and I chatted when you said you're going to vertically integrate build this network as a service platform. It's pretty impressive to sort of say that's I think a little bit of what we are also doing right when people say wow there's

Networks, Fairwater and the AI WAN

03:36a lot of capex being spent. I mean the reality is the data center is the computer is becoming more real today right I mean that's kind of where you want to really think about uh uh the complete thought is how do you really go deliver on that tokens per dollar per watt uh but just take one of the new data centers we just brought live uh last week or a week before uh called fairwater uh I mean interesting enough we built it as a double-decka uh data center this is the first time we did that uh and we did that because to in fact for the network which is can you have the 5 million network connections uh that are there in that particular data center optimized for cable length uh right because we wanted the most

04:22dense GPU environment uh and so in order to do that uh you kind of had to even think rethink the construction uh if you will um and so to me and then not only did we do that in one data center but we also built an AI van And I was sort of very pleased to see you all working with Lumen and building your own sort of connection between what you're doing and what they're doing. In fact, that's kind of a similar thing we did. We now have uh we have a Wisconsin Fairwater and Atlanta Fairwater and both of them are sort oforked through this AI van. So you can now run distributed training across these. So to me that's I think what you kind of take the workload. I worked for this gentleman who at Microsoft uh

05:03called Paul Merittz long time ago. He used to always teach us as you know systems folks growing up saying look at the end of the day uh you really want to fall in love with sort of the what the new workload is and then design infrastructure uh and application servers. So that's what we're doing. You look at these new data parallel workloads which kind of are unlike anything we've seen before a training inference data gen data pipelines for all these evalu rubrics what have you you go build for that and that's where I think the infrastructure is being rethought end to end yeah is here somewhere in the audience but I I watched the interview you did with him and the scale of the buildout um how do

Engineering awe and social permission

05:45you feel as an engineer like these are masscale projects X how do you think about this and you know as an engineer I know like you understand these systems are you in awe just as much as the rest of us are to look at the scale of this >> yeah I mean I must say we're in awe of it but I'm also quite frankly humbled by one fact Anel which is uh the amount of expertise that is required to pull something like this off right um if you think about the electrical engineers the mechanical engineers, the construction civil engineers, the obviously the computer engineers, all of them coming together, composing on a project timeline, making something like this happen, delivering on an SLA. Uh so to some degree uh it is what I'm in awe of

06:34and I I also feel um that this is one of those places where we've got to get it right that not only do we have to build stuff that works and works super efficiently. Uh but it's also to do what uh I think that's the main thing as engineers. uh because for the first time as I mean when I say uh tokens per dollar per watt if you're really going to consume a lot of energy uh one thing that I think we all in this era have to be more grounded on is the social permission uh to go take that type of energy convert it into tokens is not going to be about celebrating like some eal that you and I can get hyped up about. It has to be something that the world actually benefits from. you guys are building out

Security, reliability and sovereignty

07:24the entire stack like you mentioned which is amazing to watch. How do you think you maintain that while also thinking about reliability and trust because so many companies depend on what you guys are doing and how much you're building up? >> I mean that's a you know sort of when I think about my you know priorities for us quite frankly there are three uh at the company. One is security uh because the cyber landscape just becomes even more complex every day. uh and then quality and reliability and if you do a good job on those two you get sort of permission to even think about AI that's kind of I'm pretty clear that that's the sequence the order of operations even for us because uh you know if I look at uh Azure today in the

08:0760 plus regions around the world uh people count on it uh as critical infrastructure and so therefore we don't take it for granted and so you have to think about uh operating it uh with those qualities and the fundamentals and that I think is the key right which is these are um at scale nothing is a commodity it's sort of very complex and so therefore what and in fact if if you think about it the primitives have to be rethought right for the next 10x everything right um uh and so what you built for like we added more data center capacity in the last 18 months uh than perhaps in the first 10 years of Azure right so if you sort of say that's the scale uh then some of the core primitives uh have to be rebuilt. Uh but then on top

08:56of it, there's one other thing that you brought up which I think is super important which is operating around the world requires us to respect what I think of as the legitimate sovereignty needs uh people have right because in Europe they have their own data protection laws. So therefore, for example, one of the hardest projects we did over the last five years is the EU data boundary. uh right which is how do you make sure that all the data that uh is European remains in Europe right that so that's the data plane now we have a control plane that goes with it that is resident um and so we've kind of continuously built for what I think is a distributed infrastructure but with real sovereignty boundaries uh and now with

09:42AI we're even getting to an another new sovereignty boundary which is at the end of the day every firm wants to know that their data and their unique understanding of data is being parlayed into their competitive differentiation versus just fodder for a model company. And so that I think is one of the more interesting sovereignty debates right what's the future of a firm right if the firm is all about knowledge you have which is tacet inside uh but if a model you know can see like you know 200 great samples of data and pick it up then that's a problem so therefore uh you really need to think about even sovereignty uh at the firm level very differently >> I remember you saying like the hardest

Changing how people think and work

10:25part is not the models but changing how people think and work with the stuff. What have you guys seen work best? What are you seeing from your customers and how does that work? Because you know, everybody has a different perception of how well these things work, how they don't or whether they're creative or not. How what have you seen in that change of in how people think and work with these models? >> Yeah. I mean I think the the first thing that is interesting uh that I see when somebody you know besides sort of let's say the hit app right whether it is chat GPT or copilot that you can deploy use and I mean you know if you go get beyond any firstparty stuff that we're doing right even GitHub

11:06copilot or what have you uh for an enterprise they need to start with I think a little bit of a change in mindset which is um we built all the workflows and uh at work uh based on a particular regime of digital infrastructure. I mean the reality is nothing much has changed since the be you know when I started in computing in terms of the workflow right I mean we had PCs uh and then talking to minis and then PCs started talking to servers and it became client servers right that's sort of what uh was the last time people said oh we'll have standard issues of office productivity we will have email and it'll replace fax machines and inter office memos and we will now be able to do a forecast in Excel, put in an email, send it around,

11:57people can enter numbers and you know, so there it went and oh, we'll buy an ERP system, a CRM system. We'll have a system of record. So, we kind of build a pretty sophisticated, I'll call a digital information back plane and now it's time to relitigate the entirety of it, right? because you kind of say, okay, this is how work happens today, but I need a mindset shift on if a mo a I have multiple agents working alongside all of the human workflow with digital systems. Uh how do I change the process uh of something like even forecasting? Um and I think that that's the first one. The second thing is you need a new tool chain, right? So you kind of need uh just like how the tools of the past were or I had

12:43Word or I had Excel. what are the new tools to be able to build agents uh and you know deploy agents, monitor agents, steer agents in fact so there's a complete new tool chain uh that is required um then that that requires us to develop skills not like the elite sort of AI scientist skill like it's sort of yeah we may have taken a class or two of Excel macros uh but we all could sort of use Excel though so what's I think everybody's going to be building agents like they did a spreadsheet need uh and so that's the skill set that needs to be broadly diffused and then the last thing is data because if you think about it the most interesting thing that we do is to do this context engineering which is how how do I make

13:27sure that the data meets the intelligence that meets the process change I want to affect right it's not like data defined in the silos I have because that's kind of like you have a very powerful intelligence in a reasoning engine but it's only as good as the data you feed it uh in terms of what it should reason and its action space and so I think these four or five things have to fall in place and that's the hard change management uh that we have and it's sort of we have to compress what hap think about it right all through the 90s and the 2000s you could say even in the cloud like what did we do mostly we moved the same systems to the cloud so quite frankly we had a 30-year run which was pretty

14:11stable in terms of how we thought about work getting done and now we suddenly have to take that 30 years and maybe compress it you know in the next three four years five years into something that yields massive productivity gains. >> You all started working on Azure 10 15 years ago and a lot of the fundamentals that were set that led to being able to build these large models. If you knew that this is how much buildout was going to happen, is there some things you would change on how you would do it differently? Like if you knew, you know, in 2025, this is the mass scale buildout that's going to happen to the level of, you know, national projects that hadn't been done before. Are there things you

Rethinking Azure with hindsight

14:49would do differently with that benefit of hindsight? >> Yeah, I mean the so the interesting thing is we did have, you know, basically the the cloud buildout was about making servers, right? because um much more you know it was much more elastic than any of us thought right I mean virtual machines uh there was insatiable demand for it right people wanted to just spin up a lot more compute uh one of the other things I had not realized quite frankly is the amount of I'll call it uh capital that was spent on say storage systems underneath virtual machines uh which suddenly now you could deploy a lot more efficiently and so that I think created And then of course the new workloads like the Hadoops and the you know the

15:36the data parallel workloads of that day more batch processing stuff that work has done at scale this time around I think it's kind of a new infrastructure right which is I feel this we've not seen anything as synchronous data parallel stuff like this and so therefore that's why I think even the shape of what we're building we've adjusted what we were doing to build for AI. But now when I look at even the way I think Azure will look five years from now, it'll look unlike anything we've did in the first 20 years of Azure. Um and so the you kind of reorient your core compute and storage and the network um or even by the way the memory interface. I think one of the more interesting things that may even happen

16:27is what's going to be uh the model architecture, the micro architecture in the AI accelerator and the memory. Uh that's the thing that perhaps is going to completely change. Um and so yes, so I think my short answer is we have to rethink uh the core footprint of what one considers a cloud. um knowing at one end I have to run an SAP workload at 59s uh and on the other end I have to run a highly efficient uh data parallel training job u right which gets has the maximum token throughput and so balancing that out in the right way I think is going to be one of the fundamental challenges for us >> in the last few buildouts one of the interesting things that happened is after this buildout starts maybe in the

17:2080s then in the 90s um things start to standardize. Today, if you go look at data centers and other places, every company is doing interesting things that they hold true themselves, but others might be doing different things than them. When do you think it'll be more standard on how to build this type of infrastructure where we have to standardize and then maybe follow up to that is what do you think should be standardized or is it the way we're doing it now where everybody does their own proprietary thing is the way to go? Yeah, I mean it it's it's a good point because I think the specialization will come as the supply chain and the value chain here sort of really achieves scale right I mean if I look at it um it's

Standardization and heterogeneous silicon

18:02sort you know what I mean if you take the the the classic argument right which is if you have scale you vertically integrate it can be very t you know very efficient very tight to sort of the idiosyncrasies of sort of how you think about whether it's your model or your application scaffolding and its requirements. Uh but for us as a platform provider, I mean like at this point we have to make sure that any new thing that gets written runs on Azure. Um and so therefore uh even at the silicon accelerator level we have to be heterogeneous, right? I mean at the end of the day the workloads are going to be uh heterogeneous. This the system architecture underneath those has to support right whether it's Nvidia, AMD

18:45and our own accelerators or what have you. uh and so I think that that's how we are going to have to standardize which is to basically build out uh what is a way to meet the real world needs um which by definition is going to be heterogeneous and therefore what's the set of standards that emerge I think the first place is networking uh if I look at the scale up and scale out um network for these new workloads uh all coming together may be the place right because I think that you know the model has still and the model evolution has still a lot more um influence on I'll call it the micro architecture of the one chiplet in the AI you know accelerator for sure uh and its connection to memory so some of that

19:36stuff may still be more fastm moving but everything else quite frankly you can standardize um u and so that I think is where some of the exciting uh work by the way and it has to go all the way to the edge as well and I think that this is an obviously a massive thing for you all to participate in and we're excited about sort of being able to sort of say at the end of the day what's this managed service around networking uh that spans uh the entirety of the distributed computing fabric and that requires uh what you are doing as well >> and there's so many people here in the audience that are CIOS CTO's that are figuring out how to think about all these new tools that are coming out. What where have you seen the biggest ROI

Biggest ROI inside Microsoft

20:21in Microsoft? Um because you guys are obviously at the forefront. You have access to everything. How have those workflows changed the most? Yeah, I mean I I mean one of the fascinating things is the best news now for me Anil is when I start you know hearing of the ground up use of the tools right whether it's copilot whether it's foundry whether it's GitHub copilot people using them right so this is not uh you know some top- down mandate of hey because like the obvious use cases I think we're all well on our way right to customer service or what have you but the thing for example on networking I had myself like lost track of the number of for example fiber operators uh that power uh Azure like there are 500 plus uh

21:06operators that we have contracts with all over the world uh on our fiber operations and I mean at the end of the day it's a physical asset things happen there are fiber cuts there's issues and it's fascinating the devops pipeline that is there I mean I mean sometimes oh devops seems sophisticated it's not like GitHub actions I mean the thing is literally emails going on and and so one of our engineers what she did was she took Foundry built herself a bunch of agents. She's given them all names uh because she she had to there was no headcount. There was no way even if she had headcount she could grow a team to manage what was becoming the most manual of processes. And so she fundamentally identified the entire process of dealing

21:54with triages around fiber operations and maintenance and cuts and what have you. And to me that was like a great firsthand example. Same thing you know to your point about supply chain and the data you know these massive buildouts of our data center. You should see like the I thought man maybe somebody in uh you know in finance or in legal will figure out how to do contracts. the people who needed to really figure out the agentic way to go get contracts done uh because I did one data center contract here with this one supplier and replicated everywhere and so on. That's a place where completely the supply chain operations was completely inverted. I mean if you think about supply chain the basic way I I've learned is at the end

22:38of the quarter you go to your CFO you get beaten by the CFO for all the mistakes you made. Uh and the idea now is to actually have these agents help with the daily beating so to speak and so that is uh the fundamental way you can completely transform a business process. >> Amazing. Thank you so much for your time Sati. I really appreciate you doing this. >> My pleasure Anel. Thank you so much.