Greg Brockman on what the best engineers and leaders do today

Best engineers and AI

00:00We are here with Greg Brockman. He is the president and one of the co-founders of Open AI and we get the opportunity to chat with him, ask him some follow-up questions following Sam Alman's keynote, kicking off the day. So, Greg, thank you so much for being here and for helping millions of people figure out what the heck is coming next. >> I'm excited to be here. >> Yeah, >> happy to chats. >> So, you were the founding engineer and first CTO at Stripe. you are on a mission to become the best engineer in the world. if you were meeting with the best engineer in the world and maybe it's you. What is the best engineer in the world doing right now >> and then building with codeex but I mean I think I think that being an AI is just

00:39such an incredible uh opportunity right now and I think that we've very much been seeing that uh people who are engineers in startups or in other domains HFT anything like that coming into AI can have this massive impact and so I I think that the best engineering is possible and is really rewarding within AI. Do you have like specific features that you're thinking about when you think of the best engineer that is super ahead of the curve like day-to-day what are they doing? >> Well, I guess the way that I think about it, I mean, I can't I can't comment necessarily on every engineer. I could talk, right? So, the way that I work is that um I'm constantly context switching, >> right? So, I'll like be sort of, you

Iteration speed and background processing

01:20know, writing some function or something and I'll hit a little mental block. I'm like, oh, exactly how should this be structured? What should what should this abstraction be? Something like that. and I'll flip to Slack for like 10 seconds and then I'll flip back or I'll go to email and it's just like I find this constant reset is something that means that I can background process a problem um and then by the time I get back to it I know the answer and so I think it's just about like really about iteration speed about problem flux and just trying to get as much like context as much speed as possible. >> God, this reminds me of like when people would go to sleep with a math problem in their head in high school and they would

01:53wake up and it wouldn't solve but you're doing that while awake. >> Yes. And I I actually literally did the fall asleep. I had this research problem I been working on in high school uh in math for months and I remember I was >> for math's last theorem or something quite wasn't wasn't quite that that advanced. It was a problem in uh in uh in duch sequences uh which is pretty cool stuff. U but I remember falling asleep and I was half asleep and I saw the answer. It was like it was this phenomenal experience. I remember feeling like I need to figure out how to replicate this. There is something about like their subconscious just like just chewing on a problem. >> So almost like early exposure is one of

Getting started and vibe coding

02:31the most helpful things. So getting in practice cycles is definitely one of those starting >> I So right and I think it's also just like it's just doing like I think the most important thing if you're starting out is not to be afraid right like there's you write some code it doesn't work fine >> there's no consequence to it right I think it's really about trying to find projects where you can learn where you're a little bit uncomfortable right and where you can just like go and try and especially with tools like codeex like it is just so much more accessible to get into programming >> for people when I was showing them chatbt and they had never seen it for more often than not I would ask it to

03:09like write a poem about their hobby and that was just a really fast way of seeing the magic. Is there like a do x for y that is a great starter spot for codeex users? >> Well, codeex is incredible at front-end engineering. Yeah. >> And so, okay, I have a I have a particular hobby project that I started working on like the first time I built it and I actually built it myself was back in 2008 or something which is a uh competitive chatbot game. I call it a reverse touring test, right? Traditional touring test, you have um you have two human participants um and that one of them is a judge, one of them is a is a player. Um and the goal is for the judge to be able to tell are they talking to a human or an AI. Now, in my competitive

Reverse Turing test project

03:54uh reverse turning test, the way that it works is that you are hooked up to two terminals, each player. Um, one is your opponent, a human, one is an AI. Your goal is to tell which one's which before your opponent does the same. >> Okay. >> Now, the thing that's interesting about this is that the optimal strategy for the bot is to act uh kind of botlike uh or so for the human is to act kind of botlike, right? Because you don't want to sort of distinguish whether or not you are the human. Um, but also to ask somewhat discerning questions to figure out is your opponent that or not. And so back in 2008, I remember I wrote like this huristic based bot that would like replay conversations from from its

04:33database that were similar to the ones that was having currently. And I got like a 30 or 40% win rate. Um and so it shows you that that it was just a cool problem. It was a cool setup. Um and uh the thing the reason I bring this up is that I've actually started using this as just a test of vibe coding. I just be like, describe that to an AI. And I've made versions of that with zero lines of code written by myself that are like 10x better than the thing that I wrote back in the day. >> Really? And when you see that happen, you're excited because you're adding on new things. You're bummed because it took your best idea and just did it. >> I mean, for me, it feels so empowering because I remember the struggle of

05:13implementing this back in the day and there were a bunch of CS fundamentals I did not know. Like for example, um serialization was something I'd not heard of like JSON, you know, JSON formatting of something or you know, whatever your favorite serialization library is. And I remember I just invented my own like, okay, I'll like have this like secret delimiter that was this like, you know, sequence of of characters that would be unlikely to not. Exactly. Exactly. I can't tell you what it is because then you'd be able to break my relation scheme and I would just use that to to separate records. And I, you know, it's just like a concept where when you're starting out, you just wouldn't like it's not it's

05:46like a fairly deep thing. Um, >> and to even realize what the question is you should ask, whereas today an AI looking at that code would point out to you, hey, this is like a much better way of doing what you're trying to accomplish. Um, or you can even just like do so much more than is possible otherwise. Like for myself, I've always been like a infrastructure backend kind of person. my front-end skills are a little bit atrocious. Um, [laughter] I used to, the way that I used to work is that I would basically just do like the hackiest possible HTML, JavaScript, CSS, and then have a designer come and like, you know, clean up my mess. Um, and now I don't need to. And so for me, I just feel like it just supplements me in a

Removing workflow friction with AI

06:26way that I've always wanted. >> I love that. And as someone who's only capable of making horrific HTML, I'm with you. Have you interacted with an engineer recently where they said something about the way they're working and you said, "Wow." Like what was that person doing that might have surprised you? Well, I actually remember around December or so, one of the engineers on my team who's absolutely phenomenal, one one of the best engineers I would say in the world, um telling me that he was getting a lot of value out of the chatbt desktop integration with his terminal and that it was a very simple thing. He was just like, >> if there's an error in my terminal, I just ask chatbt like what went wrong

07:08here? >> Yeah. >> And it debugs it for me. And it's just like you just realize the like little pieces of friction, right? It's like obviously this engineer is capable of reading the error message, >> but it's annoying. It's like overhead. It's slowness. >> And I think that what I see is people trying to find where does AI map to their workflow? Where can it remove little bits of friction? And sometimes you're just like that was it. That was the thing that that that made the difference. But the thing is >> all these things compound. And I think that people who are very discerning and people who are just really focused on their own productivity, their own tools, and it propagates into how people

07:43structure code bases, right? Because you think about, well, if I want to be using AI tools as much as possible, then how do I have code set up so that it's most amendable to AI? So fast tests, modular code, good readies for um for you know the API definitions between things, all of that stuff. >> It's almost like I know everyone talks about AI as an operating system. I lived in a 20 person AI house in San Francisco and I think that that term was like thrown around all the time. And what I'm seeing now more and more with like these real time overlays is that it's not just the foundation of how things are built. It's also watching over I mean for lack of a better word and co-working or pairing with everyone in every single

08:22line of business. It's amazing. >> Yeah, it it is really phenomenal. And I think we're still at the beginning to >> Okay, even at the beginning that's this slope. >> Oh, that that was that was an exponential. That was an exponential. I think I think we're we're still just like at the point >> my acid reflux feels like it's a little bit super >> for for early founders right now. Um and that could be Silicon Valley early founders or it could be you know an SMB solarreneur coach in the middle of you know their career. What are they building? What should they be building today? Well, I think two things. Like one is honestly there is so much to explore >> that I think people should just like if you have an idea, build it, right? It's

What founders should build

09:04like as simple as that. Like there is the barriers to entry are the lowest they've ever been. They're going to get lower. Don't worry. But we're at a point now where it's just like if you have a creative idea, you should see it, implement it, try it out. And >> I think that is maybe the highest order bit. um if you're looking to build like a durable business I think thinking about what are things that will benefit as the models get better not be obsolete and I that I think that things that require going into specific domains building a lot of expertise there that's the kind of thing that I think can be can be really uh durable in this in this time >> okay so vertical AI applying your expertise using AI to support that

09:39mission >> I think so yes >> yeah you know there's a lot of talk around like lean teams and I work with Fortune 500 enterprises that might have tens thousands of employees, but we're seeing that there's a lot of traction that comes from having a lean team. What does a lean team look like today? Like are you envisioning new roles, new uh processes, you know, do they still have daily standups? Like do you have a sense of how a team right now, a lean team is functioning in the world? >> You know, it's a great question. Uh and I think that this is the kind of thing where I think that almost the way that people operate has not yet caught up to the capability of the tools. Totally. And so, you know, even simple things

Lean teams and AI context

10:18like AI note-taking tools really just open up a new possibility of like, do you really need everyone to sit in the meeting, right? It's like, do you actually have really good tools that can summarize and surface to you the right information? And some of these require integrations that I just haven't seen be good yet, right? What you really want is you want something like chat pulse that just tells you here's everything happening across your team. Here's what you need to know. This person was missing this thing. You want that to happen in real time, right? That the AI is really being this mediator. Um, but maybe you can build that, right? Maybe you actually can do that. And maybe it requires changing how your team operates

10:51to make sure that the AI has that context and that if it's something that's like very sophisticated, requires a lot of the back context, you have all all that there. Um, this is probably true for development environments too of like if you want AI to not just sort of spit out code at you, which is only a small fraction of what software engineering is, right? Be able to operate your infrastructure, things like that. It would be very nice to be in a world where the AI has exactly the same environment and affordances that human developers do. And so I think that maybe it starts for me from really thinking about how to give AI all of that accessibility and all those affordances and then you build your processes around

11:25it. >> I'm obsessed with this because enterprise change is taking two 3 years. So having that builder mindset even if you're in finance is brilliant. Can we talk a little bit about chach pulse? We didn't get to hear it a little bit in the in the keynote, but I would love to hear your perspective. You're always thinking about AI in the future. Chachet Falls is one of the best proactive AI systems that I've seen and it is putting proactive AI into the hands of really any user of Chhatab. Can you talk a little bit about where you see proactive AI heading? >> Yes. So, you know, I think that at the beginning of the year, I tweeted something like, you know, 2025 is the year of agents. Um, I think that's a a

Proactive AI and ChatGPT Pulse

12:04theme that a lot of people have have said. Um, and I think that pulse is an example of an agent which is very different from how people have started to think about agents, right? I think that this proactive agent category is something that just doesn't really exist yet because the models have not been good enough. But it's kind of no longer an excuse because the models are really smart. Yeah. And I think that what we have found is that I it's just like where you want to go is you want this amplification of human intent through AI and we view chatbt as we want to shift it to be something that is a goal amplifier that helping you achieve your goals and that part of that is knowing what your goals are and part of that is

12:46actually then being able to do something intelligent be able to add value to them and I think that it just makes sense that there's only so much that you either have attention for or really want to do in terms of managing a fleet of agents. Like, you know, you want to do some of that, right? It's just like if you have a whole team of people working for you, you want to do a little bit of of micromanaging, but >> yeah, >> not that much. You really want people who are going to go and be trustworthy and dependable and going to go add value and do good things and come back with great reports or come back with like a suggestion of like, oh, you probably should, you know, not be eating as much red meat or, you know, whatever whatever

13:20the thing is. Um, and so, you know, and something like the the example I just said really requires the AI to have the relevant context, right? It needs information to be able to give you good actionable suggestions. And so, I think that that's going to be a really interesting part of how these kinds of platforms get built. >> And what I'm also telling audience and community and clients is that they also need like an SOP for goals. That a lot of these documents are knowledge gathering or uh relationship gathering. who's working on what uh time gathering, when did we announce this versus this, but there aren't a lot of documents that completely declare what the goal of something is. And so, do you think it's

Goal engineering and AI coworkers

14:00worth it making like a goals doc, like a goal engineering? >> I I think it's really interesting, right? It's like we we're almost moving from like these like self-contained prompts. Yeah. couple years ago to now this like context engineering to you think about like the gold and the high level you know really almost containing all of the knowledge work in one place and I think once we have AI that are as capable as a remote human then you really need to start thinking about well what would a remote human coworker need in order to be effective and you know you see this all the time where it's like someone wasn't there for the >> Davy standup or whatever and now they're confused they don't know what's going on

14:40they're on the wrong thing, right? Like all that happens for humans who have a lot of context. They're, you know, very plugged in. And so I think that we need as much sort of thought and attention to what does the AI need as we do for for human co-workers. >> W and so just tapping into that if it's as good as a remote worker, I get a lot of questions. Do you imagine that there's going to be like an AI workforce or we're going to name all these AI agents like Derek, John, whatever. Do you think that we're going to have every single employee have a suite of agents? Do you think that there's management of that or is it just they're working on their behalf? Like what is that structure? What are you imagining the AI

Managing fleets of agents

15:16first enterprise with agents looking like? >> Well, to some extent all of the above. To some extent we don't know. Like I think that AI presents a new opportunity to just like work in a totally different way from what we can imagine. We're seeing a little bit of it where it's like okay you take CEX, you ask it to do things, you sit there staring at your terminal. Okay, that starts to get boring. So you can run five of your codeexes, right? But then it's like well actually managing five codex's cotices uh cotices uh is like it's just not very scalable. It's not very fun. So you're like okay I want really good interfaces for actually managing the agents. Um so that makes sense. And so will they have

15:50persistent personalities and memory? Well in some ways it makes sense but in some ways it's almost a waste of the possibilities of an of an AI, right? Because for AIs you can control so many aspects of them that are difficult to control for humans, right? You can control what do they remember, right? What information do they still have access to? Um you can you can sort of fork them. You can um make decisions about how much computational power they get. And so that there is some amount of threaded identity that I think makes sense, but it also feels like it's kind of limiting to just think of it as okay, this is like one agent who, you know, did all this stuff yesterday and is doing more stuff now. Uh so I think that

16:26I'll be very interested to see how we build around all these new possibilities. Um, but I can tell you just my main observation from 10 years of working in this field is that the answer will be more surprising than any of us can anticipate. >> And how do you think we should deal with mass surprise? Like how are we ought to maintain sanity through this? >> Well, honestly, I think that a lot of this starts with small scale experimentation. Like you really got to try it out because hearing the words, describing it, thinking about it is so different from just seeing it. And chatting was a very surprising thing but now feels very natural right and and I think that the fact that we got to try out chatting when it was still 3.5 right

Forking, Sora and collaboration

17:08it's like that model compared to >> you know like the 20 billion parameter model today right it's like night and day how much better today's stuff is versus 3.5 but that was a moment where the world got to learn and got to adapt and so for me I I think we just need to keep doing that >> it's so interesting just how you were talking about forking because when people are looking at the new Sora app and then they think about it as just entertainment, I'm like, you're looking at it completely wrong. Like the fact that one person can create X and that 50 or 50,000 people can fork off of that and continue to create like why is that limited to art or to video like you should be thinking about code like that,

17:48images like that, processes like that? Like I really do think that there is a infinite amount of forking. Yes. and that kind of finding those seeds are going to be even more important inside the enterprise. >> I think it's a I think it's a great perspective. I remember when we first were looking at GPD3 thinking about well how will this change things? What can people use it for? um one of my co-workers named Yakob our chief scientist uh and his collaborator Shimone were saying well I think that there's going to be a new way of writing right new way of collaborating right you could imagine that you have like you know today you kind of have these sort of single documents that get produced like Wikipedia in a collaborative

18:25fashion but what if you can easily have this AI collaborator who's like swapping different parts of it and we haven't seen that yet um but I think we're starting to see it with Sora which is super cool >> like canvas was almost that a little bit It was more human with AI and I was hoping, you know, for more collaboration, but then all of a sudden we're seeing that in a multimodal way. It's really exciting stuff. >> It is. And I think part of the reason we didn't see it as much with text is honestly because the models just have not been smart enough, >> right? It's like if the models were really coming up with really clever stuff that's like kind of funny or that's like really insightful that adds

18:57a lot of value. You could imagine these remixes starting to come to life there. So, I think we will see it. I just think the models aren't aren't there yet. >> Wow. And when do you think they will be? two years, one year, three years, something like that. But the the predicting the exact timeline is is the difficult part. Um, but I think it's like very clear it's in the possibility space. >> Yeah, there's so much that you've shared. You've shared proactive AI, vertical AI, the importance of just getting your hands in there and actually building, the importance of figuring out what everyone else in your org is doing, right? What processes can we actually change? Lean teams and how they should function, collaborating with patients.

19:33We've covered so much. Do you have one piece of advice for every business leader watching this? And before you answer, some of them are incredibly excited about AI. They are using it every single day. Their kid is building bedtime stories every single night, right? And then you've got some that are like, "Yeah, I use it every so often. I feel like I'm not even using it to the full extent." And then you've got some that are still hesitant, which I still can't believe that. But you've got those three categories. How are you trying to inspire and lift up the business space? What's one piece of, you know, advice, one final thought that you would share with those folks? >> Well, I think perspective is the most

Advice for business leaders

20:12important thing here. >> And the way I think about AI is it's really about making computers useful, right? And you think about computers back in like I don't know 1950 or 1960, right? Compared to 1980s, compared to a modern laptop and you just think about the internet, right? Just so you think about how much more useful these computers are than what we had. Um, and how much if your workforce was just to lose access to all computers right now, like there's no way you'd be competitive. There's no way that your business would continue. >> And I think we're moving to a world very rapidly where the same will be true for AI adoption, right? Because AI is about imbuing this this sort of human amplifier, this amplifier of

20:54intelligence into the computer, bringing it much closer to you to then be able to carry out your intent and do, you know, whatever whatever business function people want. So, I think that having this perspective of both how important AI will be to your company and that it's an exciting thing, right, that there's an opportunity here that no one else has tapped, no one else in your market has tapped. And so I think that those who really lean in and figure out how it can add value um and you know that we try to add a bunch of you you can look at things like our agent kit and and these other tools that we bring uh bring to bear that let you add in the guard rails control what information is in there.

21:27Think about enterprise controls all of those things like we're here to help. >> Unbelievable. And also I just want to say the guard rail guard rail release. Love that. Like I'm very excited to show that off. Let me just ask a followup and then and then we'll wrap. Um, we talked earlier about competitive advantage. You had mentioned speed of iteration and then you're also saying, listen, we're going to be at a point in time where if you haven't adopted this, not only have you lost your competitive advantage, you don't exist anymore. So, what does competitive advantage look like? And maybe it is speed of iteration. Maybe it's just constantly changing and adapting your perspective. Do you have a

Competitive advantage and domain expertise

22:05perspective on that? Yeah, I do think that there is something that is very hard for AI to magically replace, right, of like real domain knowledge and we see it is that there is a G in the AGI in general where the AI just gets better at all these different domains at once. But to really figure out how do you make AI useful in education, there's all these different stakeholders, right? There's the parent, there's the teacher, there's the student, you need to think about if the student is using one of these tools, like what information is shared with with the teacher and the parent, all those things. And that's something where having these relationships, having a track record, really figuring out how to

22:40get good feedback, those are very difficult to have the technology itself be the main, at least at the current moment, the main the main driver for. And so I think that there is something that's very sticky about these deep domain uh expertise. And uh that's something that I think you can really lean into. >> Sounds like the people who have loved people watching are going to be at a really strong advantage. people that are just able to witness a market and are able to then adapt ability to grab feedback and jump on. >> I think that's a great way to put it and I think there's something, you know, we're building AI but it's really about the humans. >> Yeah, I love that. Thank you so much for sharing and again if you guys have any

23:16questions, drop them below and I will try and jump on them. But Greg, thank you so much for the chat. Really appreciate it. >> Thank you. It's been a lot of fun.