1
00:00:00,360 --> 00:00:04,157
It Open AI has multiple stacks to the
business, one of which is compute, and

2
00:00:04,240 --> 00:00:09,197
you guys have been very aggressive, very
aggressive on securing compute. Why is

3
00:00:09,280 --> 00:00:12,957
that? >> Well, in many ways,
we have a very simple business.

4
00:00:13,040 --> 00:00:16,957
We buy, rent, build compute,

5
00:00:17,040 --> 00:00:21,197
and we resell it at a margin.
That's it. As long as the margin's

6
00:00:21,280 --> 00:00:25,517
positive, then you want to scale
it because the demand for

7
00:00:25,600 --> 00:00:30,797
solving problems, the demand
for intelligence, that's unlimited.

8
00:00:30,880 --> 00:00:33,997
And the AIs that we have right now
really are able to rise to the challenge

9
00:00:34,080 --> 00:00:36,957
of effectively any kind of problem
that you want to throw at them.

10
00:00:37,040 --> 00:00:38,357
>> Do you have enough compute?

11
00:00:38,440 --> 00:00:41,317
>> No. >> Really? >> Yeah, definitely not.

12
00:00:41,400 --> 00:00:43,757
>> was just with Matt Garman, and he says

13
00:00:43,840 --> 00:00:48,957
the GPU compute availability in 2026
rounds to zero. Don't you guys

14
00:00:49,040 --> 00:00:52,117
have all of it? >> I mean,
we have we we would love more.

15
00:00:52,200 --> 00:00:56,437
We're we're constantly out there hunting
for more, honestly. And I'll tell you

16
00:00:56,520 --> 00:00:59,277
like when we first launched
when we launched ChatGPT,

17
00:00:59,360 --> 00:01:02,757
I remember being on a call with my team,
and they were like, "All right, how much

18
00:01:02,840 --> 00:01:06,477
compute should we buy?"
And I said, "All of it."

19
00:01:06,560 --> 00:01:08,517
And they're like, "No, no, no, we're
seriously like like come on, how much

20
00:01:08,600 --> 00:01:12,277
should we buy?" I'm like, "No matter how
fast we try to ramp compute, I guarantee

21
00:01:12,360 --> 00:01:14,877
we're not going to be able to keep up
with demand, and that has been true ever

22
00:01:14,960 --> 00:01:19,037
since." >> Um that's that's
fascinating. Moving up

23
00:01:19,120 --> 00:01:23,917
from compute since I don't know if
much of this audience can help you with

24
00:01:24,000 --> 00:01:27,237
securing more compute because
most of them are founders of

25
00:01:27,320 --> 00:01:31,077
startups.
About architecture and scaling laws,

26
00:01:31,160 --> 00:01:34,917
what are the what are
the where are we in the scaling

27
00:01:35,000 --> 00:01:37,397
laws? Are they still doubling each year?

28
00:01:37,480 --> 00:01:42,477
Are you changing architecture? What's
what is what are you guys pushing on the

29
00:01:42,560 --> 00:01:44,277
frontier on the research side?

30
00:01:44,360 --> 00:01:48,597
>> Well, I would say first of all,
the scaling laws are a deep and very

31
00:01:48,680 --> 00:01:50,917
beautiful mystery, right? They feel

32
00:01:51,000 --> 00:01:52,997
deeply fundamental. It's like the

33
00:01:53,080 --> 00:01:58,437
scientific truth that just like you
think about physics and, you know,

34
00:01:58,520 --> 00:02:00,757
Newton's laws and things like that,
they're somehow this truth of the

35
00:02:00,840 --> 00:02:04,957
universe. And they're
empirical. Like we don't

36
00:02:05,040 --> 00:02:09,037
necessarily have all the theory
to explain exactly why it works, but to me

37
00:02:09,120 --> 00:02:12,277
the most beautiful thing
is that neural networks

38
00:02:12,360 --> 00:02:14,917
were really designed like in the 1940s

39
00:02:15,000 --> 00:02:16,797
before there were computers.

40
00:02:16,880 --> 00:02:19,637
And somehow we've been able to take the

41
00:02:19,720 --> 00:02:24,117
exact ideas that were developed back
then and apply increasing amounts of

42
00:02:24,200 --> 00:02:28,197
computation. And as you pour more
compute into the models, they get

43
00:02:28,280 --> 00:02:32,077
correspondingly more capable.
And it just keeps going. There's no wall.

44
00:02:32,160 --> 00:02:34,837
And that's I think
that's a beautiful thing.

45
00:02:34,920 --> 00:02:36,637
>> That's pretty beautiful. Are there more

46
00:02:36,720 --> 00:02:40,197
researched or more algorithms that are

47
00:02:40,280 --> 00:02:43,797
in the works that because you
know, in the past we had neural

48
00:02:43,880 --> 00:02:47,477
networks to your point in 1940s, but we
couldn't we didn't have the compute for

49
00:02:47,560 --> 00:02:50,717
it. Now that we have the compute for it,
you just or are we just pushing the same

50
00:02:50,800 --> 00:02:54,077
things or are there new architectures
and new ideas coming out?

51
00:02:54,160 --> 00:02:58,357
>> Yeah, so I would I would think of it as
we we absolutely have new ideas that are

52
00:02:58,440 --> 00:03:01,797
constantly powering what we do. It's
very simplified to say, well, let's take

53
00:03:01,880 --> 00:03:06,077
a neural network from the 1940s and you
know, put put it in a gigawatt data

54
00:03:06,160 --> 00:03:09,757
center, right? We have made tons
of innovations and we constantly are

55
00:03:09,840 --> 00:03:13,517
improving things. And sometimes these
are micro tweaks. Like you just realize

56
00:03:13,600 --> 00:03:17,277
that the way you've been formatting
data was not quite right and that can

57
00:03:17,360 --> 00:03:21,237
actually be a a very big deal. Sometimes
it's larger. You think about the shift

58
00:03:21,320 --> 00:03:24,597
from the LSTM to the transformer and I
don't think the transformer is, you

59
00:03:24,680 --> 00:03:27,517
know, like everyone's moved past
the transformer as described in the other

60
00:03:27,600 --> 00:03:30,797
2018 paper. So there's
there's constant innovation

61
00:03:30,880 --> 00:03:34,157
happening and I think of places that
have been perhaps the most invested in

62
00:03:34,240 --> 00:03:37,677
long-term research on how to improve
the architectures, how to improve the

63
00:03:37,760 --> 00:03:41,117
fundamental algorithms, and how to get
the paradigm shifts. I think OpenAI has

64
00:03:41,200 --> 00:03:43,877
been leading the pack there and that's
something we continue to invest in and I

65
00:03:43,960 --> 00:03:45,557
see lots of fruit on the horizon.

66
00:03:45,640 --> 00:03:49,757
>> Got it. And on the models,
um, does OpenAI have

67
00:03:49,840 --> 00:03:54,117
a formal definition for AGI?
Are we closed? Are we not close?

68
00:03:54,200 --> 00:03:58,917
Pat and Sonia published this thing
that we are at AGI functionally.

69
00:03:59,000 --> 00:04:00,837
Do you agree with that?
Do you not agree with that?

70
00:04:00,920 --> 00:04:04,477
>> Well, we do have a formal
definition, but to some extent

71
00:04:04,560 --> 00:04:08,277
I one thing I have learned is that
everyone has their own intuitions about

72
00:04:08,360 --> 00:04:12,837
what AGI is and maybe you can view it
as like according to my view of where we

73
00:04:12,920 --> 00:04:17,477
are. I think we're about 80% of the way
there and that we have models that are

74
00:04:17,560 --> 00:04:21,557
smart. They're very capable.
They're able to if you give

75
00:04:21,640 --> 00:04:24,157
>> Are they smarter than you?
>> I mean I they're certainly more capable

76
00:04:24,240 --> 00:04:28,197
than I am at writing software, right? If
you give it all the context, then yes, I

77
00:04:28,280 --> 00:04:33,157
I I think that they are they're just
so capable. It's it's really remarkable.

78
00:04:33,240 --> 00:04:39,720
Like does anyone here feel better
at writing software than GPD 5.4?

79
00:04:39,960 --> 00:04:42,360
>> Oh.

80
00:04:43,040 --> 00:04:47,837
>> All right, writing kernels. Um so
even there we're seeing massive gains from

81
00:04:47,920 --> 00:04:51,877
Exactly. We're and for some of our
internal results um that there we're

82
00:04:51,960 --> 00:04:55,517
really seeing if you
pour the right kinds of of

83
00:04:55,600 --> 00:04:58,477
you know, if you have the right setup
for your problem, then you're able to

84
00:04:58,560 --> 00:05:03,157
get really massive results out of very
low-level um even low-level tasks. And

85
00:05:03,240 --> 00:05:06,557
just to give you one example
of how things have been trending um

86
00:05:06,640 --> 00:05:10,757
one of my systems engineers also very
similar was like, "Hey, like I haven't

87
00:05:10,840 --> 00:05:12,277
been able to get value out of the models

88
00:05:12,360 --> 00:05:14,957
for GPD 5 or 5.1

89
00:05:15,040 --> 00:05:19,477
for 5.2 as well. For 5.3 he on a lark

90
00:05:19,560 --> 00:05:22,517
prepar- prepared this design document
for a very complicated systems

91
00:05:22,600 --> 00:05:24,437
optimization he was about to do.

92
00:05:24,520 --> 00:05:29,037
He handed it over to the model, went
to sleep waking up intending to like give

93
00:05:29,120 --> 00:05:31,357
this to his team to work
on for the next week.

94
00:05:31,440 --> 00:05:33,077
And when he woke up it was done.

95
00:05:33,160 --> 00:05:37,317
That the model had actually
implemented the initial spec, had seen

96
00:05:37,400 --> 00:05:40,557
that it was slow, had added
instrumentation, had actually run the

97
00:05:40,640 --> 00:05:44,317
code, used a profiler to figure out
where things were slow, and iterated

98
00:05:44,400 --> 00:05:45,757
multiple times until it gotten to an

99
00:05:45,840 --> 00:05:52,360
optimized result. And like like
that is incredible. That's where we are.