WEBVTT

00:00:00.320 --> 00:00:03.237
A million tokens is a lot. That's
that's, you know, that can be days of

00:00:03.320 --> 00:00:07.277
human learning, right? You know,
if you think about the model, you know,

00:00:07.360 --> 00:00:10.677
you know, kind of read reading reading
a million words, you know, it it you know,

00:00:10.760 --> 00:00:14.277
it takes me how long would it take me
to read a million? I mean, you know, like

00:00:14.360 --> 00:00:19.117
days or weeks at least. Um so,
you have these two things and and I

00:00:19.200 --> 00:00:22.957
think these two these two things within
the existing paradigm may just be enough

00:00:23.040 --> 00:00:26.397
to get you the countries of genesis in
a data center. I don't know for sure, but

00:00:26.480 --> 00:00:27.957
I think they're going to get you a large

00:00:28.040 --> 00:00:30.917
fraction of it. There may be gaps, but I

00:00:31.000 --> 00:00:35.397
I certainly think just as things are,
this I believe is enough to generate

00:00:35.480 --> 00:00:38.677
trillions of dollars of revenue.
That's one. That's all one.

00:00:38.760 --> 00:00:42.197
Two is this idea of continual learning,

00:00:42.280 --> 00:00:44.637
this idea of a single model learning on

00:00:44.720 --> 00:00:47.317
the job. Um I think we're working on

00:00:47.400 --> 00:00:49.357
that, too, and I think there's a good

00:00:49.440 --> 00:00:51.997
chance that in the next year or two we

00:00:52.080 --> 00:00:55.677
also make we also solve that. Um I I

00:00:55.760 --> 00:00:58.397
again, I I I I you know, I think you get

00:00:58.480 --> 00:01:00.717
most of the way there without it. I

00:01:00.800 --> 00:01:05.557
think the trillions of dollars of of,
you know, the the I think the trillions

00:01:05.640 --> 00:01:09.237
of dollars a year market may be all
of the national security implications and

00:01:09.320 --> 00:01:10.637
the safety implications that I wrote

00:01:10.720 --> 00:01:12.637
about in adolescence of technology can

00:01:12.720 --> 00:01:16.477
happen without it, but I I I also think

00:01:16.560 --> 00:01:21.237
we and I imagine others are working on it,
and I think there's a good chance

00:01:21.320 --> 00:01:25.237
that that, you know, that we get there
within the next year or two. There are a

00:01:25.320 --> 00:01:29.277
bunch of ideas. I won't go into all
of them in detail, but um you know, one is

00:01:29.360 --> 00:01:33.157
just make the context longer.
There's There's nothing preventing longer

00:01:33.240 --> 00:01:36.957
context from working. You just have
to train at longer context and then learn

00:01:37.040 --> 00:01:40.677
to to serve them at inference, and both
of those are engineering problems that

00:01:40.760 --> 00:01:43.477
we are working on and that I would
assume others are working on as well.

00:01:43.560 --> 00:01:45.757
>> Yeah, so this context
length increase It

00:01:45.840 --> 00:01:49.837
seemed like there was a period from 2020
to 2023 where from GBD3 to GBD4 Turbo

00:01:49.920 --> 00:01:53.677
there was an increase from like
2,000 context length to 128K.

00:01:53.760 --> 00:01:57.157
I feel like for the next for the two-ish
years since then, we've been in the

00:01:57.240 --> 00:02:00.517
same-ish ballpark. >> Yeah.
And when model context lines get

00:02:00.600 --> 00:02:05.917
much longer than that, people report
qualitative degradation in the ability

00:02:06.000 --> 00:02:09.037
of the model to consider
that full context. Um

00:02:09.120 --> 00:02:11.797
So, I'm curious what you're internally
seeing that makes you think like, "Oh,

00:02:11.880 --> 00:02:14.997
10 million contacts, 100 million
contacts." To get human-like 6-month

00:02:15.080 --> 00:02:16.477
learning, billion billion contacts.

00:02:16.560 --> 00:02:20.077
>> a research problem. This is a This
is an engineering and inference problem,

00:02:20.160 --> 00:02:21.837
right? If you want to serve long

00:02:21.920 --> 00:02:23.837
context, you have to like store your

00:02:23.920 --> 00:02:27.117
entire KV cache. You have to, you know,

00:02:27.200 --> 00:02:30.957
uh you know, it's it's it's it's
difficult to store all of the memory in

00:02:31.040 --> 00:02:34.877
the GPUs, to juggle the memory around.
I don't even know the detail, you know, at

00:02:34.960 --> 00:02:37.117
at at this point this
is at a level of detail

00:02:37.200 --> 00:02:40.437
that that that that I'm no longer able
to follow, although, you know, I I knew

00:02:40.520 --> 00:02:43.837
it in the GPT-3 era of like, you know,
these are the weights, these are the

00:02:43.920 --> 00:02:48.117
activations you have to store. Um but,
you know, you know, these days the whole

00:02:48.200 --> 00:02:50.197
thing has flipped cuz we have MoE models

00:02:50.280 --> 00:02:55.437
and and and kind of all of that. But,
um and and this degradation you're talking

00:02:55.520 --> 00:02:59.317
about, like again, without getting too
specific, like a question I would ask is

00:02:59.400 --> 00:03:03.197
like there's two things. There's the
context length you train at, and there's

00:03:03.280 --> 00:03:07.357
a context length that you serve at.
If you train at a small context length and

00:03:07.440 --> 00:03:10.117
then try to serve at a long context
length, like maybe you get these

00:03:10.200 --> 00:03:13.517
degradations. It's better than nothing,
you might still offer it, but you get

00:03:13.600 --> 00:03:16.597
these degradations. And maybe it's
harder to train at a long context

00:03:16.680 --> 00:03:18.597
length. Yeah, I so, you know,
there's there's a lot.

00:03:18.680 --> 00:03:20.117
>> If you enjoyed this clip, you can watch

00:03:20.200 --> 00:03:25.760
the full episode here and subscribe
for more clips. Thanks.