Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Laws, Carefully

lilianweng.github.io

Post reply on HN
Print view View original

11–20 of 22 posts

Re: Scaling Laws, Carefully

#11

Post by gdiamos »

When I first saw scaling laws in that deep speech experiment notebook, I didn’t believe it could be real. I was worried for months that we made a mistake, or that it only worked for that one dataset.

I started to believe it after we (Joel Hestness in particular) reproduced it in so many experiments in “scaling is predictable empirically”.

The OpenAI work replicated it in a completely different environment, and at that point I was sure it was real.

Sometimes people ask me why I was so surprised by it. Prior work like Banko and Brill and the unreasonable effectiveness of data argued for more data. ML theory had similar models for toy problems, eg coin flips.

At the time I thought deep learning was supposed to be complex. Speech and language datasets seemed much more complex than toy problems. Optimization of deep transformers was complex.

The idea that it was possible for the whole thing to be governed by a 3 term equation seemed too simple. The implication was that it was simple to manufacture intelligence.

Ten years later, I still think it is still the most interesting observation I have seen. We are still learning what it looks like to live in a world where it is possible to manufacture intelligence.

Re: Scaling Laws, Carefully

#12

Post by nok22kon »

gdiamos wrote: post #11

When I first saw scaling laws in that deep speech experiment notebook, I didn’t believe it could be real. I was worried for months that we made a mistake, or that it only worked for that one dataset. I started to believe it after we (Joel Hestness in particular) reproduced it in so many experiments in “scaling is predictable empirically”. The OpenAI work replicated it in a completely different environment, and at tha…

the scaling laws work within a "generation". but what about across them?

GPT-3 was 175B, models like Gemma4 with 31B vastly outperform it, so there is more to it

as Karpathy noted, the initial GPTs were trained on complete garbage (literally, the average document from the Common Crawl is random nonsense), yet they worked. now we can use present LLMs to curate the data for the next generation

Re: Scaling Laws, Carefully

#13

Post by hnuser123456 »

nok22kon wrote: post #12
gdiamos wrote: post #11

When I first saw scaling laws in that deep speech experiment notebook, I didn’t believe it could be real. I was worried for months that we made a mistake, or that it only worked for that one dataset. I started to believe it after we (Joel Hestness in particular) reproduced it in so many experiments in “scaling is predictable empirically”. The OpenAI work replicated it in a completely different environment, and at tha…

the scaling laws work within a "generation". but what about across them? GPT-3 was 175B, models like Gemma4 with 31B vastly outperform it, so there is more to it as Karpathy noted, the initial GPTs were trained on complete garbage (literally, the average document from the Common Crawl is random nonsense), yet they worked. now we can use present LLMs to curate the data for the next generation

I dunno if you've seen the subreddit, "Sub Simulator GPT2", but I found it around 2020-2021. It seemed to contain GPT2-style models trained/finetuned on several popular subreddits, talking to each other as stereotypical regulars of each sub would. Most of the replies were fairly coherent and somewhat related to the "thread topic", but of course even GPT3.5 would make all of them look beyond drunk only a few years later. I already had a vague understanding of neural networks and the advances in image processing at the time, but couldn't have predicted where we are now. I wonder what it'll look like in a few more years as we continue how to learn how to make this capability useful and reliable, and hopefully sometimes keep finding additional conscionable entertainment and educational applications.

Re: Scaling Laws, Carefully

#14

Post by gdiamos »

nok22kon wrote: post #12
gdiamos wrote: post #11

When I first saw scaling laws in that deep speech experiment notebook, I didn’t believe it could be real. I was worried for months that we made a mistake, or that it only worked for that one dataset. I started to believe it after we (Joel Hestness in particular) reproduced it in so many experiments in “scaling is predictable empirically”. The OpenAI work replicated it in a completely different environment, and at tha…

the scaling laws work within a "generation". but what about across them? GPT-3 was 175B, models like Gemma4 with 31B vastly outperform it, so there is more to it as Karpathy noted, the initial GPTs were trained on complete garbage (literally, the average document from the Common Crawl is random nonsense), yet they worked. now we can use present LLMs to curate the data for the next generation

Scaling laws assume the error metric and data distribution.

There is a lot of follow on work that explains what happens as you change them, e.g. Scaling Laws for Transfer - https://arxiv.org/pdf/2102.01293

I think it’s fortunate that transfer works in a similar way.

Common crawl (and Reddit, stack overflow, etc but not 4chan) was much easier to get access to at the time than using mechanical Turk.

There is certainly room for more work. There were many papers on scaling laws in NeurIPS this year.

Re: Scaling Laws, Carefully

#16

Post by beyonddream »

ekelsen wrote: post #15

Jeff Dean has a paper in 2007 that has proto scaling law plots for ngram language models. https://aclanthology.org/anthology-files/anthology-files/pdf...

Nice find! The final paragraph of the Conclusion is amazingly prescient!

"Significantly, we found that translation quality as indicated by BLEU score continues to improve with increasing language model size, at even the largest sizes considered. This finding underscores the value of being able to train and apply very large language models, and suggests that further performance gains may be had by pursuing this direction further."

Re: Scaling Laws, Carefully

#17

Post by openclawclub »

aspenmartin wrote: post #2

I really wish more people skeptical of AI capabilities would read about scaling laws -- Lilian is always so marvelous at giving a deep overview of the technical side but the whole point of this is: there are scaling laws, and they hold and continue to hold. This is such a huge basis for the predictions about AI capabilities for the past like 5 years.

[flagged]

Re: Scaling Laws, Carefully

#18

Post by 317070 »

FromTheFirstIn wrote: post #3
aspenmartin wrote: post #2

I really wish more people skeptical of AI capabilities would read about scaling laws -- Lilian is always so marvelous at giving a deep overview of the technical side but the whole point of this is: there are scaling laws, and they hold and continue to hold. This is such a huge basis for the predictions about AI capabilities for the past like 5 years.

And sitting right next to the data and compute factors in every cross entropy loss equation is the entropy of the language, which is just a fixed constant. There’s such a hard cap on cross entropy loss training and I never hear it come up!

Right, and what happens at that limit is most exciting! A model that has a cross entropy at that limit for a data stream of text, produces a stream of text that is both theoretically and practically indistinguishable from the original stream.

And so if the datastream has been produced by something intelligent, the resulting model is indistinguishable from that intelligence. That is the whole compression idea behind artificial intelligence.

The limit is not a bug, it's a feature!

Re: Scaling Laws, Carefully

#19

Post by FromTheFirstIn »

317070 wrote: post #18
FromTheFirstIn wrote: post #3

Earlier quoted context omitted.

And sitting right next to the data and compute factors in every cross entropy loss equation is the entropy of the language, which is just a fixed constant. There’s such a hard cap on cross entropy loss training and I never hear it come up!

Right, and what happens at that limit is most exciting! A model that has a cross entropy at that limit for a data stream of text, produces a stream of text that is both theoretically and practically indistinguishable from the original stream. And so if the datastream has been produced by something intelligent, the resulting model is indistinguishable from that intelligence. That is the whole compression idea behind a…

It’s a bit like saying copy and paste must be intelligent because its output is indistinguishable from intelligent output. Reproduction of intelligence has never been equivalent to intelligence- that’s why we look down on copying, plagiarism or derivative work.

Re: Scaling Laws, Carefully

#20

Post by 317070 »

FromTheFirstIn wrote: post #19
317070 wrote: post #18

Earlier quoted context omitted.

Right, and what happens at that limit is most exciting! A model that has a cross entropy at that limit for a data stream of text, produces a stream of text that is both theoretically and practically indistinguishable from the original stream. And so if the datastream has been produced by something intelligent, the resulting model is indistinguishable from that intelligence. That is the whole compression idea behind a…

It’s a bit like saying copy and paste must be intelligent because its output is indistinguishable from intelligent output. Reproduction of intelligence has never been equivalent to intelligence- that’s why we look down on copying, plagiarism or derivative work.

Well if it was the data that was copied, nobody would be using LLMs. The data-generating process is the thing that would be copied.

That's why we do appreciate the nth artist making classical or techno music.

Post reply on HN