Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TinyLlama: An Open-Source Small Language Model

arxiv.org

Post reply on HN
Print view View original

31–40 of 46 posts

Re: TinyLlama: An Open-Source Small Language Model

#31

Post by kiratp »

WhitneyLand wrote: post #22
minimaxir wrote: post #13

It was fun to follow the public TinyLlama loss curves in near real-time, although it showed that it can be frustrating since the loss curves barely moved down even after an extra trillion tokens: https://wandb.ai/lance777/lightning_logs/reports/metric-trai... (note the log-scaled X-axis) But they did move down and that's what's important. There should probably be more aggressive learning rate annealing for models try…

More aggressive learning rate? Wouldn’t we need more information on why they decided to stop training at this point to conclude that? My current understanding of the story is, to recap: - First the game was increase model size massively - For example GPT3 had 175B parameters, but less than 0.5T tokens of training data - Then Chinchilla showed for a given compute budget we can scale better by increasing training data…

> For any model, the loss curve going down could mean it’s learning, or could mean it’s overfitting, we don’t know which without looking at validation loss, which is like a second set of test data the model hasn’t seen before.

You want to look at validation accuracy.

Re: TinyLlama: An Open-Source Small Language Model

#32

Post by minimaxir »

zozbot234 wrote: post #28
WhitneyLand wrote: post #22

Earlier quoted context omitted.

More aggressive learning rate? Wouldn’t we need more information on why they decided to stop training at this point to conclude that? My current understanding of the story is, to recap: - First the game was increase model size massively - For example GPT3 had 175B parameters, but less than 0.5T tokens of training data - Then Chinchilla showed for a given compute budget we can scale better by increasing training data…

Overfitting is quite unlikely with a smaller model though. Model parsimony provides a kind of regularization "for free", in fact with the extra benefit of saving on compute costs.

The dirty secret behind modern selfsupervised training is that no one cares about a test/validation dataset anymore.

Re: TinyLlama: An Open-Source Small Language Model

#33

Post by minimaxir »

kiratp wrote: post #31
WhitneyLand wrote: post #22

Earlier quoted context omitted.

More aggressive learning rate? Wouldn’t we need more information on why they decided to stop training at this point to conclude that? My current understanding of the story is, to recap: - First the game was increase model size massively - For example GPT3 had 175B parameters, but less than 0.5T tokens of training data - Then Chinchilla showed for a given compute budget we can scale better by increasing training data…

> For any model, the loss curve going down could mean it’s learning, or could mean it’s overfitting, we don’t know which without looking at validation loss, which is like a second set of test data the model hasn’t seen before. You want to look at validation accuracy.

Accuracy is a bad metric for LLMs, especially since a LLM tokenizer can have thousands of "classes": 32,000 in the case of TinyLlama.

Re: TinyLlama: An Open-Source Small Language Model

#35

Post by viraptor »

minimaxir wrote: post #24
WhitneyLand wrote: post #22

Earlier quoted context omitted.

More aggressive learning rate? Wouldn’t we need more information on why they decided to stop training at this point to conclude that? My current understanding of the story is, to recap: - First the game was increase model size massively - For example GPT3 had 175B parameters, but less than 0.5T tokens of training data - Then Chinchilla showed for a given compute budget we can scale better by increasing training data…

> Wouldn’t we need more information on why they decided to stop training at this point to conclude that? The experiment was fixed at 3 epochs on 1T tokens, they didn't decide to "stop" at a given criterion. > we don’t know which without looking at validation loss, which is like a second set of test data the model hasn’t seen before. The data I linked shows the validation loss, which has the same behavior as the train…

I'd love to see someone go for another few epochs in the future. Two of the benchmarks got a significant jump almost at the end of training. I wonder if there's a chance for more of that - looks like an interesting effect on its own.

Re: TinyLlama: An Open-Source Small Language Model

#36

Post by dadeyemi »

minimaxir wrote: post #32
zozbot234 wrote: post #28

Earlier quoted context omitted.

Overfitting is quite unlikely with a smaller model though. Model parsimony provides a kind of regularization "for free", in fact with the extra benefit of saving on compute costs.

The dirty secret behind modern selfsupervised training is that no one cares about a test/validation dataset anymore.

does overfitting even matter if your dataset is large enough?

Re: TinyLlama: An Open-Source Small Language Model

#38

Post by WhitneyLand »

dadeyemi wrote: post #36
minimaxir wrote: post #32

Earlier quoted context omitted.

The dirty secret behind modern selfsupervised training is that no one cares about a test/validation dataset anymore.

does overfitting even matter if your dataset is large enough?

I think a lot of it depends on what you mean by “large enough”.

In principle, a data set could be infinitely large in size, but not cover little edge cases here and there due to repetition. So you might be OK if you had infinite size and infinite diversity.

Even if you had very large finite data, let’s say all language ever conceived by mankind… The second you finish training, what your overfit model knows is locked in.

The world as we know it would continue to generate vast amounts of new data that you might not be able to generalize to.

Re: TinyLlama: An Open-Source Small Language Model

#39

Post by smpanaro »

viraptor wrote: post #35
minimaxir wrote: post #24

Earlier quoted context omitted.

> Wouldn’t we need more information on why they decided to stop training at this point to conclude that? The experiment was fixed at 3 epochs on 1T tokens, they didn't decide to "stop" at a given criterion. > we don’t know which without looking at validation loss, which is like a second set of test data the model hasn’t seen before. The data I linked shows the validation loss, which has the same behavior as the train…

I'd love to see someone go for another few epochs in the future. Two of the benchmarks got a significant jump almost at the end of training. I wonder if there's a chance for more of that - looks like an interesting effect on its own.

The jump was due to them fixing a bug. There’s a footnote about it on the bottom of page 5.

In the Discord, they mentioned a TinyLLaMa v2, presumably that would have this bug (and another bug, footnote page 4) fixed.

Re: TinyLlama: An Open-Source Small Language Model

#40

Post by TheCoreh »

From the GitHub repo Readme:

> we can achieve this within a span of "just" 90 days using 16 A100-40G GPUs

I knew the computational power required to train LLMs was absurd, but seeing the figures of larger networks (which are just too large to intuitively understand) it didn't really register. With this one I could actually imagine the 16 machines with A100 GPUs sitting on a server room running at full blast for 90 days so it was more tangible... And now to think about the larger ones is kinda scary

Edit: Did the math and just the GPUs (at 250W each) consumed around 8.64 MWh, which is at the same ballpark of the power consumption of the average US home in one year (10.5MWh)

Post reply on HN