Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

github.com

Post reply on HN
Print view View original

21–30 of 61 posts

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#21

Post by brucethemoose2 »

minimaxir wrote: post #13

A robust 1.1B model compared to a 7B model would be strongly appreciated. The bottleneck of Llama 2 7B is that inference latency is still infeasible for Production use cases unless you have a good supply of expensive A100; dropping it by an order of magnitude and letting it run on other cloud GPUs will open new opportunities.

> The bottleneck of Llama 2 7B is that inference latency is still infeasible for Production use cases unless you have a good supply of expensive A100

?? A 3060 or a slightly bigger AMD/Intel GPU can stream llama 7B about as fast as someone can read, if not faster. A somewhat bigger consumer GPU can batch it and serve dozens of users.

I use 13B finetunes on my 2020 14" laptop all the time, with 6GB of VRAM and 16GB of CPU RAM.

I have seen many people on HN say this, and I can't help but wonder why the optimized, quantized llama implementations are flying under the radar.

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#22

Post by cypress66 »

minimaxir wrote: post #13

A robust 1.1B model compared to a 7B model would be strongly appreciated. The bottleneck of Llama 2 7B is that inference latency is still infeasible for Production use cases unless you have a good supply of expensive A100; dropping it by an order of magnitude and letting it run on other cloud GPUs will open new opportunities.

1.1B with 3T tokens will never be comparable to 7B with 2T tokens.

And I'm not sure what you mean by inference latency being infeasible. Most people using thsss models at home don't even bother with the 7B and go straight to 13B because it's easy to run too and much smarter. And any cloud gpu can run 13B.

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#23

Post by imjonse »

From the FAQ:

' Why would pretraining a 1.1B model for so long make sense? Doesn't it contradict the Chinchilla Scaling Law?

Above is the training loss curve taken from the Llama 2 paper. Here I quote from that paper: "We observe that after pretraining on 2T Tokens, the models still did not show any sign of saturation". That is why we believe pretraining a 1.1B model for 3T tokens is a reasonable thing to do. Even if the loss curve does not go down eventually, we can still study the phenomenon of saturation and learn something from it.'

It is something I have been wondering about: why did Meta not keep the training process going on while the loss curves seemed to go down? Could they conceivably release a Llama 2.1 being checkpoints taken a month after 2.0 was 'cut'? Maybe the expected gain is too small compared to what can be gained with fine/instruct tuning afterward anyway?

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#24

Post by cypress66 »

GaggiX wrote: post #3

>It means you can train a chinchilla-optimal TinyLlama (1.1B param, 22B tokens) in 32 hours with 8 A100. They are training the model on 3000/22=136 times the value of the chinchilla scale. It will be interesting to see how much it will improve after way beyond this value.

It's a bit amusing how people treat chinchilla scaling laws as a law of nature, when it's just about a certain architecture and dataset.

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#25

Post by naillo »

imjonse wrote: post #23

From the FAQ: ' Why would pretraining a 1.1B model for so long make sense? Doesn't it contradict the Chinchilla Scaling Law? Above is the training loss curve taken from the Llama 2 paper. Here I quote from that paper: "We observe that after pretraining on 2T Tokens, the models still did not show any sign of saturation". That is why we believe pretraining a 1.1B model for 3T tokens is a reasonable thing to do. Even if…

The most plausible explanation I've seen (other than the carmack 'sudden grokking' beyond the cutoff idea) is that they're planning to release llama3 sooner than later with some arcitecture changes for even better performance, so it makes sense to dedicate resources there instead.

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#26

Post by rodonn »

Havoc wrote: post #8

What does “pretrain” mean in this context? It sounds like normal training

GPT stands for Generative Pre-trained Transformer.

The "main" training step using huge amounts of inputs is called pre-training. The idea is that after that pre-training, you might fine tune the model for your specific use case.

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#27

Post by RC_ITR »

ljlolel wrote: post #18
RC_ITR wrote: post #14

Not to be a downer, but wasn’t one of OpenAI’s earliest discoveries that training small models on huge datasets leads to over-fitting? It’s my understanding that the entire race to ever-more parameters was driven by that.

Learn about the magic of double descent

https://openai.com/research/deep-double-descent

Yeah, the line keeps going down as the model gets bigger. What's your point? That there's a hump in the middle?

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#29

Post by jofi1 »

imjonse wrote: post #23

From the FAQ: ' Why would pretraining a 1.1B model for so long make sense? Doesn't it contradict the Chinchilla Scaling Law? Above is the training loss curve taken from the Llama 2 paper. Here I quote from that paper: "We observe that after pretraining on 2T Tokens, the models still did not show any sign of saturation". That is why we believe pretraining a 1.1B model for 3T tokens is a reasonable thing to do. Even if…

> It is something I have been wondering about: why did Meta not keep the training process going on while the loss curves seemed to go down? Could they conceivably release a Llama 2.1 being checkpoints taken a month after 2.0 was 'cut'? Maybe the expected gain is too small compared to what can be gained with fine/instruct tuning afterward anyway?

Because choosing the LR decay requires knowing the # of steps in advance. LR is too small after the 2T tokens, and changing it afterwards doesn't tend to help.

https://twitter.com/sherjilozair/status/1687837844729966592

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#30

Post by snovv_crash »

brucethemoose2 wrote: post #21
minimaxir wrote: post #13

A robust 1.1B model compared to a 7B model would be strongly appreciated. The bottleneck of Llama 2 7B is that inference latency is still infeasible for Production use cases unless you have a good supply of expensive A100; dropping it by an order of magnitude and letting it run on other cloud GPUs will open new opportunities.

> The bottleneck of Llama 2 7B is that inference latency is still infeasible for Production use cases unless you have a good supply of expensive A100 ?? A 3060 or a slightly bigger AMD/Intel GPU can stream llama 7B about as fast as someone can read, if not faster. A somewhat bigger consumer GPU can batch it and serve dozens of users. I use 13B finetunes on my 2020 14" laptop all the time, with 6GB of VRAM and 16GB of…

That's still wildly too expensive if you want to make a profitable service that is scalable beyond VC capital injections.
Post reply on HN