Couldn't immediately find it but who sponsors/pays for the compute?
TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens
31–40 of 61 posts
Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens
#32A robust 1.1B model compared to a 7B model would be strongly appreciated. The bottleneck of Llama 2 7B is that inference latency is still infeasible for Production use cases unless you have a good supply of expensive A100; dropping it by an order of magnitude and letting it run on other cloud GPUs will open new opportunities.
> The bottleneck of Llama 2 7B is that inference latency is still infeasible for Production use cases unless you have a good supply of expensive A100 ?? A 3060 or a slightly bigger AMD/Intel GPU can stream llama 7B about as fast as someone can read, if not faster. A somewhat bigger consumer GPU can batch it and serve dozens of users. I use 13B finetunes on my 2020 14" laptop all the time, with 6GB of VRAM and 16GB of…
> ?? A 3060 or a slightly bigger AMD/Intel GPU can stream llama 7B about as fast as someone can read,
That's the thing: you need a whole GPU per concurrent user, this is insanely expensive if you want to run it as part of a SaaS (which is what most for-profit want to do). Of course running models locally is much better in almost every regard, but nobody is gonna be a billionaire with that…
Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens
#33What does “pretrain” mean in this context? It sounds like normal training
GPT stands for Generative Pre-trained Transformer. The "main" training step using huge amounts of inputs is called pre-training. The idea is that after that pre-training, you might fine tune the model for your specific use case.
I see...that makes sense. Thanks for explaining
Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens
#34>It means you can train a chinchilla-optimal TinyLlama (1.1B param, 22B tokens) in 32 hours with 8 A100. They are training the model on 3000/22=136 times the value of the chinchilla scale. It will be interesting to see how much it will improve after way beyond this value.
I now come to understand that the technobable in Star Trek wasn't that well predicted, in the future we will not be reversing polarities by alligning field cores. Picard will have us align our llamas with chiwawas to get an alpacafied chinchilla model.
There’s should also be a tribble in there, somewhere.
Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens
#35From the FAQ: ' Why would pretraining a 1.1B model for so long make sense? Doesn't it contradict the Chinchilla Scaling Law? Above is the training loss curve taken from the Llama 2 paper. Here I quote from that paper: "We observe that after pretraining on 2T Tokens, the models still did not show any sign of saturation". That is why we believe pretraining a 1.1B model for 3T tokens is a reasonable thing to do. Even if…
> It is something I have been wondering about: why did Meta not keep the training process going on while the loss curves seemed to go down? Could they conceivably release a Llama 2.1 being checkpoints taken a month after 2.0 was 'cut'? Maybe the expected gain is too small compared to what can be gained with fine/instruct tuning afterward anyway? Because choosing the LR decay requires knowing the # of steps in advance…
You could manually increase the learning rate or change the decay at any time.
Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens
#36Earlier quoted context omitted.
> The bottleneck of Llama 2 7B is that inference latency is still infeasible for Production use cases unless you have a good supply of expensive A100 ?? A 3060 or a slightly bigger AMD/Intel GPU can stream llama 7B about as fast as someone can read, if not faster. A somewhat bigger consumer GPU can batch it and serve dozens of users. I use 13B finetunes on my 2020 14" laptop all the time, with 6GB of VRAM and 16GB of…
> ?? A 3060 or a slightly bigger AMD/Intel GPU can stream llama 7B about as fast as someone can read, That's the thing: you need a whole GPU per concurrent user, this is insanely expensive if you want to run it as part of a SaaS (which is what most for-profit want to do). Of course running models locally is much better in almost every regard, but nobody is gonna be a billionaire with that…
Your point is anticipated by the next sentence in the comment you replied to:
"A somewhat bigger consumer GPU can batch it and serve dozens of users."
Did you not read it?
Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens
#37From the FAQ: ' Why would pretraining a 1.1B model for so long make sense? Doesn't it contradict the Chinchilla Scaling Law? Above is the training loss curve taken from the Llama 2 paper. Here I quote from that paper: "We observe that after pretraining on 2T Tokens, the models still did not show any sign of saturation". That is why we believe pretraining a 1.1B model for 3T tokens is a reasonable thing to do. Even if…
> It is something I have been wondering about: why did Meta not keep the training process going on while the loss curves seemed to go down?
If I remember correctly, it's because the main reason they trained multiple models was to show a scaling trend. Each model was trained using a chinchilla-optimal mix of model size, cpu amount, and parameter size. The point was to provide an empirical scaling law that could possibly be extrapolated to estimate the performance of more expensive models, like imagine a billion dollar model for which the model size, data size, and cpu amount is picked in the chinchilla optimal ratios.
On small models the chinchilla optimal scaling stops training the model even when the model is still improving.
The problem comes when people are actually using these small llama models rather than treating them as just data points. If you are actually using these models, what you want is one that is trained forever on as many tokens and training time as possible.
Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens
#38Earlier quoted context omitted.
Lora and Alpaca at Tanagra.
Llama, when the loss fell.
from this episode if i’m not mistaken: https://en.m.wikipedia.org/wiki/Darmok
i watched that series so many times…
Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens
#39Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens
#40From the FAQ: ' Why would pretraining a 1.1B model for so long make sense? Doesn't it contradict the Chinchilla Scaling Law? Above is the training loss curve taken from the Llama 2 paper. Here I quote from that paper: "We observe that after pretraining on 2T Tokens, the models still did not show any sign of saturation". That is why we believe pretraining a 1.1B model for 3T tokens is a reasonable thing to do. Even if…
> It is something I have been wondering about: why did Meta not keep the training process going on while the loss curves seemed to go down? Could they conceivably release a Llama 2.1 being checkpoints taken a month after 2.0 was 'cut'? Maybe the expected gain is too small compared to what can be gained with fine/instruct tuning afterward anyway? Because choosing the LR decay requires knowing the # of steps in advance…
AFAIK re-warming it up and then gradually decreasing it again ought to work fine. Have you seen any research showing that it doesn't?