Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

github.com

Post reply on HN
Print view View original

41–50 of 61 posts

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#41

Post by brucethemoose2 »

littlestymaar wrote: post #32
brucethemoose2 wrote: post #21

Earlier quoted context omitted.

> The bottleneck of Llama 2 7B is that inference latency is still infeasible for Production use cases unless you have a good supply of expensive A100 ?? A 3060 or a slightly bigger AMD/Intel GPU can stream llama 7B about as fast as someone can read, if not faster. A somewhat bigger consumer GPU can batch it and serve dozens of users. I use 13B finetunes on my 2020 14" laptop all the time, with 6GB of VRAM and 16GB of…

> ?? A 3060 or a slightly bigger AMD/Intel GPU can stream llama 7B about as fast as someone can read, That's the thing: you need a whole GPU per concurrent user, this is insanely expensive if you want to run it as part of a SaaS (which is what most for-profit want to do). Of course running models locally is much better in almost every regard, but nobody is gonna be a billionaire with that…

A single GPU with a batch size of 1 can serve many users, higher batch sizes can serve many dozens, pool a few and you can serve a sizable userbase.

It may not be super profitable, but its not untenable either.

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#42

Post by minimaxir »

brucethemoose2 wrote: post #41
littlestymaar wrote: post #32

Earlier quoted context omitted.

> ?? A 3060 or a slightly bigger AMD/Intel GPU can stream llama 7B about as fast as someone can read, That's the thing: you need a whole GPU per concurrent user, this is insanely expensive if you want to run it as part of a SaaS (which is what most for-profit want to do). Of course running models locally is much better in almost every regard, but nobody is gonna be a billionaire with that…

A single GPU with a batch size of 1 can serve many users, higher batch sizes can serve many dozens, pool a few and you can serve a sizable userbase. It may not be super profitable, but its not untenable either.

LLMs are GPU compute-bound. If you infer at batch_size = 1 on a model like Llama 2 7B on a "cheap" GPU like a T4 or an L4 it'll use about 100% of the compute, which means you get no benefit from batching.

The exception is the A100 GPU which does not use 100% of GPU compute and therefore you get benefit from batching, but is hella expensive.

The economics are not simple, and in most cases "just use the ChatGPT API" is also the most cost-effective option anyways. A smaller 1.1B model (which would likely not be compute-bound) with similar performance to a 7B model may tip the scales.

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#43

Post by brucethemoose2 »

minimaxir wrote: post #42
brucethemoose2 wrote: post #41

Earlier quoted context omitted.

A single GPU with a batch size of 1 can serve many users, higher batch sizes can serve many dozens, pool a few and you can serve a sizable userbase. It may not be super profitable, but its not untenable either.

LLMs are GPU compute-bound. If you infer at batch_size = 1 on a model like Llama 2 7B on a "cheap" GPU like a T4 or an L4 it'll use about 100% of the compute, which means you get no benefit from batching. The exception is the A100 GPU which does not use 100% of GPU compute and therefore you get benefit from batching, but is hella expensive. The economics are not simple, and in most cases "just use the ChatGPT API" is…

> LLMs are GPU compute-bound.

From what I understand, they are severely bandwidth bound at a GPU batch size of 1. Even llama.cpp is fairly RAM speed bound on a CPU with much less compute than a GPU.

It's just that batching is quite inefficient without an implementation like this: https://www.anyscale.com/blog/continuous-batching-llm-infere...

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#45

Post by thawab »

kristianp wrote: post #44

Could this be used as a source of speculative tokens for larger llama models?, as per https://github.com/ggerganov/llama.cpp/pull/2926 Also, when are we going to start seeing open weights MOE models being released?

1- yes, Gorgie twetted he is looking into it[0].

2- The only 2 i know of are airoboros[1] and Hydra which is still in progress.

[0] https://x.com/ggerganov/status/1698667093711880687?s=46&t=Jp...

[1] https://github.com/jondurbin/airoboros#lmoe

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#46

Post by fpgaminer »

jph00 wrote: post #40
jofi1 wrote: post #29

Earlier quoted context omitted.

> It is something I have been wondering about: why did Meta not keep the training process going on while the loss curves seemed to go down? Could they conceivably release a Llama 2.1 being checkpoints taken a month after 2.0 was 'cut'? Maybe the expected gain is too small compared to what can be gained with fine/instruct tuning afterward anyway? Because choosing the LR decay requires knowing the # of steps in advance…

AFAIK re-warming it up and then gradually decreasing it again ought to work fine. Have you seen any research showing that it doesn't?

That would work, in that it would allow one to continue decreasing the loss, but I wouldn't say that it would work "fine". A model trained with restarts always performs worse than a model trained for the same duration without restarts.

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#48

Post by e12e »

Are they upsampling - whatever that means in the context of datasets?

AFAIU slim pajama is about 627B tokens, and Starcoder:

> approximately 250 Billion tokens.

Ed: I see TFA says:

> Combined Dataset Size - Around 950B tokens

> Total Tokens During Training - 3 trillion (slightly more than 3 epochs/1430k steps)

... but I'm not seeing how one becomes three? That's more like 1 trillion than 3 trillion tokens?

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#49

Post by kristianp »

thawab wrote: post #45
kristianp wrote: post #44

Could this be used as a source of speculative tokens for larger llama models?, as per https://github.com/ggerganov/llama.cpp/pull/2926 Also, when are we going to start seeing open weights MOE models being released?

1- yes, Gorgie twetted he is looking into it[0]. 2- The only 2 i know of are airoboros[1] and Hydra which is still in progress. [0] https://x.com/ggerganov/status/1698667093711880687?s=46&t=Jp... [1] https://github.com/jondurbin/airoboros#lmoe

Thanks. Yes, I've seen airoboros, it aims to use a mixture of fine-tunes of the base model if I recall correctly. Not a truly pre-trained MOE, but could be useful.

Hydra, is this it? https://github.com/SkunkworksAI/hydra-moe

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#50

Post by jph00 »

fpgaminer wrote: post #46
jph00 wrote: post #40

Earlier quoted context omitted.

AFAIK re-warming it up and then gradually decreasing it again ought to work fine. Have you seen any research showing that it doesn't?

That would work, in that it would allow one to continue decreasing the loss, but I wouldn't say that it would work "fine". A model trained with restarts always performs worse than a model trained for the same duration without restarts.

I haven't seen any researcher that supports your contention. SGDR (SGD with restarts) has been shown to work well. https://arxiv.org/abs/1608.03983
Post reply on HN