Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

github.com

Post reply on HN
Print view View original

11–20 of 61 posts

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#11

Post by sp332 »

GaggiX wrote: post #3

>It means you can train a chinchilla-optimal TinyLlama (1.1B param, 22B tokens) in 32 hours with 8 A100. They are training the model on 3000/22=136 times the value of the chinchilla scale. It will be interesting to see how much it will improve after way beyond this value.

Chinchilla predicts that you could get lower loss by training a larger model with that amount of data. But the model size in this case was chosen for other reasons, mostly speed of inference and cost of fine-tuning. So it's just irrelevant here.

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#12

Post by GaggiX »

sp332 wrote: post #11
GaggiX wrote: post #3

>It means you can train a chinchilla-optimal TinyLlama (1.1B param, 22B tokens) in 32 hours with 8 A100. They are training the model on 3000/22=136 times the value of the chinchilla scale. It will be interesting to see how much it will improve after way beyond this value.

Chinchilla predicts that you could get lower loss by training a larger model with that amount of data. But the model size in this case was chosen for other reasons, mostly speed of inference and cost of fine-tuning. So it's just irrelevant here.

Well it's relevant if you want to compare the model trained optimally using the same amount of compute and this one parameter-bound to see how much you're trading.

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#13

Post by minimaxir »

A robust 1.1B model compared to a 7B model would be strongly appreciated. The bottleneck of Llama 2 7B is that inference latency is still infeasible for Production use cases unless you have a good supply of expensive A100; dropping it by an order of magnitude and letting it run on other cloud GPUs will open new opportunities.

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#15

Post by wongarsu »

syntaxing wrote: post #9

This sounds like a really fun project, running small models would change a lot of industries like games in their example. But how do people afford these projects?! If I am doing my numbers right, it'll cost them 50K to train this model for 3T tokens.

$38k, based on the "90 days using 16 A100-40G" and lambdalabs prices.

That's a lot for a hobby, but small enough that it might be running on a university machine (the TinyLlama devs provide a way to cite them and all seem to work or study at Singapore University of Technology) or could be sponsored (no indication of that now, but "people made an awesome model in our cloud" is good advertisement). Government grants or grants in general also aren't out of the question, especially for a topic with this much hype.

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#16

Post by minimaxir »

RC_ITR wrote: post #14

Not to be a downer, but wasn’t one of OpenAI’s earliest discoveries that training small models on huge datasets leads to over-fitting? It’s my understanding that the entire race to ever-more parameters was driven by that.

A workaround to overfitting is to train on so much distinct data that the model can't overfit.

Newer large datasets like the ones used here optimize for diversity. (e.g. SlimPajama is a heavily-deduped dataset)

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#17

Post by jlokier »

syntaxing wrote: post #9

This sounds like a really fun project, running small models would change a lot of industries like games in their example. But how do people afford these projects?! If I am doing my numbers right, it'll cost them 50K to train this model for 3T tokens.

That's less than a month's income for a few people on here. I recall a comment from an engineer at Nvidia a year or two ago saying $700k/year was about much they were paid, in response to someone else not believing those levels.

Get together 5 people in that position and it's less than a week's income for the group. That sounds doable as a hobby for those lucky people.

More realistically, it's within range for a grant, or use of someone else's hardware if they aren't using it, as the sibling comment from wongarsu said.

Also cloud vendors sometimes give out large batches of credits to startups and such as marketing incentive to get future customers.

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#18

Post by ljlolel »

RC_ITR wrote: post #14

Not to be a downer, but wasn’t one of OpenAI’s earliest discoveries that training small models on huge datasets leads to over-fitting? It’s my understanding that the entire race to ever-more parameters was driven by that.

Learn about the magic of double descent

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#19

Post by isoprophlex »

npsomaratna wrote: post #4
GaggiX wrote: post #3

>It means you can train a chinchilla-optimal TinyLlama (1.1B param, 22B tokens) in 32 hours with 8 A100. They are training the model on 3000/22=136 times the value of the chinchilla scale. It will be interesting to see how much it will improve after way beyond this value.

Possibly a lot. See: https://espadrine.github.io/blog/posts/chinchilla-s-death.ht...

Very interesting, thanks for sharing!

Re: TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

#20

Post by DarmokJalad1701 »

elpocko wrote: post #6
pluijzer wrote: post #5

Earlier quoted context omitted.

I now come to understand that the technobable in Star Trek wasn't that well predicted, in the future we will not be reversing polarities by alligning field cores. Picard will have us align our llamas with chiwawas to get an alpacafied chinchilla model.

Lora and Alpaca at Tanagra.

Llama, when the loss fell.
Post reply on HN