61–70 of 107 posts
I'm surprised not to see anything about data-to-parameter ratios for optimal scaling. My superficial understanding per the Chinchilla paper is to target 20 to 1. I'm also confused about this: > ~$1 million: Cost to train a 13 billion parameter model on 1.4 trillion tokens This is apparently related to the LLaMa paper, but that paper seems to cite 1.0T tokens (rather than 1.4T tokens) for the 13B model. Also, if 20 to…
I would add the following two numbers if you're generating realtime text or speech for human consumption: - Human Reading Speed (English): ~250 words per minute - Human Speaking Speed (English): ~150 words per minute Should be treated like the Doherty Threshold [1] for generative content. [1] https://lawsofux.com/doherty-threshold/
I would add the following two numbers if you're generating realtime text or speech for human consumption: - Human Reading Speed (English): ~250 words per minute - Human Speaking Speed (English): ~150 words per minute Should be treated like the Doherty Threshold [1] for generative content. [1] https://lawsofux.com/doherty-threshold/
Human reading speed varies by a factor of 10 or more between individuals, while speaking speed is much more consistent.
Earlier quoted context omitted.
[Author] Completely disagree. Any analysis shows that you see perplexity reduction at 4 bits. Have a look at llama.cpp's results here: https://github.com/ggerganov/llama.cpp#quantization 4 bit has a perplexity score 0.13 or so higher.
You're just wrong. You're looking at the wrong numbers. The perplexity score of a model with twice the parameters in half the bits (4bit) is FAR LOWER (ie better). If you are limited to X RAM and have two 16bit models of size 4X and 2X then the 4X model in 4bit will always be far superior to the 2X model in 8bit, with far lower perplexity. Compare 13B's 4bit perplexity of 5.3607 to 7B's 8bit perplexity of 5.9069. Tha…
You have to wonder if running a huge model, say, 300B parameters at 2-bit quantization might be "optimal" in that it would fit into a single A100 or H100 GPU and likely outperform an 80B parameter 8-bit model...
> Of course there are efforts to reduce this, notably llama.cpp which runs a 13 billion parameter model on a 6GB GPU by quantizing aggressively down to 4 bits (and 8 bits without too much impact), but that’s atypical. No, 4bit quantization is the typical case. At 4bit you can fit twice the parameters of 8bit in the same space for far better performance/perplexity/quality. Running LLMs higher than 4bit is atypical and…
Can somebody please explain how quantization below 8 bit works? Since a byte is the smallest addressable unit I think, is the dimensionality of the weights somehow reduced?
> ~$1 million: Cost to train a 13 billion parameter model on 1.4 trillion tokens Llama paper mentioned 135,168 A100 hours for training 13 billion model on 1 trillion tokens, which means ~$150k for lambdalabs on demand instance.
[Author] Good luck trying to use clusters of Lambda machines. Lambda labs are cheap for a reason: their API is not very featureful (we looked at them and we saw they didn't even support machine tagging). If you're looking for a box or two, lambda labs is fine. If you're looking for 1,000, not so much. Plus they don't actually have any actually A100s available at the moment (2022-05-17). CoreWeave is a nice middle gro…
Earlier quoted context omitted.
I had the same reaction as the OP. I’m not a data scientist by trade or title, but I would personally be a little offended. If you designed the Porsche 911, would you not be offended by the shade tree mechanic who simply knows how to change the oil calling himself a Porsche designer/engineer?
There are people making applications based on LLMs. You may quibble with the term LLM Developer, but to sneer or roll your eyes at it as if it were prima facie inaccurate or laughable is unjustified.
> Of course there are efforts to reduce this, notably llama.cpp which runs a 13 billion parameter model on a 6GB GPU by quantizing aggressively down to 4 bits (and 8 bits without too much impact), but that’s atypical. No, 4bit quantization is the typical case. At 4bit you can fit twice the parameters of 8bit in the same space for far better performance/perplexity/quality. Running LLMs higher than 4bit is atypical and…
> llama.cpp which runs a 13 billion parameter model on a 6GB GPU I think that's a typo there too, the 13B model needs like 10G of memory for 4 bits, it's the 7B one that fits into 6G. Well unless you do the split thing with some layers on the CPU I guess.
> ~$1 million: Cost to train a 13 billion parameter model on 1.4 trillion tokens MosaicML claims they trained a 7 billion parameter on 1 trillion tokens with a budget of $200k. https://www.mosaicml.com/blog/mpt-7b Does training cost scale linearly with model size and token count? If so, that suggests a lower bound of $600k to train the 13 billion params model. (Still roughly the same magnitude)
[Author] Mosaic must be getting some kind of sweetheart deals on A100 80GB and A100 40GB. The prices they are quoting are not what say the AWS on-demand prices are. They quote $2 per GPU for A100 40GB and $2.50 for A100 80GB. That's literally half the AWS on-demand rate for A100s here: https://aws.amazon.com/ec2/instance-types/p4/ And these are impossible to get. We tried to get some for Anyscale, and we were told th…
I have a suggested modification. You are mixing references in your document.
Re: '~$1 million: Cost to train a 13 billion parameter model on 1.4 trillion tokens The LLaMa paper mentions it took them 21 days to train LLaMa using 2048 GPUs A100 80GB GPUs.'
The LLaMA-13B model took 2.75 days of 2048xA100 (135,168 GPU-hours) with 1 trillion tokens. The 21 days for 1.4 trillion was for LLaMA-65B.
I would suggest using the LLaMa-13B numbers since those are the most relevant for this section, or at least modify "21 days to train LLaMa" to "21 days to train LLaMa-65B" for clarity.