Numbers every LLM developer should know

Post by Zuiii » Thu, May 18, 2023, 3:31 AM UTC

waleedk wrote: post #32

PoignardAzur wrote: post #30

> ~$1 million: Cost to train a 13 billion parameter model on 1.4 trillion tokens MosaicML claims they trained a 7 billion parameter on 1 trillion tokens with a budget of $200k. https://www.mosaicml.com/blog/mpt-7b Does training cost scale linearly with model size and token count? If so, that suggests a lower bound of $600k to train the 13 billion params model. (Still roughly the same magnitude)

[Author] Mosaic must be getting some kind of sweetheart deals on A100 80GB and A100 40GB. The prices they are quoting are not what say the AWS on-demand prices are. They quote $2 per GPU for A100 40GB and $2.50 for A100 80GB. That's literally half the AWS on-demand rate for A100s here: https://aws.amazon.com/ec2/instance-types/p4/ And these are impossible to get. We tried to get some for Anyscale, and we were told th…

Post by EvgeniyZh » Thu, May 18, 2023, 4:28 AM UTC

Post by sifar » Thu, May 18, 2023, 4:49 AM UTC

kherud wrote: post #19

MacsHeadroom wrote: post #13

> Of course there are efforts to reduce this, notably llama.cpp which runs a 13 billion parameter model on a 6GB GPU by quantizing aggressively down to 4 bits (and 8 bits without too much impact), but that’s atypical. No, 4bit quantization is the typical case. At 4bit you can fit twice the parameters of 8bit in the same space for far better performance/perplexity/quality. Running LLMs higher than 4bit is atypical and…

Can somebody please explain how quantization below 8 bit works? Since a byte is the smallest addressable unit I think, is the dimensionality of the weights somehow reduced?

Post by renonce » Thu, May 18, 2023, 5:24 AM UTC

furyofantares wrote: post #60

armchairhacker wrote: post #33

Earlier quoted context omitted.

But I'd say LLMs produce content faster than I can read or write it, because they can produce content which is really dense. Ask GPT-4 a question and then answer it yourself. Maybe your answer will be as good or better than GPT-4's but GPT-4 writes its answer a lot faster.

It certainly doesn't produce content as fast as I can read it.

Post by bombolo » Thu, May 18, 2023, 6:31 AM UTC

armchairhacker wrote: post #33

abetlen wrote: post #15

I would add the following two numbers if you're generating realtime text or speech for human consumption: - Human Reading Speed (English): ~250 words per minute - Human Speaking Speed (English): ~150 words per minute Should be treated like the Doherty Threshold [1] for generative content. [1] https://lawsofux.com/doherty-threshold/

But I'd say LLMs produce content faster than I can read or write it, because they can produce content which is really dense. Ask GPT-4 a question and then answer it yourself. Maybe your answer will be as good or better than GPT-4's but GPT-4 writes its answer a lot faster.