Are there any open source host-your-own LLMs that have licensing that allows for commercial use?
Vicuna-13b is on Apache License 2.0.
Numbers every LLM developer should know
31–40 of 107 posts
Re: Numbers every LLM developer should know
#32> ~$1 million: Cost to train a 13 billion parameter model on 1.4 trillion tokens MosaicML claims they trained a 7 billion parameter on 1 trillion tokens with a budget of $200k. https://www.mosaicml.com/blog/mpt-7b Does training cost scale linearly with model size and token count? If so, that suggests a lower bound of $600k to train the 13 billion params model. (Still roughly the same magnitude)
[Author] Mosaic must be getting some kind of sweetheart deals on A100 80GB and A100 40GB. The prices they are quoting are not what say the AWS on-demand prices are. They quote $2 per GPU for A100 40GB and $2.50 for A100 80GB. That's literally half the AWS on-demand rate for A100s here: https://aws.amazon.com/ec2/instance-types/p4/
And these are impossible to get. We tried to get some for Anyscale, and we were told there were no on-demand available and lead time for reserved (ouchie on the price! You're talking a quarter of a million dollars a year for one machine at list) was in weeks.
Once you take the model size and hefty sweetheart deals into account, you're within 10%. Mosaic does have some nice whitebox optimizations, but nothing that radically changes the equation.
Re: Numbers every LLM developer should know
#33I would add the following two numbers if you're generating realtime text or speech for human consumption: - Human Reading Speed (English): ~250 words per minute - Human Speaking Speed (English): ~150 words per minute Should be treated like the Doherty Threshold [1] for generative content. [1] https://lawsofux.com/doherty-threshold/
But I'd say LLMs produce content faster than I can read or write it, because they can produce content which is really dense.
Ask GPT-4 a question and then answer it yourself. Maybe your answer will be as good or better than GPT-4's but GPT-4 writes its answer a lot faster.
Re: Numbers every LLM developer should know
#34Re: Numbers every LLM developer should know
#35> Of course there are efforts to reduce this, notably llama.cpp which runs a 13 billion parameter model on a 6GB GPU by quantizing aggressively down to 4 bits (and 8 bits without too much impact), but that’s atypical. No, 4bit quantization is the typical case. At 4bit you can fit twice the parameters of 8bit in the same space for far better performance/perplexity/quality. Running LLMs higher than 4bit is atypical and…
[Author] Completely disagree. Any analysis shows that you see perplexity reduction at 4 bits. Have a look at llama.cpp's results here: https://github.com/ggerganov/llama.cpp#quantization 4 bit has a perplexity score 0.13 or so higher.
Well, if you have a fixed RAM size, you're better off with the largest model you can fit at 4 bits (13B 4b is way better than 7B 16b despite being twice smaller).
Re: Numbers every LLM developer should know
#36> Of course there are efforts to reduce this, notably llama.cpp which runs a 13 billion parameter model on a 6GB GPU by quantizing aggressively down to 4 bits (and 8 bits without too much impact), but that’s atypical. No, 4bit quantization is the typical case. At 4bit you can fit twice the parameters of 8bit in the same space for far better performance/perplexity/quality. Running LLMs higher than 4bit is atypical and…
[Author] Completely disagree. Any analysis shows that you see perplexity reduction at 4 bits. Have a look at llama.cpp's results here: https://github.com/ggerganov/llama.cpp#quantization 4 bit has a perplexity score 0.13 or so higher.
You're just wrong. You're looking at the wrong numbers. The perplexity score of a model with twice the parameters in half the bits (4bit) is FAR LOWER (ie better).
If you are limited to X RAM and have two 16bit models of size 4X and 2X then the 4X model in 4bit will always be far superior to the 2X model in 8bit, with far lower perplexity.
Compare 13B's 4bit perplexity of 5.3607 to 7B's 8bit perplexity of 5.9069. That is over 0.54 lower perplexity for the same RAM amount by using 4bit! That is MASSIVE!
Re: Numbers every LLM developer should know
#37> Of course there are efforts to reduce this, notably llama.cpp which runs a 13 billion parameter model on a 6GB GPU by quantizing aggressively down to 4 bits (and 8 bits without too much impact), but that’s atypical. No, 4bit quantization is the typical case. At 4bit you can fit twice the parameters of 8bit in the same space for far better performance/perplexity/quality. Running LLMs higher than 4bit is atypical and…
No it isn't, quantization is not free. You lose a significant amount of performance that you are not measuring properly in automated benchmarks when you quantize to that level. You can see it in real time when you take most LLMs and compare them at different quantization levels. I can see the degradation even in the largest llama quite badly even at 8 bits.
If you take a model and quantize it it's obviously going to get worse, but what if you train it again after that?
Re: Numbers every LLM developer should know
#38> Of course there are efforts to reduce this, notably llama.cpp which runs a 13 billion parameter model on a 6GB GPU by quantizing aggressively down to 4 bits (and 8 bits without too much impact), but that’s atypical. No, 4bit quantization is the typical case. At 4bit you can fit twice the parameters of 8bit in the same space for far better performance/perplexity/quality. Running LLMs higher than 4bit is atypical and…
> llama.cpp which runs a 13 billion parameter model on a 6GB GPU
I think that's a typo there too, the 13B model needs like 10G of memory for 4 bits, it's the 7B one that fits into 6G. Well unless you do the split thing with some layers on the CPU I guess.
Re: Numbers every LLM developer should know
#39> There’s usually no need to go beyond 16-bit accuracy, and most of the time when you go to 8-bit accuracy there is too much loss of resolution. I'm not sure this is accurate. From what I have seen, 8-bit quantization is usually fine, and even 4-bit is a viable tradeoff. Here are some benchmarks from TextSynth showing no significant degradation between 16 and 8 bit: https://textsynth.com/technology.html 8-bit uses ha…
The article is right, 8-bit (and especially 4-bit) is atypical for deep learning models and highly depends on the amount of parameters (larger model can handle more quantization) and can even depend on specific training hyperparameters (mainly dropout & weight decay which can induce sparsity)
Thing is, even when the impact from 4-bit is substantial, the larger parameter count it allows on the same hardware more than makes up for it. E.g. llama-30b is better at 4-bit than any derivative of llama-13b, no matter how fine-tuned or quantized.
Re: Numbers every LLM developer should know
#40Earlier quoted context omitted.
Can somebody please explain how quantization below 8 bit works? Since a byte is the smallest addressable unit I think, is the dimensionality of the weights somehow reduced?
[Author] You approximate the weights using fewer bits. You also switch to ints instead of floats and then do some fancy stuff when multiplying to make it all work together. More detail than you probably wanted: https://huggingface.co/blog/hf-bitsandbytes-integration
The latest release of bitsandbytes uses a new fp4 format. 4bit floating point scailing results in much lower perplexity than int4.
Also note that for a fixed memory (RAM) size, 4bit (even int4) is always superior, resulting in lower perplexity than 8bit.
E.g. LLaMA-13B int4 is far better/lower perplexity than LLaMA-7B fp8 while using the same amount of RAM.