Excellent! Thank you so much for making/posting this
[Author] You're welcome -- glad it was useful!
Numbers every LLM developer should know
21–30 of 107 posts
Re: Numbers every LLM developer should know
#22> LLM Developer This is the fastest I've rolled my eyes in a long time!
The amount of get-off-my-lawn grognardness that LLM activity inspires is really ridiculous.
I really would ask you to take a second look at the spirit of your comment and think carefully about how much you really understand about the work being done on top of LLMs and if it justifies this kind of response.
Re: Numbers every LLM developer should know
#23How come the token to word ratio is smaller than 1 if tokens are either words or part of words? Shouldn't you expect more tokens than words?
[Author] Fair point -- I clarified the language and gave a concrete example. Hope that helps!
Re: Numbers every LLM developer should know
#24Are there any open source host-your-own LLMs that have licensing that allows for commercial use?
Re: Numbers every LLM developer should know
#25Are there any open source host-your-own LLMs that have licensing that allows for commercial use?
Dolly from Databricks is one at least
Re: Numbers every LLM developer should know
#26> Of course there are efforts to reduce this, notably llama.cpp which runs a 13 billion parameter model on a 6GB GPU by quantizing aggressively down to 4 bits (and 8 bits without too much impact), but that’s atypical. No, 4bit quantization is the typical case. At 4bit you can fit twice the parameters of 8bit in the same space for far better performance/perplexity/quality. Running LLMs higher than 4bit is atypical and…
Can somebody please explain how quantization below 8 bit works? Since a byte is the smallest addressable unit I think, is the dimensionality of the weights somehow reduced?
I believe it's locally (inner-loop or simd op) up-cast to float8/float16/int8, but I haven't looked at the internals of llama.cpp myself
Re: Numbers every LLM developer should know
#27> Of course there are efforts to reduce this, notably llama.cpp which runs a 13 billion parameter model on a 6GB GPU by quantizing aggressively down to 4 bits (and 8 bits without too much impact), but that’s atypical. No, 4bit quantization is the typical case. At 4bit you can fit twice the parameters of 8bit in the same space for far better performance/perplexity/quality. Running LLMs higher than 4bit is atypical and…
No it isn't, quantization is not free. You lose a significant amount of performance that you are not measuring properly in automated benchmarks when you quantize to that level.
You can see it in real time when you take most LLMs and compare them at different quantization levels. I can see the degradation even in the largest llama quite badly even at 8 bits.
Re: Numbers every LLM developer should know
#28> Of course there are efforts to reduce this, notably llama.cpp which runs a 13 billion parameter model on a 6GB GPU by quantizing aggressively down to 4 bits (and 8 bits without too much impact), but that’s atypical. No, 4bit quantization is the typical case. At 4bit you can fit twice the parameters of 8bit in the same space for far better performance/perplexity/quality. Running LLMs higher than 4bit is atypical and…
Can somebody please explain how quantization below 8 bit works? Since a byte is the smallest addressable unit I think, is the dimensionality of the weights somehow reduced?
[Author] You approximate the weights using fewer bits. You also switch to ints instead of floats and then do some fancy stuff when multiplying to make it all work together.
More detail than you probably wanted: https://huggingface.co/blog/hf-bitsandbytes-integration
Re: Numbers every LLM developer should know
#29Are there any open source host-your-own LLMs that have licensing that allows for commercial use?
Dolly from Databricks is one at least
[Author] TL;DR OS LLM models are coming.
Dolly's not that great -- I've hit lots of issues using it to be honest .
MosaicML has a nice commercially usable model here: https://www.mosaicml.com/blog/mpt-7b
I think they're one of the leading ones (bias: they're kinda competitors to my employer Anyscale, but you gotta say something's good when it is).
Red Pajama are leading an effort to build a fully open source model similar to LLaMa. https://www.together.xyz/blog/redpajama
Re: Numbers every LLM developer should know
#30> ~$1 million: Cost to train a 13 billion parameter model on 1.4 trillion tokens
MosaicML claims they trained a 7 billion parameter on 1 trillion tokens with a budget of $200k.
https://www.mosaicml.com/blog/mpt-7b
Does training cost scale linearly with model size and token count? If so, that suggests a lower bound of $600k to train the 13 billion params model. (Still roughly the same magnitude)