Are there any open source host-your-own LLMs that have licensing that allows for commercial use?
Numbers every LLM developer should know
41–50 of 107 posts
Re: Numbers every LLM developer should know
#42> Of course there are efforts to reduce this, notably llama.cpp which runs a 13 billion parameter model on a 6GB GPU by quantizing aggressively down to 4 bits (and 8 bits without too much impact), but that’s atypical. No, 4bit quantization is the typical case. At 4bit you can fit twice the parameters of 8bit in the same space for far better performance/perplexity/quality. Running LLMs higher than 4bit is atypical and…
No it isn't, quantization is not free. You lose a significant amount of performance that you are not measuring properly in automated benchmarks when you quantize to that level. You can see it in real time when you take most LLMs and compare them at different quantization levels. I can see the degradation even in the largest llama quite badly even at 8 bits.
Quantization is not free, but VRAM is even less free.
If you have X amount of VRAM and can fit a 16bit model of size 2X in 8bit or a model of size 4X in 4bit then the 4X model in 4bit is ALWAYS superior with lower perplexity and better performance.
You LOSE performance by using a smaller model in 8bit vs a larger model in 4bit.
Re: Numbers every LLM developer should know
#43RANDOM THOUGHT:
i wonder when we are getting docker for llm ... a Modelfile ?
FROM "PAAMA/16b"
APPLY "MNO/DATASET"
each layer could be lora adapter like thing maybe.
maybe when AI chips are finally here.
Re: Numbers every LLM developer should know
#44> LLM Developer This is the fastest I've rolled my eyes in a long time!
The amount of get-off-my-lawn grognardness that LLM activity inspires is really ridiculous. I really would ask you to take a second look at the spirit of your comment and think carefully about how much you really understand about the work being done on top of LLMs and if it justifies this kind of response.
I had the same reaction as the OP. I’m not a data scientist by trade or title, but I would personally be a little offended. If you designed the Porsche 911, would you not be offended by the shade tree mechanic who simply knows how to change the oil calling himself a Porsche designer/engineer?
Re: Numbers every LLM developer should know
#45> There’s usually no need to go beyond 16-bit accuracy, and most of the time when you go to 8-bit accuracy there is too much loss of resolution. I'm not sure this is accurate. From what I have seen, 8-bit quantization is usually fine, and even 4-bit is a viable tradeoff. Here are some benchmarks from TextSynth showing no significant degradation between 16 and 8 bit: https://textsynth.com/technology.html 8-bit uses ha…
It's true if you're doing training. But for inference severe quantization is mostly okay. And there are some internal parts of a transformer running inference with a quantized model where you might want the x-bit inputs to do calculations with 16 bits like the dot product similarity between vectors.
Re: Numbers every LLM developer should know
#46Earlier quoted context omitted.
The amount of get-off-my-lawn grognardness that LLM activity inspires is really ridiculous. I really would ask you to take a second look at the spirit of your comment and think carefully about how much you really understand about the work being done on top of LLMs and if it justifies this kind of response.
I had the same reaction as the OP. I’m not a data scientist by trade or title, but I would personally be a little offended. If you designed the Porsche 911, would you not be offended by the shade tree mechanic who simply knows how to change the oil calling himself a Porsche designer/engineer?
Context matters. Is a "web developer" someone who makes web pages, or works on a browser rendering engine?
Re: Numbers every LLM developer should know
#47> There’s usually no need to go beyond 16-bit accuracy, and most of the time when you go to 8-bit accuracy there is too much loss of resolution. I'm not sure this is accurate. From what I have seen, 8-bit quantization is usually fine, and even 4-bit is a viable tradeoff. Here are some benchmarks from TextSynth showing no significant degradation between 16 and 8 bit: https://textsynth.com/technology.html 8-bit uses ha…
It's true if you're doing training. But for inference severe quantization is mostly okay. And there are some internal parts of a transformer running inference with a quantized model where you might want the x-bit inputs to do calculations with 16 bits like the dot product similarity between vectors.
Even that is being tackled by newer GPU architectures. For example, novelai is currently training an LLM in fp8 precision, using H100 GPUs.[1]
[1] https://blog.novelai.net/anlatan-acquires-hgx-h100-cluster-4...
https://blog.novelai.net/text-model-progress-is-going-good-8...
Re: Numbers every LLM developer should know
#48I think parts of the write-up are great.
There are some unique assumptions being made in parts of the gist
> 10: Cost Ratio of OpenAI embedding to Self-Hosted embedding
> 1: Cost Ratio of Self-Hosted base vs fine-tuned model queries
I don't know how useful these numbers are if you take away the assumptions that self-hosted will work as well as API.
> 10x: Throughput improvement from batching LLM requests
I see that the write up mentions memory being a caveat to this, but it also depends on the card specs as well. Memory Bandwidth / TFLOPs offered by say 4090 is superior while having the same amount of VRAM as 3090. The caveat mentioned with token length in the gist itself makes the 10x claim not a useful rule of thumb.
Re: Numbers every LLM developer should know
#49RANDOM THOUGHT: i wonder when we are getting docker for llm ... a Modelfile ? FROM "PAAMA/16b" APPLY "MNO/DATASET" each layer could be lora adapter like thing maybe. maybe when AI chips are finally here.
SQLFlow[0] looks sort of like that:
SELECT * FROM iris.train
TO TRAIN DNNClassifier
WITH model.hidden_units = [10, 10], model.n_classes = 3, train.epoch= 10
COLUMN sepal_length, sepal_width, petal_length, petal_width
LABEL class
INTO sqlflow_models.my_dnn_model;
Re: Numbers every LLM developer should know
#50Earlier quoted context omitted.
It's true if you're doing training. But for inference severe quantization is mostly okay. And there are some internal parts of a transformer running inference with a quantized model where you might want the x-bit inputs to do calculations with 16 bits like the dot product similarity between vectors.
Even that is being tackled by newer GPU architectures. For example, novelai is currently training an LLM in fp8 precision, using H100 GPUs.[1] [1] https://blog.novelai.net/anlatan-acquires-hgx-h100-cluster-4... https://blog.novelai.net/text-model-progress-is-going-good-8...
Cool stuff. I looked at https://en.wikipedia.org/wiki/Hopper_%28microarchitecture%29 and I noticed that that the fp8 support is only for the tensor cores and not the CUDA side. Does that mean training with H100 GPU in fp8 mode would use some software ecosystem that's not the existing vast existing CUDA one? Or am I just misunderstanding CUDA cores vs tensor cores?
PS, as a joke, they should implement GPU fluint8 and get baked in non-linearity for the activation function without even using a non-linear function, https://www.youtube.com/watch?v=Ae9EKCyI1xU ("GradIEEEnt half decent: The hidden power of imprecise lines" by suckerpinch)