Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Numbers every LLM developer should know

github.com

Post reply on HN
Print view View original

11–20 of 107 posts

Re: Numbers every LLM developer should know

#11

Post by sebzim4500 »

Flux159 wrote: post #4

I think that it would be helpful to add a fine-tuning costs for an open source model (think LLaMA to Alpaca). From the phrasing around fine tuning right now it seems like it's using openai's fine tuning api to determine that cost, but it's not very clear. Also this would be helpful for other foundation models if that doesn't already exist - how much VRAM to run Stable Diffusion v2.1 at different resolutions, running…

They mention that they could finetune a 6B model for $7. Obviously the number depends on the amount of data and the model size but it's probably not going to be a significant expense in practice.

Re: Numbers every LLM developer should know

#13

Post by MacsHeadroom »

> Of course there are efforts to reduce this, notably llama.cpp which runs a 13 billion parameter model on a 6GB GPU by quantizing aggressively down to 4 bits (and 8 bits without too much impact), but that’s atypical.

No, 4bit quantization is the typical case.

At 4bit you can fit twice the parameters of 8bit in the same space for far better performance/perplexity/quality.

Running LLMs higher than 4bit is atypical and almost always sub-optimal (compared to running a model half the size in 8bit).

Even pretraining and finetuning in 4bit is likely to become the norm soon as fp4 becomes more well understood.

Re: Numbers every LLM developer should know

#14

Post by qeternity »

jncraton wrote: post #7

> There’s usually no need to go beyond 16-bit accuracy, and most of the time when you go to 8-bit accuracy there is too much loss of resolution. I'm not sure this is accurate. From what I have seen, 8-bit quantization is usually fine, and even 4-bit is a viable tradeoff. Here are some benchmarks from TextSynth showing no significant degradation between 16 and 8 bit: https://textsynth.com/technology.html 8-bit uses ha…

The problem with 8bit at the moment is massive performance degradation with bitsandbytes. Recent improvements in 4bit inference mean that 8bit is now a massive laggard (although there’s no reason not to expect this to resolve).

Re: Numbers every LLM developer should know

#15

Post by abetlen »

I would add the following two numbers if you're generating realtime text or speech for human consumption:

- Human Reading Speed (English): ~250 words per minute

- Human Speaking Speed (English): ~150 words per minute

Should be treated like the Doherty Threshold [1] for generative content.

[1] https://lawsofux.com/doherty-threshold/

Re: Numbers every LLM developer should know

#16

Post by f_devd »

jncraton wrote: post #7

> There’s usually no need to go beyond 16-bit accuracy, and most of the time when you go to 8-bit accuracy there is too much loss of resolution. I'm not sure this is accurate. From what I have seen, 8-bit quantization is usually fine, and even 4-bit is a viable tradeoff. Here are some benchmarks from TextSynth showing no significant degradation between 16 and 8 bit: https://textsynth.com/technology.html 8-bit uses ha…

The article is right, 8-bit (and especially 4-bit) is atypical for deep learning models and highly depends on the amount of parameters (larger model can handle more quantization) and can even depend on specific training hyperparameters (mainly dropout & weight decay which can induce sparsity)

Re: Numbers every LLM developer should know

#17

Post by waleedk »

jncraton wrote: post #7

> There’s usually no need to go beyond 16-bit accuracy, and most of the time when you go to 8-bit accuracy there is too much loss of resolution. I'm not sure this is accurate. From what I have seen, 8-bit quantization is usually fine, and even 4-bit is a viable tradeoff. Here are some benchmarks from TextSynth showing no significant degradation between 16 and 8 bit: https://textsynth.com/technology.html 8-bit uses ha…

[Author] Fair point. Adjusted the language.

Nonetheless people do tend to use 16 bit huggingface models, and if you do go to 8 bits and it's wrong, you're never quite sure if it's the quant or the model.

Re: Numbers every LLM developer should know

#18

Post by waleedk »

MacsHeadroom wrote: post #13

> Of course there are efforts to reduce this, notably llama.cpp which runs a 13 billion parameter model on a 6GB GPU by quantizing aggressively down to 4 bits (and 8 bits without too much impact), but that’s atypical. No, 4bit quantization is the typical case. At 4bit you can fit twice the parameters of 8bit in the same space for far better performance/perplexity/quality. Running LLMs higher than 4bit is atypical and…

[Author] Completely disagree. Any analysis shows that you see perplexity reduction at 4 bits. Have a look at llama.cpp's results here:

https://github.com/ggerganov/llama.cpp#quantization

4 bit has a perplexity score 0.13 or so higher.

Re: Numbers every LLM developer should know

#19

Post by kherud »

MacsHeadroom wrote: post #13

> Of course there are efforts to reduce this, notably llama.cpp which runs a 13 billion parameter model on a 6GB GPU by quantizing aggressively down to 4 bits (and 8 bits without too much impact), but that’s atypical. No, 4bit quantization is the typical case. At 4bit you can fit twice the parameters of 8bit in the same space for far better performance/perplexity/quality. Running LLMs higher than 4bit is atypical and…

Can somebody please explain how quantization below 8 bit works? Since a byte is the smallest addressable unit I think, is the dimensionality of the weights somehow reduced?

Re: Numbers every LLM developer should know

#20

Post by qeternity »

furyofantares wrote: post #8
contravariant wrote: post #2

How come the token to word ratio is smaller than 1 if tokens are either words or part of words? Shouldn't you expect more tokens than words?

I think all the ratios given are x:1 and they tell you x.

It’s the other way around.

1 GPT4 token is equivalent to 50 GPT3.5 tokens.

1 token is equivalent to 0.75 words.

Post reply on HN