Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Numbers every LLM developer should know

github.com

Post reply on HN
Print view View original

91–100 of 107 posts

Re: Numbers every LLM developer should know

#91

Post by telesilla »

skykooler wrote: post #63
abetlen wrote: post #15

I would add the following two numbers if you're generating realtime text or speech for human consumption: - Human Reading Speed (English): ~250 words per minute - Human Speaking Speed (English): ~150 words per minute Should be treated like the Doherty Threshold [1] for generative content. [1] https://lawsofux.com/doherty-threshold/

Human reading speed varies by a factor of 10 or more between individuals, while speaking speed is much more consistent.

Exceptions for languages where rapidity of speech really varies according to context, such as in Spanish.

Re: Numbers every LLM developer should know

#92

Post by Tostino »

bombolo wrote: post #79
armchairhacker wrote: post #33

Earlier quoted context omitted.

But I'd say LLMs produce content faster than I can read or write it, because they can produce content which is really dense. Ask GPT-4 a question and then answer it yourself. Maybe your answer will be as good or better than GPT-4's but GPT-4 writes its answer a lot faster.

dense content? Not in my experience. It seems to be really overly verbose for me.

Prompt it to be information dense in its response then.

Re: Numbers every LLM developer should know

#93

Post by bsaul »

vidarh wrote: post #87
cornfutes wrote: post #84

> LLM developer This is the first time I heard this term, and when I Google search "LLM developer" in an incognito tab, different device, this article is one of the first results. Seems like we should first establish what exactly is an LLM developer. > When I was at Google, there was a document put together by Jeff Dean, the legendary engineer, called Numbers every Engineer should know. The personal plug and appeal t…

The term "LLM developer" is clear enough from context.

I’m still not sure if the advices apply to people developing LLM models, or to software developers using LLM in their daily job to produce code.

Re: Numbers every LLM developer should know

#94

Post by furyofantares »

renonce wrote: post #77
furyofantares wrote: post #60

Earlier quoted context omitted.

It certainly doesn't produce content as fast as I can read it.

Only if you use gpt-4. gpt-3.5-turbo is much faster, and gpt-4 is only going to get faster as GPUs get faster.

Yep. I use GPT-4 extensively and exclusively, and the comment I was replying to mentioned GPT-4. I can't wait for it to get faster.

Re: Numbers every LLM developer should know

#95

Post by furyofantares »

cubefox wrote: post #85
renonce wrote: post #77

Earlier quoted context omitted.

Only if you use gpt-4. gpt-3.5-turbo is much faster, and gpt-4 is only going to get faster as GPUs get faster.

Bing also uses GPT-4 and it is very fast. Microsoft spends more ok compute.

It doesn't exclusively use GPT-4, you might be right anyway that their GPT-4 is much faster but you're also not always seeing GPT-4 with them.

Re: Numbers every LLM developer should know

#96

Post by vidarh »

bsaul wrote: post #93
vidarh wrote: post #87

Earlier quoted context omitted.

The term "LLM developer" is clear enough from context.

I’m still not sure if the advices apply to people developing LLM models, or to software developers using LLM in their daily job to produce code.

Most of this applies to people developing applications that depends on LLMs. Some of it also applies to people using LLMs for other purposes. Very little of it is applicable to someone developing LLMs.

Re: Numbers every LLM developer should know

#97

Post by JohnFen »

skykooler wrote: post #63
abetlen wrote: post #15

I would add the following two numbers if you're generating realtime text or speech for human consumption: - Human Reading Speed (English): ~250 words per minute - Human Speaking Speed (English): ~150 words per minute Should be treated like the Doherty Threshold [1] for generative content. [1] https://lawsofux.com/doherty-threshold/

Human reading speed varies by a factor of 10 or more between individuals, while speaking speed is much more consistent.

> speaking speed is much more consistent.

Is it? I've noticed a huge variance in speaking speed in the US, but it tends to vary more between regions rather than individuals.

Re: Numbers every LLM developer should know

#98

Post by Ambix »

jiggawatts wrote: post #65
MacsHeadroom wrote: post #36

Earlier quoted context omitted.

You're just wrong. You're looking at the wrong numbers. The perplexity score of a model with twice the parameters in half the bits (4bit) is FAR LOWER (ie better). If you are limited to X RAM and have two 16bit models of size 4X and 2X then the 4X model in 4bit will always be far superior to the 2X model in 8bit, with far lower perplexity. Compare 13B's 4bit perplexity of 5.3607 to 7B's 8bit perplexity of 5.9069. Tha…

Another factor is that larger models degrade less when quantized. You have to wonder if running a huge model, say, 300B parameters at 2-bit quantization might be "optimal" in that it would fit into a single A100 or H100 GPU and likely outperform an 80B parameter 8-bit model...

Not sure here. The LLaMA models - yes, all weights fit in the small range between -2.0 .. 2.0

And some other models have more crazy numbers with even more crazier outliers within them, like you might have a weight of 12.00 between long array of typical small numbers around 0.00

I've read story about attempt to quantize RWKV model into the 4/5 bits which failed short due to the presence of outlier weights.

The author told somewhere that bigger models had worse perplexity because of this.

Re: Numbers every LLM developer should know

#99

Post by Ambix »

Taek wrote: post #54
waleedk wrote: post #18

Earlier quoted context omitted.

[Author] Completely disagree. Any analysis shows that you see perplexity reduction at 4 bits. Have a look at llama.cpp's results here: https://github.com/ggerganov/llama.cpp#quantization 4 bit has a perplexity score 0.13 or so higher.

There's also research showing that the perplexity reduction is less at higher parameter counts. E.g. a 65b parameter model barely has any impact at all when reducing from 16bit to 4bit

Not actually, you might see here how bigger models have much worse perplexity with 4bit due to the weight outliers:

https://github.com/saharNooby/rwkv.cpp/issues/12

For LLaMA models - yeah, different story.

Re: Numbers every LLM developer should know

#100

Post by Ambix »

kherud wrote: post #19
MacsHeadroom wrote: post #13

> Of course there are efforts to reduce this, notably llama.cpp which runs a 13 billion parameter model on a 6GB GPU by quantizing aggressively down to 4 bits (and 8 bits without too much impact), but that’s atypical. No, 4bit quantization is the typical case. At 4bit you can fit twice the parameters of 8bit in the same space for far better performance/perplexity/quality. Running LLMs higher than 4bit is atypical and…

Can somebody please explain how quantization below 8 bit works? Since a byte is the smallest addressable unit I think, is the dimensionality of the weights somehow reduced?

Go see yourself :)

https://github.com/ggerganov/llama.cpp/blob/master/examples/...

There's too many schemes right now with 4_0 and 5_1 really popular between LLM geeks.

Post reply on HN