Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Numbers every LLM developer should know

github.com

Post reply on HN
Print view View original

101–107 of 107 posts

Re: Numbers every LLM developer should know

#101

Post by cubefox »

furyofantares wrote: post #95
cubefox wrote: post #85

Earlier quoted context omitted.

Bing also uses GPT-4 and it is very fast. Microsoft spends more ok compute.

It doesn't exclusively use GPT-4, you might be right anyway that their GPT-4 is much faster but you're also not always seeing GPT-4 with them.

I'm pretty sure it at least mostly uses GPT-4.

Re: Numbers every LLM developer should know

#102

Post by bombolo »

Tostino wrote: post #92
bombolo wrote: post #79

Earlier quoted context omitted.

dense content? Not in my experience. It seems to be really overly verbose for me.

Prompt it to be information dense in its response then.

So I must specifically ask for it, but it's not at all the default.

Re: Numbers every LLM developer should know

#103

Post by Tostino »

bombolo wrote: post #102
Tostino wrote: post #92

Earlier quoted context omitted.

Prompt it to be information dense in its response then.

So I must specifically ask for it, but it's not at all the default.

I get it, but it is just about infinitely configurable to your specific needs so it doesn't bother me too much what the default response is.

Re: Numbers every LLM developer should know

#104

Post by darknoon »

cubefox wrote: post #85
renonce wrote: post #77

Earlier quoted context omitted.

Only if you use gpt-4. gpt-3.5-turbo is much faster, and gpt-4 is only going to get faster as GPUs get faster.

Bing also uses GPT-4 and it is very fast. Microsoft spends more ok compute.

afaict OpenAI's instance is massively overloaded, you can see with the 32k context model actually being faster in practice rather than slower

Re: Numbers every LLM developer should know

#105

Post by darknoon »

superkuh wrote: post #50
Jackson__ wrote: post #47

Earlier quoted context omitted.

Even that is being tackled by newer GPU architectures. For example, novelai is currently training an LLM in fp8 precision, using H100 GPUs.[1] [1] https://blog.novelai.net/anlatan-acquires-hgx-h100-cluster-4... https://blog.novelai.net/text-model-progress-is-going-good-8...

Cool stuff. I looked at https://en.wikipedia.org/wiki/Hopper_%28microarchitecture%29 and I noticed that that the fp8 support is only for the tensor cores and not the CUDA side. Does that mean training with H100 GPU in fp8 mode would use some software ecosystem that's not the existing vast existing CUDA one? Or am I just misunderstanding CUDA cores vs tensor cores? PS, as a joke, they should implement GPU fluint8 and…

you can access the tensor cores from cuda, in practice you might generate the code with something like openai triton

Re: Numbers every LLM developer should know

#106

Post by SethTro »

MacsHeadroom wrote: post #36
waleedk wrote: post #18

Earlier quoted context omitted.

[Author] Completely disagree. Any analysis shows that you see perplexity reduction at 4 bits. Have a look at llama.cpp's results here: https://github.com/ggerganov/llama.cpp#quantization 4 bit has a perplexity score 0.13 or so higher.

You're just wrong. You're looking at the wrong numbers. The perplexity score of a model with twice the parameters in half the bits (4bit) is FAR LOWER (ie better). If you are limited to X RAM and have two 16bit models of size 4X and 2X then the 4X model in 4bit will always be far superior to the 2X model in 8bit, with far lower perplexity. Compare 13B's 4bit perplexity of 5.3607 to 7B's 8bit perplexity of 5.9069. Tha…

Another factor in favor of 8 bits is that inference speed will generally be 2x faster than a larger model at 4 bits.
Post reply on HN