Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Quantization from the Ground Up

ngrok.com

Post reply on HN
Print view View original

51–60 of 74 posts

Re: Quantization from the Ground Up

#51

Post by adrian_b »

epaulson wrote: post #43

I was a little confused by this part: "This is what's happening to the parameters of models when they're quantized down to sizes that are possible to run on your laptop. Instead of floats, small integers are what get stored and loaded into memory. When the time comes to use the quantized values, to generate an answer to a question for example, the values are dequantized on the fly. You might think this sounds slower,…

Very old CPUs had support only down to FP16, which is useful in graphics applications.

Then support for Bfloat16 and for INT8 has been added, which are not useful for anything else but AI/ML applications. Then support for FP8 has been added. Even smaller formats are supported only on some very recent GPUs.

If you have a recent enough GPU, it might support something like float2 or float4, but if you have an older GPU you must convert the short format to the next bigger format that is supported, before performing some operations.

Re: Quantization from the Ground Up

#52

Post by charcircuit »

mrsilencedogood wrote: post #8

Quantization is important for me because it's the only way out I can see for a future of programming that doesn't involve going through a giant bigco who can run, as the article says, a machine with 2TB of memory. And not just memory, but my understanding is that for the model to be performant, it has to be VRAM to boot. This comes as the latest concern of mine in a long line around "how software gets written" remain…

You can rent compute from small companies to run the models. It's even cheaper if multiple people are able to use the same model at once that way you can pay for being included within a bigger batch as opposed to for the entire compute.

Re: Quantization from the Ground Up

#53

Post by nl »

all2 wrote: post #24
mrsilencedogood wrote: post #8

Quantization is important for me because it's the only way out I can see for a future of programming that doesn't involve going through a giant bigco who can run, as the article says, a machine with 2TB of memory. And not just memory, but my understanding is that for the model to be performant, it has to be VRAM to boot. This comes as the latest concern of mine in a long line around "how software gets written" remain…

I've been watching the drizzle of LLM papers come through, and I think we're going to hit a 1T param MoE on consumer hardware before this year is out. It'll still be behind the bigco models, but it'll be a force multiplier. Ideally, we'd get these models to run on a CPU. MS BitNet is one way to do this. You can already run ternary LLMs on consumer CPUs with a decent tps.

I mean you can run a 1T model on consumer hardware now by doing things like layer offloading and streaming from SSD. It's just too slow to be useful.

Re: Quantization from the Ground Up

#54

Post by nl »

mattmanser wrote: post #36
mrsilencedogood wrote: post #28

Earlier quoted context omitted.

While I think a lot of the AI hype is just hype - everyone saying most of these things have _hitherto untold riches_ levels of financial incentives to say them - I think it's also undeniable that LLMs speed up many aspects of coding. I also think that AI might be the beginning of the end of copyright. While before, everyone with money clearly had tremendous incentive to keep copyright strong, now all of a sudden tril…

Only for the AI companies. Not for you or I. It's the corrupting nature of capitalism really laid bare. A net loss for so many of their constituents that politicians all over the world are falling over themselves to pave the way for foreign companies to exploit their constituents IP. A true tragedy of the commons unfolding before us. I get why, and I get why it's the only realistic choice, but it really is showing th…

Strong disagree.

I love AI because I love building things and it lets me build more things I like faster.

If anything it's anti-capitalist: For example I built a software bluetooth proxy for Docker that let me use the underlaying BT device for Home Assistant even though the HA docs said I'd have to buy a new device. There is no way I'd do that without AI.

And I've built many many random project that I'd never have thought about doing without AI.

Re: Quantization from the Ground Up

#56

Post by AbanoubRodolf »

msbhogavi wrote: post #39

The hardware situation is way better than you think, and quantization is a huge part of why. Take Qwen 3.5 27B, which is a solid coding model. At FP16 it needs 54GB of VRAM. Nobody's running that on consumer hardware. At Q4_K_M quantization, it needs 16GB. A used RTX 3090 has 24GB and goes for about $900. That model runs locally with room for context. For 14B coding models at Q4, you're looking at about 10GB. A used…

[flagged]

Re: Quantization from the Ground Up

#57

Post by est »

epaulson wrote: post #43

I was a little confused by this part: "This is what's happening to the parameters of models when they're quantized down to sizes that are possible to run on your laptop. Instead of floats, small integers are what get stored and loaded into memory. When the time comes to use the quantized values, to generate an answer to a question for example, the values are dequantized on the fly. You might think this sounds slower,…

> I am confused what actually happens in the vectorized ADD and MULT instructions in the GPU with these quantized numbers.

I might be wrong, but I think LLM is all about comparing distance between tokens. You can tell that -255 and +255 are very separated, but you are also away that -8 and +8 are also very far away.

Microsoft Bitnet and Google TurboQuant shows that in extreme you can use just -1, 0, +1

Re: Quantization from the Ground Up

#58

Post by faangguyindia »

msbhogavi wrote: post #39

The hardware situation is way better than you think, and quantization is a huge part of why. Take Qwen 3.5 27B, which is a solid coding model. At FP16 it needs 54GB of VRAM. Nobody's running that on consumer hardware. At Q4_K_M quantization, it needs 16GB. A used RTX 3090 has 24GB and goes for about $900. That model runs locally with room for context. For 14B coding models at Q4, you're looking at about 10GB. A used…

U are better off just buying their coding plan.

Running LLM makes no sense whatsoever

Re: Quantization from the Ground Up

#59

Post by oompydoompy74 »

faangguyindia wrote: post #58
msbhogavi wrote: post #39

The hardware situation is way better than you think, and quantization is a huge part of why. Take Qwen 3.5 27B, which is a solid coding model. At FP16 it needs 54GB of VRAM. Nobody's running that on consumer hardware. At Q4_K_M quantization, it needs 16GB. A used RTX 3090 has 24GB and goes for about $900. That model runs locally with room for context. For 14B coding models at Q4, you're looking at about 10GB. A used…

U are better off just buying their coding plan. Running LLM makes no sense whatsoever

Remaining dependent on proprietary frontier models that you can only access via an API makes no sense whatsoever. My hope is that the future is open weight models running on local hardware.

Re: Quantization from the Ground Up

#60

Post by rdos »

msbhogavi wrote: post #39

The hardware situation is way better than you think, and quantization is a huge part of why. Take Qwen 3.5 27B, which is a solid coding model. At FP16 it needs 54GB of VRAM. Nobody's running that on consumer hardware. At Q4_K_M quantization, it needs 16GB. A used RTX 3090 has 24GB and goes for about $900. That model runs locally with room for context. For 14B coding models at Q4, you're looking at about 10GB. A used…

14B even at Q4 isn't realistic for coding on a single 12GB RTX 3060. Token speed is too slow. After all they are dense models. You aren't getting a good MoE model under 30B. You can do OCR, STT, TTS really well and for LLMs, good use cases are classification, summarization and extraction with <10B models.
Post reply on HN