since when ngrok is doing ai
Quantization from the Ground Up
31–40 of 74 posts
Re: Quantization from the Ground Up
#32[dead]
Re: Quantization from the Ground Up
#33Quantization is important for me because it's the only way out I can see for a future of programming that doesn't involve going through a giant bigco who can run, as the article says, a machine with 2TB of memory. And not just memory, but my understanding is that for the model to be performant, it has to be VRAM to boot. This comes as the latest concern of mine in a long line around "how software gets written" remain…
[dead]
Re: Quantization from the Ground Up
#34something I have been wondering about is doing regressive layer specific quantization based on large test sets. ie reduce very specifically layers that don't improve general quality.
This is a very well established idea. It's called dynamic quantization. Vary the quantization bit-width (or skip quantization altogether) on a layer by layer basis, using a calibration dataset.
EvoPress is the first time that comes to my mind, when I think of dynamic quantization.
Re: Quantization from the Ground Up
#35Quantization is important for me because it's the only way out I can see for a future of programming that doesn't involve going through a giant bigco who can run, as the article says, a machine with 2TB of memory. And not just memory, but my understanding is that for the model to be performant, it has to be VRAM to boot. This comes as the latest concern of mine in a long line around "how software gets written" remain…
I've been watching the drizzle of LLM papers come through, and I think we're going to hit a 1T param MoE on consumer hardware before this year is out. It'll still be behind the bigco models, but it'll be a force multiplier. Ideally, we'd get these models to run on a CPU. MS BitNet is one way to do this. You can already run ternary LLMs on consumer CPUs with a decent tps.
Though what is consumer hardware right now?
Can we still classify 5090s as consumer hardware given how expensive they are? They're £3k at the moment, and it looks like it's only going to get worse unless the AI bubble pops.
Re: Quantization from the Ground Up
#36Earlier quoted context omitted.
You can still continue to master actual software engineering while others spend their time turning their minds into a palimpsest of tricks and lessons of how to convince one model after another after another after another into giving reasonable output. That you'd still have to vet yourself anyway.
While I think a lot of the AI hype is just hype - everyone saying most of these things have _hitherto untold riches_ levels of financial incentives to say them - I think it's also undeniable that LLMs speed up many aspects of coding. I also think that AI might be the beginning of the end of copyright. While before, everyone with money clearly had tremendous incentive to keep copyright strong, now all of a sudden tril…
Only for the AI companies. Not for you or I.
It's the corrupting nature of capitalism really laid bare. A net loss for so many of their constituents that politicians all over the world are falling over themselves to pave the way for foreign companies to exploit their constituents IP.
A true tragedy of the commons unfolding before us.
I get why, and I get why it's the only realistic choice, but it really is showing the weaknesses of modern politics.
Re: Quantization from the Ground Up
#37[flagged]
Re: Quantization from the Ground Up
#38Earlier quoted context omitted.
I've been watching the drizzle of LLM papers come through, and I think we're going to hit a 1T param MoE on consumer hardware before this year is out. It'll still be behind the bigco models, but it'll be a force multiplier. Ideally, we'd get these models to run on a CPU. MS BitNet is one way to do this. You can already run ternary LLMs on consumer CPUs with a decent tps.
Though what is consumer hardware right now? Can we still classify 5090s as consumer hardware given how expensive they are? They're £3k at the moment, and it looks like it's only going to get worse unless the AI bubble pops.
I was thinking more in terms of 24GB of VRAM total. I started sketching the architecture for such a model this afternoon, nothing novel, just combining existing advancements in the field. It looks achievable.
Re: Quantization from the Ground Up
#39The hardware situation is way better than you think, and quantization is a huge part of why.
Take Qwen 3.5 27B, which is a solid coding model. At FP16 it needs 54GB of VRAM. Nobody's running that on consumer hardware. At Q4_K_M quantization, it needs 16GB. A used RTX 3090 has 24GB and goes for about $900. That model runs locally with room for context.
For 14B coding models at Q4, you're looking at about 10GB. A used RTX 3060 12GB handles that for under $270.
The gap between "needs a datacenter" and "runs on my desk" is almost entirely quantization. A 27B model at Q4 loses surprisingly little quality for most coding tasks. It's not free, but it's not an RTX 7090 either. A used 3090 is probably the most recommended card in the local LLM community right now, and for good reason.
Re: Quantization from the Ground Up
#40The float comparison slider is great.
One thing from practical experience - the quality gap between model sizes shows up in a way benchmarks don't capture. I have a system where a smaller model generates plans and a larger model can override them. On any single output they look comparable. The difference shows up 3-4 steps later — small model makes a decision that sounds reasonable but compounds into a bad plan. Perplexity won't catch that, KL divergence won't either. They both measure one prediction at a time.