221–227 of 227 posts
Earlier quoted context omitted.
By early 2028, major players like Intel, AMD, QC will ship accelerators in consumer laptops capable of running ~1T MoE models at ~100 tok/s
Literally the only way this is going to happen is if aliens come to earth and gift us some amazing technology.
Earlier quoted context omitted.
We're too easily conflating parameter count with capability. That Bonsai 27B you're running is at 2-bit quantization. Is it really better than the best 10-18B models?
1. yes. https://www.alphaxiv.org/abs/2607.bonsai-27b table 14 shows that bonsai retains roughly 95% of the fp16 27b model's average performance and outperforms post-training quantization at a similar bit width. it doesn't directly compare against every top 10-18b model, but it is clearly still performing like a large model. 2. quantization != native low precision training. a model trained in native ternary should gen…
Earlier quoted context omitted.
> AI will replace humans because giant business believe they can do the same work, not because they will actually be able to. This has been being claimed for at least 2 years now. Wouldn't we already be seeing disasters if this was the case? It's certainly been around long enough to cause some real damage. Instead there is a slow trickle of things that makes the news by people who didn't bother instituting a single f…
I can easily assert than in corporation consulting there are no human translation teams any longer, all translations are AI driven, and at very least there might exist some QA regarding those translations. Likewise for image assets for CMS and ecommerce sites, now instead of hiring agencies, many of those images are AI generated. Just because it hasn't come yet to some coder jobs, doesn't mean others haven't been alr…
Earlier quoted context omitted.
1. yes. https://www.alphaxiv.org/abs/2607.bonsai-27b table 14 shows that bonsai retains roughly 95% of the fp16 27b model's average performance and outperforms post-training quantization at a similar bit width. it doesn't directly compare against every top 10-18b model, but it is clearly still performing like a large model. 2. quantization != native low precision training. a model trained in native ternary should gen…
Models are not like a fps counter where losing a few percentage has no actual impact. These percentages may be the difference between a model that writes code and one that goes into loops, and think rm -rf . is a good idea. There is a reason why most models try to stay in the FP4 or higher range, because the reduced accuracy can have major consequences. You are better off with a 8b FP4+ model then a 27b Q2 model.
Consider two models: one is 16B and trained natively in 2 bits; one is 8B and trained natively in FP4. These models have the same total number of weight bits, but one has twice as many parameters. There is no real reason the FP4 one should be better just because it's FP4. It might be, but that is an empirical question, not a general rule.
Post-training quantization is another thing entirely. Taking a model trained at higher precision and forcing it down to 2 bits is going to hurt performance, often very badly. But this was never my point.
Earlier quoted context omitted.
I have a dual xeon also, same as OP: Ivy Bridge + 128GB DRAM, and was never really able to get decent LLM performance out of it. So I ended up biting the bullet and adding a "budget tier" A4000 20GB GPU. Too bad all my DRAM is wasted now--not sure if there is a way to take advantage of lots of DRAM once you move over to having inference happening on the GPU.
Have you tried putting the KV cache on the GPU and running inference from RAM? From what I gather, prompt processing is particularly painful using RAM alone.
Earlier quoted context omitted.
We're too easily conflating parameter count with capability. That Bonsai 27B you're running is at 2-bit quantization. Is it really better than the best 10-18B models?
No. But do you need to run every small problem through a 10B-30B model? We're smashing ants with hammers most of the time. We're asking frontier Opus/Fable models to classify text and build frontend code. Once we start dissecting these problems into smaller discreet tasks and having the big reasoning models do the tough stuff, we suddenly have an economical system. Not for the company hoping for a big IPO, but for th…
But I meant to counter gp’s claim that “I can run a 27b model on an iPhone” is kind of pointless and disingenuous. Yes I’m sure someone will come up with a way to run a “27b model” at 0.1 bit quantization on an Apple Watch pretty soon misses the whole point of saying a model is “27b” in capability.
Achieving a parameter count is not the point. And is almost meaningless