181–189 of 189 posts
Getting so close to good! I consider Gemma 4 31B (dense / no MoE), the new baseline for local models. It's obviously worse than the frontier models, but it feels less like a science experiment than any previous local model I’ve run, including GPT OSS 120B and Nemotron Super 120B. On my M5 Max with 128 GB of RAM and the full 256K context window, I see RAM use spike to about 70 GB, with something like 14 GB of system o…
Earlier quoted context omitted.
This. I have seen way too many people who are overly optimistic about local LLMs. Having spent a decent amount of time playing with them on consumer nvidia GPUs, I understand well that they not going to be widely usable any time soon. Unfortunately not many people share that.
So the cofounder of hugging face made a post about qwen 3.6 being atclaude level of performance for the lols? When were you trying local models? The model releases from April 2026 are a serious change in performance.
Getting so close to good! I consider Gemma 4 31B (dense / no MoE), the new baseline for local models. It's obviously worse than the frontier models, but it feels less like a science experiment than any previous local model I’ve run, including GPT OSS 120B and Nemotron Super 120B. On my M5 Max with 128 GB of RAM and the full 256K context window, I see RAM use spike to about 70 GB, with something like 14 GB of system o…
Well, but if I have a MacBook Air M4 with 16GB, I don't know what useful models can I run.
Earlier quoted context omitted.
Not to be nitpicky, but many of the 4-12b models are somewhere between GPT-3.5 and GPT-4o-mini. It's hard to find a good comparison though, because the benchmarks people score models against change so often. For reference, Sonnet 3.6 came out about a year after GPT 3.5
Don't worry about being nitpicky! I'm going to out-nitpick you.... Actually.... I write and publish my own benchmark for this stuff. It's an agentic SQL benchmark which isn't in the training data yet and I've found can separate frontier models from close-followers (the only models to get 100% are Opus 4.6 and GPT 5.5). The best small model I've found is a fine-tune of Opus-3.5 9B which scores 18/25: https://sql-bench…
Earlier quoted context omitted.
But what if you were going to buy a laptop anyway? Obviously you can't do anything with less than 64 GBytes these days, so the question is just whether you go for the jump to 128. In the UK, it's currently an extra £800 to get a 128 GB vs the 64 GB equivalent. So that's more like 3 years of Claude - I think? - assuming current prices stay the same. Or: you might just feel like £800 isn't an unjustifiable amount of mo…
Realistically it's 48 M5 Pro vs 128 M5 Max due to constraints on how you can configure them. So a more substantial difference of ~2k US.
I got qwen3.6:27B running on my 4090 (24GB) with ~128K context leveraging some of the recent turboquant/rotorquant memory optimizations for activations. Highly suggest going up to that. the q4_xl+rotorquant combo is pretty good. Some reference code if you want to throw your agent at it. https://github.com/rapatel0/rq-models
Forgive my ignorance but aren't they already on huggingface? I assumed turboquant optimizations are already everywhere - in llama-cpp, or the quantization machinery of unsloth and the likes.
I got qwen3.6:27B running on my 4090 (24GB) with ~128K context leveraging some of the recent turboquant/rotorquant memory optimizations for activations. Highly suggest going up to that. the q4_xl+rotorquant combo is pretty good. Some reference code if you want to throw your agent at it. https://github.com/rapatel0/rq-models
What is your exp on performance +40k tokens? I've not gone past that as I've heard reports that were problems start to arise. I'd be happy to know your experience in that regard.