31–40 of 125 posts
Are there benchmarks for the various Qwen3.8-27B quants that actually measure writing code, maybe even with multiple steps? Low KL divergence does not mean much when the model gets stuck in doom loops all the time. I could of course download and test myself, but that would take days with my internet connection.
I tested Qwen 3.8 on the Blade CTF last night, it took 3 hours but got the correct answer. I know that didn’t answer your question but I was looking for a test suite and couldn’t find anything. After reading the logs, there is far less doom looping than with 3.6, but whether that’s a one off or not is up for debate. Q4_K_P
Interestingly, it also seems to tend toward self-correcting, which makes lower quantizations borderline usable. There'll be more faffing around, but still converging toward a solution. I wonder if that's a deliberate product of its RL.
Is it possible to use a model that needs around 64 GB VRAM if you have four GPUs with 16 GB VRAM each?
I am getting 14 t/s on my 16 GB card at full context with the UD-Q3_K_XL quant. Model link: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF .
One thing that works for me is to ask the local model to make some fake data with the same format, let Claude Code work on the fake data, and then bring the code back and run it locally on the real data.
This way the real data never leaves my machine, but I can still use a stronger model for most of the coding.
Might be off-topic but: is it possible to perform such a quantization on Apple devices? Something like Mac Studio Ultra M1 (even if it would take weeks/months)?
I use this project: https://github.com/vllm-project/llm-compressor
Is it possible to use a model that needs around 64 GB VRAM if you have four GPUs with 16 GB VRAM each?
Using multiple cards is one of the things that the models and software that Unsloth releases does really well in terms of ease of use and relatively good performance.
Might be off-topic but: is it possible to perform such a quantization on Apple devices? Something like Mac Studio Ultra M1 (even if it would take weeks/months)?
[0]: https://github.com/ggml-org/llama.cpp/blob/master/tools/quan...
Earlier quoted context omitted.
Of course. Models don't actually require VRAM. Nor do they require regular RAM. You could have 1 GB of RAM and swap the model to disk as you need different parts of it. And if you didn't have enough disks you could access weights via a network connection.
You don’t even need electricity. You could print the model weights onto millions of sheets of paper, and hire a team of carrier pigeons to fly them into your office one by one. No VRAM!
Cool. Now run TerminalHard and compare to unquantized 27B. KLD of 1%, or similar error metric that multiplies, on 10000 tokens would give accumulated error of 2,000,000%
I don't think you can extrapolate that measurement across multiple sequential draws like that. We presumably are comparing against a single trajectory rather than a tree of trajectories. So once we make the wrong choice and step off of the blessed path, we have no way to assign a ranking to the next token; it's error is undefined. I've seen LLMs self correct in chains of thought ("because of foo and bar, I need to...…
Having "Wait, bar is not true, so that won't work" is not necessarily a correction. In fact, the problem is: across a long text it is a correction of a single mistake, but we are talking about thousands here.
But yes, of course that was a rough estimate. But the problem is - we don't really know what we are measuring here. Maybe there's a 2,000,000x difference of intelligence between coding indexes 52 and 50. By some measure that just feels small because that's how we process it akin to audio db.
Regardless the point is KLD and whatever they came up with is not meaningful. And they did not publish comparisons on real benchmarks.
Is it possible to use a model that needs around 64 GB VRAM if you have four GPUs with 16 GB VRAM each?
Yes, you can. Ideally though, you want to minimize the number of cards and maximize the amount of memory in each card. Using multiple cards is one of the things that the models and software that Unsloth releases does really well in terms of ease of use and relatively good performance.