Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unsloth Dynamic 3.0 GGUFs

unsloth.ai

Post reply on HN
Print view View original

91–100 of 125 posts

Re: Unsloth Dynamic 3.0 GGUFs

#91

Post by Alephinitesimal »

illusive4080 wrote: post #79
Alephinitesimal wrote: post #34

I mostly use local models when the data has personal information. Earlier this year, I felt the coding quality was still not as good as Claude Code. One thing that works for me is to ask the local model to make some fake data with the same format, let Claude Code work on the fake data, and then bring the code back and run it locally on the real data. This way the real data never leaves my machine, but I can still use…

Have you tried on personal finance analysis? That is what I most want to do but haven’t gotten around to it.

I did try some finance analysis earlier this year. I was using a DGX Spark, so I could run some relatively large models, but the results were pretty mixed at the time. I honestly can't remember which models I used anymore.

Might be worth trying again now though.

Re: Unsloth Dynamic 3.0 GGUFs

#92

Post by cicko »

hedora wrote: post #75
prima-facie wrote: post #65

Earlier quoted context omitted.

You can manage these easily with the huggingface python app. hf download hf://unsloth/Qwen3.8-27B-GGUF \ Qwen3.8-27B-UD-Q4_K_XL.gguf and then see them with `hf cache ls`. Prune old versions with `hf cache prune`.

That's terrible. My computer shipped with a filesystem and curl.

You can't run models with filesystem and curl.

Re: Unsloth Dynamic 3.0 GGUFs

#93

Post by johndough »

danielhanchen wrote: post #44
johndough wrote: post #22

Are there benchmarks for the various Qwen3.8-27B quants that actually measure writing code, maybe even with multiple steps? Low KL divergence does not mean much when the model gets stuck in doom loops all the time. I could of course download and test myself, but that would take days with my internet connection.

We made something called Divergence-300 @32 (and later @512) which tests actual inference across 32 tokens on a held out test (Terminal Bench, DeepSWE, Math etc) We do plan to do larger benchmark suites though!

Great to hear that you are planning larger benchmarks! I am particularly interested in longer-running tasks with many steps and self-correction. Divergence is fine as long as the model can still solve the task, which Divergence-300 @32 does not measure.

The current benchmark suites that frontier AI labs use are probably a good fit, e.g.

https://z.ai/blog/glm-5.3#:~:text=Performance%20across%20com...

https://www.kimi.ai/ai-models/kimi-k3#:~:text=Performance%20...

https://www.anthropic.com/news/claude-opus-5

https://openai.com/index/gpt-5-6/

But guessing from your current benchmarks, I assume that you are severely compute-constrained. What is your time budget?

Re: Unsloth Dynamic 3.0 GGUFs

#94

Post by johnnyApplePRNG »

Alephinitesimal wrote: post #90
Forgeties79 wrote: post #80

Earlier quoted context omitted.

Man I am having a hell of a time trying to optimize 3.8 over 3.6. I don’t have a particularly powerful setup but I can usually push 20-30tok/s on 3.6 and I can barely get to 10 on 3.8. Both unsloth same VRAM/RAM distribution more or less. My 3.6 is still producing consistently better results and faster

That's interesting since both models are dense. I wonder if this is more of an optimization issue with 3.8 rather than something inherent to the architecture.

Could have sworn I read these were the same architectures the other day .... 3.6 and 3.8 at this size.

Re: Unsloth Dynamic 3.0 GGUFs

#95

Post by beacon294 »

kamranjon wrote: post #9
jadbox wrote: post #6

The new IQ4XS has been working pretty well so far on 4090 16gb.

What size context are you able to squeeze in with less than 2gb of headroom? I have had some luck using a quantized kv cache but i fear that also decreases overall quality.

Try the llama.cpp fork by thetom. It's called turboquant after the technique

Re: Unsloth Dynamic 3.0 GGUFs

#97

Post by kennywinker »

walrus01 wrote: post #89
zenoprax wrote: post #71

Earlier quoted context omitted.

Sure, but this is true for all lossy compression (audio, images, etc.) Given 16GB of VRAM, what will give me the best experience in OpenCode? Currently using Qwen3.8_Q_3

probably the best experience would be deepseek v4 flash 0731 (it takes about 170GB RAM on the server side for the full thing and RAM reserved for 1M context) via opencode's $10 a month plan until you use that up, it's either Q8 or full precision. Assuming you're ok with doing things with external inference.

Why would they have listed how much vram they had if they were looking to rent gpu time on someone else's machine?

Re: Unsloth Dynamic 3.0 GGUFs

#98

Post by walrus01 »

kennywinker wrote: post #97
walrus01 wrote: post #89

Earlier quoted context omitted.

probably the best experience would be deepseek v4 flash 0731 (it takes about 170GB RAM on the server side for the full thing and RAM reserved for 1M context) via opencode's $10 a month plan until you use that up, it's either Q8 or full precision. Assuming you're ok with doing things with external inference.

Why would they have listed how much vram they had if they were looking to rent gpu time on someone else's machine?

A casual review of my comment history would show that I've been nothing but the biggest proponent of running models locally, and I do so myself a great deal. But one also has to be realistic about the capabilities of what you can do in a 16GB GPU these days. I already said an extra small Q2 quantization was effectively lobotomized so I didn't want to repeat myself.

This person has basically run into the limit of state of the art for even a modestly sized local model (this isn't deepseek v4 flash 0731 Q8 which I am running myself locally on a great deal more hardware), this is a 27B dense, but they're just not going to have a good time if they expect good quality results out of a Q2. The choices are either upgrade hardware or pay for external inference.

Re: Unsloth Dynamic 3.0 GGUFs

#99

Post by kennywinker »

walrus01 wrote: post #98
kennywinker wrote: post #97

Earlier quoted context omitted.

Why would they have listed how much vram they had if they were looking to rent gpu time on someone else's machine?

A casual review of my comment history would show that I've been nothing but the biggest proponent of running models locally, and I do so myself a great deal. But one also has to be realistic about the capabilities of what you can do in a 16GB GPU these days. I already said an extra small Q2 quantization was effectively lobotomized so I didn't want to repeat myself. This person has basically run into the limit of stat…

Fine, but they already said they are using a Q3, so Q2 being unusable (disagree, but whatever) isn’t helpful new info.
Post reply on HN