Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unsloth Dynamic 3.0 GGUFs

unsloth.ai

Post reply on HN
Print view View original

101–110 of 125 posts

Re: Unsloth Dynamic 3.0 GGUFs

#101

Post by jacquesm »

Alephinitesimal wrote: post #91
illusive4080 wrote: post #79

Earlier quoted context omitted.

Have you tried on personal finance analysis? That is what I most want to do but haven’t gotten around to it.

I did try some finance analysis earlier this year. I was using a DGX Spark, so I could run some relatively large models, but the results were pretty mixed at the time. I honestly can't remember which models I used anymore. Might be worth trying again now though.

I'm having a really hard time doing on twin DGX spark what I could do on my quad 3090 rig (which is a scaled down version of what I was using before, the power requirements and the noise were really an issue but I loved the speed and the amount of VRAM). The results tend to be inconsistent, there is lots of looping, far more tokens generated for the same job and lower quality output. I suspect there is some kind of regression in the B12X kernels or something to that effect because none of that should happen, the exact same model on both machines gives wildly different results. Probably this will sort itself out over time. If I may ask, what model / software combo were you using?

Re: Unsloth Dynamic 3.0 GGUFs

#103

Post by jwr »

These are very good!

I'm hoping for speed improvements because the only problem running the 27B model on my Macbook pro (M4 Max) is the speed: 20 tokens per second. I benchmarked and MTP actually makes things slower, so I disabled MTP altogether. I'm hoping there will be some breakthroughs or optimizations that will allow me to run this at 30-50 tokens per second, which would make a big difference.

Re: Unsloth Dynamic 3.0 GGUFs

#104

Post by m1keil »

jwr wrote: post #103

These are very good! I'm hoping for speed improvements because the only problem running the 27B model on my Macbook pro (M4 Max) is the speed: 20 tokens per second. I benchmarked and MTP actually makes things slower, so I disabled MTP altogether. I'm hoping there will be some breakthroughs or optimizations that will allow me to run this at 30-50 tokens per second, which would make a big difference.

I have a 36gb M3 Max. I tested it across quite a few different options: llama.cpp, oLMX, ollama with different options.

So far ollama managed to be the most performant of them all. I will get 30 to 40 tokes/sec with it when using the -mlx version of Qwen3.8.

Whatever the sauce the ollama folks baked into the mlx + MTP mix is currently working the best out of the box.

Re: Unsloth Dynamic 3.0 GGUFs

#105

Post by jantse »

m1keil wrote: post #104
jwr wrote: post #103

These are very good! I'm hoping for speed improvements because the only problem running the 27B model on my Macbook pro (M4 Max) is the speed: 20 tokens per second. I benchmarked and MTP actually makes things slower, so I disabled MTP altogether. I'm hoping there will be some breakthroughs or optimizations that will allow me to run this at 30-50 tokens per second, which would make a big difference.

I have a 36gb M3 Max. I tested it across quite a few different options: llama.cpp, oLMX, ollama with different options. So far ollama managed to be the most performant of them all. I will get 30 to 40 tokes/sec with it when using the -mlx version of Qwen3.8. Whatever the sauce the ollama folks baked into the mlx + MTP mix is currently working the best out of the box.

Thanks for sharing! Did you observe a speed difference between ollamas mlx version and the mlx-community/Qwen3.8-27B-4bit from HF ran with mlx_vlm.generate (with MTP)? Or is it the same?

Re: Unsloth Dynamic 3.0 GGUFs

#106

Post by Forgeties79 »

Alephinitesimal wrote: post #90
Forgeties79 wrote: post #80

Earlier quoted context omitted.

Man I am having a hell of a time trying to optimize 3.8 over 3.6. I don’t have a particularly powerful setup but I can usually push 20-30tok/s on 3.6 and I can barely get to 10 on 3.8. Both unsloth same VRAM/RAM distribution more or less. My 3.6 is still producing consistently better results and faster

That's interesting since both models are dense. I wonder if this is more of an optimization issue with 3.8 rather than something inherent to the architecture.

I’m also not an engineer/coder so it’s equally possible I’m just doing something wrong.

Re: Unsloth Dynamic 3.0 GGUFs

#108

Post by catchnear4321 »

embedding-shape wrote: post #100
cicko wrote: post #92

Earlier quoted context omitted.

You can't run models with filesystem and curl.

To be fair, you can't run models with "hf download" either...

To be fair, the point was more that cURL isn’t a local hf cache manager, much like it isn’t an inference backend, pointing out that avoiding additional dependencies manages risk at the cost of functionality and/or maintenance overhead…

Re: Unsloth Dynamic 3.0 GGUFs

#109

Post by close04 »

walrus01 wrote: post #63
m463 wrote: post #61

Earlier quoted context omitted.

I wonder if there should be some kind of header sort of like a README inside mixed with something like EXIF information that camera images embed.

there's metadata in a GGUF (as can be seen when loading one into llama-server with verbose level 4) but it doesn't appear to show what the expected sha256 of the file is.

With large downloads ideally they'd provide all the info upfront. Even some unique code in the name that you can look up in a readme on the site.

Re: Unsloth Dynamic 3.0 GGUFs

#110

Post by julianlam »

Forgeties79 wrote: post #80
latentsea wrote: post #73

Earlier quoted context omitted.

Qwen3.8-27B has been the turning point for me. It's not as strong as the absolute frontier, but it's the first time I feel local coding models are actually functionally useable as daily drivers.

Man I am having a hell of a time trying to optimize 3.8 over 3.6. I don’t have a particularly powerful setup but I can usually push 20-30tok/s on 3.6 and I can barely get to 10 on 3.8. Both unsloth same VRAM/RAM distribution more or less. My 3.6 is still producing consistently better results and faster

You're likely using 3.6-35B-A3B, the 3.8 is currently a 27 billion parameter dense model.
Post reply on HN