41–50 of 125 posts
What we really need is a breakthrough in inference or LLM architecture to allow running GLM-5.2-level models at the size of Qwen 3.6 27b or smaller on consumer devices like a 48GB Macbook Pro, and at least at 100 tokens/second. My hypothesis is that a smaller, less capable but faster model paired with a good harness can run for longer and brute force its way out to solve problems that the bigger models can one-shot.
Earlier quoted context omitted.
That's a 2-bit quant of DS4 flash. You're probably better off running Qwen3.6-27B at Q8.
I think its good advice to test both on your own evals for sure, but the MoE parameters are already natively FP4 in ds4. Dropping to 2bpw isn't as big of a loss as it seems (and as corroborated by antirez's work). Its also only 13B active, so your decode speed would be nearly 2x that of Qwen3.6-27B. So there are other latent benefits as well.
https://huggingface.co/collections/z-lab/dflash
I'm running the qwen3.6-27B + dflash on a spark and tgen is way up, but keep the draft count low, acceptance rate is terrible beyond half a dozen and it requires more memory
Pelican from a few days ago: https://simonwillison.net/2026/Jul/6/hy3/ - I was using the free tier on OpenRouter, which expires on July 21st. I tried the preview model 41 days ago and got a pelican with a "change pelican color" button: https://static.simonwillison.net/static/2026/hy3-preview-pel...
What we really need is a breakthrough in inference or LLM architecture to allow running GLM-5.2-level models at the size of Qwen 3.6 27b or smaller on consumer devices like a 48GB Macbook Pro, and at least at 100 tokens/second. My hypothesis is that a smaller, less capable but faster model paired with a good harness can run for longer and brute force its way out to solve problems that the bigger models can one-shot.
That would be great during the winter months
Curious how people feel about this compared to DS4 Flash, given they are pretty close in size. Also curious how well it holds up to heavy quantization. DS4 Flash can currently run reasonably well on systems with ~96gb+ RAM, I wonder if Hy3 can compete there.
That's a 2-bit quant of DS4 flash. You're probably better off running Qwen3.6-27B at Q8.
I feel like I'm taking crazy pills with hy3, it's either benchmaxxed to hell and back or skill issue on my part but I'd rather use dense gemma. I don't think there's a single model that's wasted more of my time in recent memory.
I'll try it again now that it's out of preview and has been updated with more post-training. It presumably can't be worse, so maybe it's better enough to compete with a 31b model.
Been using this and GLM 5.2 back and forth. I like the speed of Hy3. Also seems very happy to follow instructions. Still haven’t found any open models that follow instructions as good as Mimo v2 pro though
But, it performs very well for its size. I just looked it up, and it's much smaller than I thought it was when I was testing it. 310B A15B is tiny for how well it performs. I guess that explains why it's so cheap.
Pelican from a few days ago: https://simonwillison.net/2026/Jul/6/hy3/ - I was using the free tier on OpenRouter, which expires on July 21st. I tried the preview model 41 days ago and got a pelican with a "change pelican color" button: https://static.simonwillison.net/static/2026/hy3-preview-pel...