161–170 of 246 posts
I've got it (Qwen 3.8 flash next) working (sans ... MTP working on that now).
https://github.com/rdaum/eider/
~80tok/sec prefill, 12tok/sec decode, ~80GiB memory resident, the n-gram table pages from SSD.
In initial testing on Ryzen 395 / Strix Halo it's about 22 tokens/s generation and the output quality is impressive. Better and faster than 3.8 27B, and enough better to justify moving away from 3.6 35B even though 35B is still faster. The Unsloth weights don't come with vision support but you can add that back yourself. llama.cpp recipe in case anyone else wants to save some time on setup: https://pastebin.com/fcqsb…
> Better... than 3.8 27B How can a "6b active per token" MoE be better than a just released 27b in the same family? Possible maybe, but quite interesting and requiring some explanation. Edit: ok, on second thought, probably because the small "experts" are really rich in specialized training compared to the dense model. Still raising questions about the details, e.g. the reasoning abilities (or all meta-skills) of a "…
Earlier quoted context omitted.
I find LLMisms very annoying to read, it’s almost like they are bullet points in the shape of a paragraph. It feels very “skippy” to me. EDITED: Removed a question that I couldn’t make feel suitably polite.
I quite agree. Any sufficiently self-stereotypical format for prose is grating to me after enough time reading or listening to it. Humans are best engaged by mixing up the length, style, and tone of their sentences, in my experience. LLMs do the opposite of that and it makes their output an irritating slog to read through in full. I can't help but wonder if this is on purpose (or an inevitable evolutionary feature as…
In initial testing on Ryzen 395 / Strix Halo it's about 22 tokens/s generation and the output quality is impressive. Better and faster than 3.8 27B, and enough better to justify moving away from 3.6 35B even though 35B is still faster. The Unsloth weights don't come with vision support but you can add that back yourself. llama.cpp recipe in case anyone else wants to save some time on setup: https://pastebin.com/fcqsb…
We should still expect significant performance improvements. I believe Unsloth’s branch (and ggml’s) don’t support MTP or the ngram embedding yet.
Earlier quoted context omitted.
Gonna have to wait a few days to see what the wizards of the HF community come up with…
They are already working on it. https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF https://unsloth.ai/docs/models/qwen3.8-next > You will need at least 75 GB of RAM or unified memory to run the model. Its smallest 1-bit quantized version is larger than usual because of the model’s architecture so 1-bit isn't really 1-bit at all. However, this also means the quantization is less aggressive, allowing the model to r…
FYI: nothing seems to be able to run this (easily) yet. llama.cpp, vllm etc I couldn't get working because of no support in the mainline version.
This branch works now: https://github.com/unslothai/llama.cpp/tree/qwen4exp/qwen3.8...
cmake -B build -DGGML_CUDA=ON cmake -B build -DGGML_METAL=ON cmake --build build --config Release -j --target llama-server llama-cliEarlier quoted context omitted.
You’re absolutely right to be hopeful. Three honest possibilities, and I’ll be straight with you about each: 1. It overthinks — Just like the previous iteration. High confidence. 2. It doesn’t overthink — Improvement from the last model for your use case. Regression for others. 3. It sometimes overthinks — Best case all around. A feature, not an impairment. One final thing worth mentioning: (I made myself irrationall…
> You’re absolutely right to be hopeful. Three honest possibilities, and I’ll be straight with you about each: > [UGC styled humorously as LLMisms] All joking aside, having interacted with Claude intensely for the last 8 months and about 30 hours/week in the last 3, I’ve started to notice how (for want of a better word) “readable” (“digestible” ? “comprehensible” ? “Predictable” is the wrong direction.) information c…
Earlier quoted context omitted.
World knowledge also means knowing the various algorithms and ways particular programming problems are solved. You can't search what you don't even know exists.
>You can't search what you don't even know exists. that's not really entirely true -- one can google for "fast pathfinding' and stumble upon A-star , all that had to be queried was the intent/desire. a lot of smaller agentic models and a lot of harnesses live on that premise.
For the impatient, I merged llama.cpp tentative branches to get it running here https://github.com/alainnothere/llama.cpp/tree/disk-cache-ev... , thing runs at 23.54 token/sec and my setup runs at high 30 the 3.8 dense 27B. and this is the pelican from the iq4_xs model https://github.com/alainnothere/llama.cpp/blob/disk-cache-ev...
NVFP4 quant
Earlier quoted context omitted.
We should still expect significant performance improvements. I believe Unsloth’s branch (and ggml’s) don’t support MTP or the ngram embedding yet.
The active param count is so small I'm not sure how much advantage MTP will have, the current llama.cpp does load the ngram embeddings but I haven't verified it uses them. I expect to redeploy all this stuff every few days as the tooling gets improved.
It's not world changing, but at those speeds I'll take anything I can get.
(The ngram embeddings in this case are paged to/from disk which seems to cost ... basically nothing).