301–310 of 848 posts
I think I am going to buy a second rtx 3060, as 27B has been just outside of my range for to long, and this looks like the parameter count tipping point
Running it on 2x3060 now. Works pretty well but VRAM is tight . 4bit quants. 1x128k context, 8bit KV, MTP on.
Earlier quoted context omitted.
Yeah, I would appreciate if someone could make sense of the pricing differences between these models. How can a provider run DSv4F at lower cost than a 27B dense or 35B A3B model? Does it come down to utilization and/or specific model tricks and efficiencies (attention, kv cache, etc.)? DeepInfra prices: Qwen 3.6 27B: $0.32 in / $3.20 out Gemma 3 27B: $0.08 in / $0.16 out DeepSeek V4 Flash 0731: $0.08 in / $0.18 out…
DeepSeek V4 Flash is natively FP4 MoE with very compact KV cache. Say 8 GB/s. Qwen 27B is about 60 GB/s at full FP16 precision.
dsv4f appears bigger and more memory/computationally expensive to run any way I look at it.
Earlier quoted context omitted.
> --cache-type-k q8_0 --cache-type-v q8_0 In my tests, even Q8 quantization for the KV cache comes with notable drops in performance for longer tasks. It does provide more context length in limited RAM budgets, but the longer context tasks are where KV quantization starts to show problems. It’s basically unnoticeable for simple and short tasks. > --spec-draft-n-max 5 5 is a lot of tokens to draft. Are you really seei…
Yes to both. The thing is that I can either use the q8 context, or have not enough context window, so I just live with whatever degradation there is. The same can be said about the IQ4_NL. I would not go any lower though. As for the draft count, indeed that depends on what you do with it, but for coding, reverse engineering and that kind of stuff it does pay off in my testing, though 5 is really pushing it, but the 4…
I hope really badly that we'll get a new 35B A3B or similar MoE model! I also miss the Qwen 3 Coder Next, which was 80B A3B, there are quite a few use cases where a non-dense model Also alternate link for viewing the images without signing in: https://xcancel.com/Alibaba_Qwen/status/2088280182356611304
I'm still confused about Qwen 3.6 35B A3B. Everything I read said that the 27B model performs better at coding tasks, so what's the purpose of the 35B model?
Earlier quoted context omitted.
Similar to the way they asked Sol to solve Erdos problems, that's what I want my model to do for programming. I don't want to try to take my best educated guess at what the best design is BEFORE implementation - especially if you're designing a feature for a codebase you're not an expert in, you don't know like the back of your hand (i.e. one that is mostly or entirely LLM generated). What sounds good on paper - ofte…
You're going to be waiting for a while. Even Fable is bad at this, I would constantly have to fix it going down architectural dead ends or just making obvious mistakes. Which sucks for people that want LLMs to do everything like a genie, but does mean senior engineers have a few more years before they become redundant.
Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM. llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spe…
Earlier quoted context omitted.
[flagged]
Yeah I'm sure everyone on r/cursor or in previous HN threads about Grok 4.5 or 4.6 are all unserious and insane. No one actually cares about the politics as long as the model codes well. Edit, quite interesting to see the reception to this comment compared to essentially the same type of comment I made on a Grok 4.6 benchmark HN post: https://news.ycombinator.com/item?id=49275385#49275571 It's true that Cursor gives…
And yeah, people that don't care DO make the world worse through their apathy.
I hope really badly that we'll get a new 35B A3B or similar MoE model! I also miss the Qwen 3 Coder Next, which was 80B A3B, there are quite a few use cases where a non-dense model Also alternate link for viewing the images without signing in: https://xcancel.com/Alibaba_Qwen/status/2088280182356611304
I'm still confused about Qwen 3.6 35B A3B. Everything I read said that the 27B model performs better at coding tasks, so what's the purpose of the 35B model?