As usual, the Jinja templates are messed up so use this [0] to reduce or turn off thinking, fix tool calling, keep a 100% KV cache hit rate, etc.
[0] https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
Qwen 3.8 27B
161–170 of 848 posts
Re: Qwen 3.8 27B
#162Earlier quoted context omitted.
What makes you say that it would be hard to do that? It's long, I guess, but not cryptic. You tell llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it. Perfectly logical blocks with all the model-specific weirdness (that does exist!) abstracted away. You could also just run -m and let llama-server do the…
> llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it. isn't that the hard part? You know the ballpark ideal values for these many parameters since you're a knowledgeable expert but the vast majority of people are just like "I want AI" and have no idea what all the jargon even means.
This is exactly it. I already have broad access to Claude, Gemini, GitHub Copilot. I want to use open models on automated tasks that chew up tokens but where I don't necessarily need the best-in class models and UX.
For Claude, I setting a single config file and then download and run Claude Code CLI. Even easier for the GitHub Copilot CLI.
Re: Qwen 3.8 27B
#163Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM. llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spe…
> --cache-type-k q8_0 --cache-type-v q8_0 In my tests, even Q8 quantization for the KV cache comes with notable drops in performance for longer tasks. It does provide more context length in limited RAM budgets, but the longer context tasks are where KV quantization starts to show problems. It’s basically unnoticeable for simple and short tasks. > --spec-draft-n-max 5 5 is a lot of tokens to draft. Are you really seei…
Yes to both.
The thing is that I can either use the q8 context, or have not enough context window, so I just live with whatever degradation there is. The same can be said about the IQ4_NL. I would not go any lower though.
As for the draft count, indeed that depends on what you do with it, but for coding, reverse engineering and that kind of stuff it does pay off in my testing, though 5 is really pushing it, but the 4090 has so much compute.
Last logline I saw scroll by right now had 47% acceptance rate for 4th and 28% for 5th, but not sure how representative that is. I think when tuning 3.6, I saw more like 33%? But not 100% sure.
Re: Qwen 3.8 27B
#164If the benchmarks don't lie, this is getting very close to Opus 4.6 capability - which was the turning point for me for when AI was "good enough" that it became very hard to justify not using it. I'm sure there's some benchmaxxing going on, and some things you get only with a a larger model. But I'm feeling pretty confident if not by Gemma 5 than by mid 2028 we'll have local models that are almost always as good as O…
What kind of things you only get with a larger model?
Asking it factual information[1]. You just can't compress the entire human knowledge into a 30GB file.
[1] Without searching the internet. And even if you allow it, you'll get much worse results because search means browsing and parsing the top results, and search results are horrible, whereas internal knowledge from training encompasses the entire internet plus all books including very niche stuff.
Re: Qwen 3.8 27B
#165Earlier quoted context omitted.
Thanks for posting! Have you had any success with running without kv cache quantization? Is there a noticeable difference in quality without any? I would assume that would eat into context but 170k is pretty generous!
According to this shitty vibecoded thing "I" built https://hypfer.github.io/will-it-fit-llama-cpp/ (and I guess according to math too), FP16 K/V would give me something like 90k context at the same model quant, which doesn't really fit my usage. But maybe someone else has experience to share there
just to clarify. yes YOU built it. just because you used some tool doesn't mean the idea, prompting, reprompting, babysitting was not your creative input and effort.
put differently, if you put a random person infront of whatever model you used (say, a 50yo receptionist at a pharmacy in india), they would not have been able to create that, because they would have lacked the motivation, idea, background knowledge, taste, etc to create such a thing.
Re: Qwen 3.8 27B
#166Earlier quoted context omitted.
What makes you say that it would be hard to do that? It's long, I guess, but not cryptic. You tell llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it. Perfectly logical blocks with all the model-specific weirdness (that does exist!) abstracted away. You could also just run -m and let llama-server do the…
> llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it. isn't that the hard part? You know the ballpark ideal values for these many parameters since you're a knowledgeable expert but the vast majority of people are just like "I want AI" and have no idea what all the jargon even means.
Sure, but front ends like LMStudio exist for that crowd
Otherwise, if you're a programmer setting up a local harness, it only takes like 20-30 minutes to learn what the right parameters are.
It's very model, hardware, and use case dependent which is why a one size fits all solution doesn't work
Re: Qwen 3.8 27B
#167Earlier quoted context omitted.
Lol at that command. Why is this stuff so hard to run locally? I've spent a few days trying to figure it all out and haven't been able to. LM Studio doesn't work behind proxies. Ollama is confusing and doesn't seem to support Qwen3? And Llama.cpp is your command. I just want to run ` ` with some default parameters set and for it to run locally.
> LM Studio doesn't work behind proxies. Woa, is that still a thing? You mean like SOCKS5 stuff that you have to manually configure in every application that uses the internet? I mean maybe I'm just living under a rock but I feel like that's a rather niche situation you got there.
> I feel like that's a rather niche situation you got there
Every big company in the world uses a network proxy. LM Studio, as far as I can tell, cannot be configured to work behind such proxies.
Re: Qwen 3.8 27B
#168Can anyone confirm whether this new Qwen release is any more concise when thinking? Overthinking was the biggest (only?) downside of the Qwen models.
Re: Qwen 3.8 27B
#169Earlier quoted context omitted.
What makes you say that it would be hard to do that? It's long, I guess, but not cryptic. You tell llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it. Perfectly logical blocks with all the model-specific weirdness (that does exist!) abstracted away. You could also just run -m and let llama-server do the…
> llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it. isn't that the hard part? You know the ballpark ideal values for these many parameters since you're a knowledgeable expert but the vast majority of people are just like "I want AI" and have no idea what all the jargon even means.
Why would they wish to handcraft this ? That is what agents are for ?
They could ask your current agent to a) search for this type of content online for the optimal setup for their hardware b) have the current agent/harness spin it up have it verify the config run few experiments.
Sure AI may make mistakes, or won't get the best possible config probably, but it certainly do a good enough setup, this is a task with feedback on whether the server crashed or poor performance easily measured so the agent can do a pretty good job.
Re: Qwen 3.8 27B
#170Why weren't the points merged again from the "dupe" thread that had 289 points?
https://news.ycombinator.com/item?id=49299684
What a weird mechanism. If someone is judging a thread/topic/event impact by the number of points it got, then doing this unfairly degrades that thread.
It should have deduped by user and combined the 168(at the time of writing this comment) + 289 points. Just add the twitter link from the previous thread as an additional link in the description, like you normally do, move all the points over, and remove the old thread.