401–410 of 848 posts
Earlier quoted context omitted.
Luna is twice the price of Deepseek V4 Flash 0731, and less capable :/
Didn't DeepSeek go up 2x?
All of this DeepSeek price increase stuff is related to their specific API, not the entire market.
I hope the bonsai team makes another 1bit quant of this model (or releases code/instructions on how to do it), using the Qwen3.6 27B on my 16GB mac mini has been wild . The 1bit quant feels like opus level… for the first couple turns. Then it has trouble eg switching from plan mode to act mode. This is mostly mitigated by starting a new session. (tbf this limitation is called out on the hf page) I saw unsloth has 1bi…
Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM. llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spe…
llama-server.exe ^
-m "Qwen3.8-27B-UD-Q2_K_XL.gguf" ^
--presence-penalty 0.0 ^
--repeat-penalty 1.0 ^
--fit-ctx 128000 ^
-ctk q4 0 ^
-ctv q4 0 ^
--reasoning-budget -1 ^
--chat-template-kwargs "{\"preserve thinking\": true}" ^
--host 0.0.0.0 ^
--port 8033
[1] https://github.com/ggml-org/llama.cpp/releases
Instructions if you want to do the same:
1. download two files llama-b10434-bin-win-cuda-13.3-x64.zip and cudart-llama-bin-win-cuda-13.3-x64.zip from that llama.cpp Github releases page, and extract both into the same folder.
2. Download the Qwen3.8-27B-UD-Q2_K_XL.gguf file from huggingface and put it into the same folder beside the `llama-server.exe`.
3. Create a file named "RUN_QWEN_3.8.bat" next to `llama-server.exe` and put the text above into that bat file. Double-click the bat file, then open http://localhost:8033 in your browser to see a chat window.
You can use it with any agents by pointing them at http://localhost:8033/v1 which is a working OpenAI compatible endpoint (it doesn't use a token, if you give one it's ignored).
Congratulations, you're now running Qwen 3.8 27B.
Note: I built the computer in question for playing games, yes it needed to be Windows 11 for anticheat reasons to play games with family, I didn't want to dual boot so here I am. I figure I should share instructions for folks who may also have a Windows PC around for such purposes. Specs for this are AMD 9800X3D, 32GB of system RAM, RTX 5070Ti 16GB
Earlier quoted context omitted.
I'd love to understand this more. Are you saying the Qwen team spends their very impressive human and compute resources on publishing these amazing models and then botches the chat template with mundane bugs? Like maybe I just misunderstand what's the hard part but wouldn't you assume that people who can put together an impressive model can also write a proper jinja chat template for it?
Yes for the first question. Google of all companies didn't even get it right with Gemma for a while until recently. For some reason it doesn't seem like people can actually get these templates right.
Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM. llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spe…
Lol at that command. Why is this stuff so hard to run locally? I've spent a few days trying to figure it all out and haven't been able to. LM Studio doesn't work behind proxies. Ollama is confusing and doesn't seem to support Qwen3? And Llama.cpp is your command. I just want to run ` ` with some default parameters set and for it to run locally.
There are some clients that will index the models and allow you to do that but I'm no expert, I've used OLama studio but it always seems to go weird for me.
Even this command above, it's not clear where op got the model from. So I'm with yah.
For example, op uses : Qwen3.8-27B-IQ4_NL.gguf.. But I cant see where to download it. It's not tagged on hugging face at least..
Earlier quoted context omitted.
This is the version we'll be testing on our rtx 6000 today! Thank you
Why not just run FP8 on vLLM with that much vRAM? It's plenty fast.
I am just now getting the benchmarks running against 3.8 27b but I expect similar results from benching 3.6 27b at the same quant.
As usual, the Jinja templates are messed up so use this [0] to reduce or turn off thinking, fix tool calling, keep a 100% KV cache hit rate, etc. [0] https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
I'd love to understand this more. Are you saying the Qwen team spends their very impressive human and compute resources on publishing these amazing models and then botches the chat template with mundane bugs? Like maybe I just misunderstand what's the hard part but wouldn't you assume that people who can put together an impressive model can also write a proper jinja chat template for it?
Earlier quoted context omitted.
> I feel like that's a rather niche situation you got there Every big company in the world uses a network proxy. LM Studio, as far as I can tell, cannot be configured to work behind such proxies.
> Every big company in the world uses a network proxy. It's becoming more rare, now. A lot of the universal truths about corporate networks from the early 2000s are no longer true today. Some companies are stuck in their ways though. The overlap between companies that require someone to use a network proxy and companies that have GPU-equipped machines with enough RAM for LLMs and and that allow people to download and…
I would love to see more data on that because I've seen it constantly. There is more isolation maybe where you can do whatever on 'open' network, but always some kind of proxy/vpn connection for hitting anything sensitive.
The operlap is there.. But I would be worried if it was just flat out taken away from secure managed connections just because of AI.. Again, would love to see the numbers of your assumptions.
Earlier quoted context omitted.
I absolutely love this comment. I wished there was a website where people would post their working command lines as well as what hardware they are using to run that stuff on + tokens / sec prefill + gen.
As a MBP and DGX spark owner, I would love such a site... (Feel like it would be a low effort feature of hugging face). Searching through Reddit and forums for best commands is annoying.
Waiting for the MTP version to pop up on Unsloth. Speculative decoding makes a huge difference. Been running quantized 3.6 at 110t/s on a cheap 5060Ti and quite happy with it. If 3.8 improves on it, it would be awesome.