Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8 27B

huggingface.co

Post reply on HN
Print view View original

481–490 of 848 posts

Re: Qwen 3.8 27B

#481

Post by SwellJoe »

xlayn wrote: post #466

The unsloth Q8kxl https://huggingface.co/unsloth/Qwen3.8-27B-GGUF for some reason is looping and going crazy on the think part (I tried to search for an email to let the guys know but didn't find one)... I used the bartowski one and that one doesn't have that issue https://huggingface.co/bartowski/Qwen3.8-27B-GGUF?show_file_... that's using llama.cpp llama-server \ -m ~/somePath/Qwen3.8-27B-UD-Q8_K_XL.gguf \ -np 1 --…

I'm not seeing that with that exact quantization from Unsloth, so far. I'm seeing a _lot_ of thinking before it starts doing, but it all seems pretty reasonable and not loopy (at least no more loopy than big models, with the expected "But, wait! I need to..." types of back-tracking). So, it's taking a long time, but I don't think it's doing anything pathological.

Re: Qwen 3.8 27B

#482

Post by bilekas »

MrDrMcCoy wrote: post #458
bilekas wrote: post #405

Earlier quoted context omitted.

I'll actually defend you on this one, but not only the command, that's more of a fine tuning option. On the hugging face link for example, there's no flat 'download' area. It's completely unintuitive for people to know how to use it. Reminds me of my first time trying to make sense of torrents when they first arrived. There are some clients that will index the models and allow you to do that but I'm no expert, I've u…

Official llama.cpp releases ship with huggingface support. If you don't want to download it yourself, you can just use the `repo/model:quant` convention and it will handle downloading locally for you.

Plain question for you, where can I find the gguf model of this to direct download ?

Re: Qwen 3.8 27B

#483

Post by parthsareen »

bmitc wrote: post #138
hypfer wrote: post #109

Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM. llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spe…

Lol at that command. Why is this stuff so hard to run locally? I've spent a few days trying to figure it all out and haven't been able to. LM Studio doesn't work behind proxies. Ollama is confusing and doesn't seem to support Qwen3? And Llama.cpp is your command. I just want to run ` ` with some default parameters set and for it to run locally.

Hi! From Ollama here - you can run: ollama run qwen3.8 (or if on mac qwen3.8:27b-mlx)

Re: Qwen 3.8 27B

#484

Post by petu »

SparkyMcUnicorn wrote: post #302
mordae wrote: post #258

Earlier quoted context omitted.

DeepSeek V4 Flash is natively FP4 MoE with very compact KV cache. Say 8 GB/s. Qwen 27B is about 60 GB/s at full FP16 precision.

If active weight bandwidth is everything, then wouldn't Qwen 35b a3b (FP8) be roughly half the cost of dsv4f (FP4)? dsv4f appears bigger and more memory/computationally expensive to run any way I look at it.

> and more memory/computationally expensive to run any way I look at it.

Just speculating, but look at attention. 35B-A3B context is heavier -- about 20GB per 1M tokens vs ~7GB. So if you have ≥10M tokens of context per inference node, then DS4 Flash uses less memory than Qwen 35B.

Qwen attention is probably more computationaly expensive as well.

Also I guess not much interest/competition in serving Qwen edge model.

Re: Qwen 3.8 27B

#485

Post by satvikpendem »

gpm wrote: post #459
satvikpendem wrote: post #177

Earlier quoted context omitted.

You should check out Grok, it's quite a good deal from the Cursor subscription side but it's cheap even by API prices.

I think it's very clear that someone who has checked out all the models but the one that called itself mechahitler and is explicitly being fine tuned to support far-right politics is making the choice for reasons other than performance and cost. It's not like all the other models even had plausible claims to those metrics.

As I said to a dead reply, for coding all of that is immaterial, as long as it codes well then that's all that matters to most people, except it seems those who have an idelogical issue in which case the other model companies also have issues.

Re: Qwen 3.8 27B

#486

Post by badsectoracula »

bilekas wrote: post #182
nubg wrote: post #165

Earlier quoted context omitted.

just to clarify. yes YOU built it. just because you used some tool doesn't mean the idea, prompting, reprompting, babysitting was not your creative input and effort. put differently, if you put a random person infront of whatever model you used (say, a 50yo receptionist at a pharmacy in india), they would not have been able to create that, because they would have lacked the motivation, idea, background knowledge, tas…

You sound like your trying to reassure yourself of something. I sure hope my boss doesn't think he built my work! He'd probably get fired pretty quickly during on call!

> I sure hope my boss doesn't think he built my work! He'd probably get fired pretty quickly during on call!

Your boss is a human.

This is a computer program running on your PC.

I hope you can see the difference.

Re: Qwen 3.8 27B

#487

Post by sixothree »

satvikpendem wrote: post #242
altruios wrote: post #235

Earlier quoted context omitted.

[flagged]

Yeah I'm sure everyone on r/cursor or in previous HN threads about Grok 4.5 or 4.6 are all unserious and insane. No one actually cares about the politics as long as the model codes well. Edit, quite interesting to see the reception to this comment compared to essentially the same type of comment I made on a Grok 4.6 benchmark HN post: https://news.ycombinator.com/item?id=49275385#49275571 It's true that Cursor gives…

> No one actually cares about the politics as long as the model codes well.

I do. There are enough models to choose from that I don't need to use one from a guy who did a nazi salute at the presidential podium, and proceeded to rid our government of brown people he felt didn't deserve their job.

Re: Qwen 3.8 27B

#488

Post by shdh »

XCSme wrote: post #384
app13 wrote: post #376

Earlier quoted context omitted.

Whats your setup? I have a single 3090 and am struggling to get it purring

5900x, 3090 24gb (slightly undervolted), 128gb ddr4, running via Ollama. I am benchmarking it now locally, will put the results and speed/tps on aibenchy.com

How did you undervolt the 3090?

Re: Qwen 3.8 27B

#489

Post by bilekas »

CamperBob2 wrote: post #452
bilekas wrote: post #411

Earlier quoted context omitted.

Promise I'm not being flippant or rude, but why not ask it to write something like a parameterized script to do the same thing for you ?

Laziness, primarily. Plus, no two models are exactly alike, and some have different instructions in their model card or on the HF page.

Hmm I'm still struggling with the downloading of all the models I want to try locally, so I might be out of the loop, but all of this, including tuning (to your local HW), feels like it could be automated. I'm old though.

Re: Qwen 3.8 27B

#490

Post by SwellJoe »

yassa9 wrote: post #46
tosh wrote: post #7

27b dense model at Opus 4.6 level Opus at home I hope there also will be a new ~10b variant

can you tell me ideas of usecases of 9 or 10B language models ? I cant find any usecases other than training a lora on them to give good bash commands for example

I use Gemma 4 12B in the 4-bit quantization for all sorts of vision tasks (image sorting, classification, description). It's also good for the same sorts of things for text (but there are probably better/faster models for text, 12B just happens to excel at vision tasks). The Qwen 9B is also very good for those tasks. If you need to do any kind of "search the web, grab some data, do some kind of action" tasks, these small models are perfect for that. Scraping data in a fuzzy format into a database or report or spreadsheet, producing a dashboard of news, etc.

Small local models can also be used for sub-agent tasks in most agent harnesses. But I'd probably run a larger MoE for that; they're faster and have broader knowledge. The dense models, even very small ones, are not blazing fast.

I don't code with any models small enough to run locally, at least not so far. Qwen 3.8 27B might be the tipping point, though. It's looking really promising, though it's probably slow enough that I won't ever actually use it. I'd rather pay $100/month for a faster model, even if Qwen 3.8 turns out to be smart enough for most of my work. Running it locally with the 8-bit quantization is going at 12-30 t/s, depending on how much context it's chewing on. So, if all you do with AI is coding, then you're better off doing it in the cloud.

But, there's lots of things a small model can do that aren't coding.

Post reply on HN