Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.6 27B is the sweet spot for local development

quesma.com

Post reply on HN
Print view View original

221–230 of 809 posts

Re: Qwen 3.6 27B is the sweet spot for local development

#221

Post by oceanplexian »

smt88 wrote: post #181
AlpacaJones wrote: post #150

Earlier quoted context omitted.

The key word there is 'currently'.

Economies of scale are a fact of nature and aren’t going to be subverted in the future by even the most advanced local models

Things can get both more expensive and cheaper at scale, hence the term.

For example (and relevant to AI) I can generate electricity on my roof at $0.20-25/kWh, batteries included. In California the electric utility can’t offer it cheaper than $0.30-0.50/kWh. Therefore at scale, electricity is actually more expensive.

There are many such examples.

Re: Qwen 3.6 27B is the sweet spot for local development

#222

Post by Arubis »

iagooar wrote: post #199

I love my MacBook Pro M5 128GB RAM and I love qwen3.6. BUT DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise. Running any kind of sophisticated job on the very laptop you are using is just not viable. Sure you can use it in clamshell mode, but forget touching it while working with AI coding or ag…

Don't forget that your OLED screen will start to color-shift as the heat cooks the panel!

Re: Qwen 3.6 27B is the sweet spot for local development

#223

Post by saganus »

verdverm wrote: post #184
dofm wrote: post #171

Earlier quoted context omitted.

I am not sure I disagree, and I certainly don't mean to disagree very fervently. But I think if you want to really learn to ride well, understand horses well, there might be some benefit in learning how to shoe a horse. At some level it should never only be someone else's job.

At the same time, most people can drive without understanding how a car works.

You actually do need some understanding of how a car works, no?

For example, you need to know it uses gasoline (or diesel), it requires oil changes every certain amount of time, break pad replacement, etc.

You also probably need to know that you can't operate cars over a certain amount of water, that you need a driver's license, stopping at red lights, etc.

Sure, you might not need to be a mechanic, but that's far from not understanding how a car works, which to me sounds similar to knowing how to shoe a horse, which is different than being a horse vet.

Re: Qwen 3.6 27B is the sweet spot for local development

#224

Post by fumeux_fume »

Otternonsenz wrote: post #174
fumeux_fume wrote: post #115

Earlier quoted context omitted.

I suspect with those specs, you're not in the game right now for reliably using local models for code generation. The easiest way in is a MacBook with at least 32GB of RAM. This should be able to run a 4bit quantization of qwen 3.6 using the MLX format really well.

Now that I’m dipping more into this space, am gonna see what I can upgrade with the motherboard I have, but RAM pricing as it is, I’ll need to be smart about when I upgrade. I very much appreciate the frank response, as it makes me feel less defeated at knowing my understanding of how it should work is not the full issue, hahaha

M series macs are usually used for running these LLMs locally because the GPU and CPU share the same pool of RAM at very low latency. If you upgrade your RAM on a different kind of chipset without the Unified Memory Architecture, then it'll be much slower to produce all the tokens you need. Just another data point to add to your upgrade equation.

Re: Qwen 3.6 27B is the sweet spot for local development

#225

Post by danilocesar »

sleepyeldrazi wrote: post #137
nok22kon wrote: post #118

Earlier quoted context omitted.

a computer with 24 GB VRAM is at least $3000

I can't speak for the US, but in Germany (where hardware is usually more expensive, not less), I got my 3090 3 months ago for 750 euro and have been running the iq4_nl 27B using q4 kv (which after recent patches in llama.cpp is in my xp indistinguishably accurate from q8 of f16) at full ctx, with MTP at 2, peaking around 70 t/s on small ctx, around 50 t/s when im around 64k and ends around 40 t/s near the cap. The re…

I can't find a 3090 for less than 2k CADs (or 1200 eur). Is this the average price in Germany? It's pretty cheap.

Re: Qwen 3.6 27B is the sweet spot for local development

#226

Post by zx76 »

I see a lot of people writing about how expensive the hardware to run these local models is - but see no mentions of the Intel Arc Pro B50/B60/B70 which seem like decent value if you're not interested in Apple kit (as much as anything can be decent value in the current status quo).

I just got a B70 with 32GB RAM for the equivalent of $1200 (incl. sales tax and import duties to my non-US location, so presumably it could be cheaper elsewhere). The memory bandwidth is 608 GB/s. For M5 Max (32-core GPU) it's 460 GB/s and for M5 Max (40-core GPU) it's 614 GB/s. A 3090 is still faster at ~900 GB/s but you're getting 32GB VRAM for a lot less than equivalent Nvidia cards. It's about 1/3 the bandwidth of a 5090 for 1/3 the cost, but with the same 32GB VRAM. If you're interested in being able to run bigger quants with some context and stay on a lower budget then it's an appealing trade off.

I'm still exploring using these local models so don't want to spend the equivalent of $5 000 - $10 000 just to test it out. I don't mind slightly slower perf to do some experimentation more affordably.

I actually got an B50 16GB (with meager 70w TDP!) first to test an Intel card with my stack - it worked easily with Ubuntu & Vulkan. I'd read a lot about hassles and people writing them off as unusable but it seems like these are often with SYCL which doesn't even seem to outperform vulkan and so why bother? (The B50 was just $370 inclusive tax and duties). Literally `apt install` the vulkan libraries and it worked with default xe driver in 26.04 and the vulkan build of llama.cpp. The SR-IOV PF/VF also just works with qemu/kvm, no tricks required. Since I got it fwupdmgr has updated the firmware twice so Intel is presumably actually trying to support these products.

Re: Qwen 3.6 27B is the sweet spot for local development

#227

Post by busymom0 »

iagooar wrote: post #199

I love my MacBook Pro M5 128GB RAM and I love qwen3.6. BUT DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise. Running any kind of sophisticated job on the very laptop you are using is just not viable. Sure you can use it in clamshell mode, but forget touching it while working with AI coding or ag…

Also look into buying the Mac mini refurbished from Apple. They come almost brand new, same warranty and you save money.

Re: Qwen 3.6 27B is the sweet spot for local development

#228

Post by manmal »

Arubis wrote: post #222
iagooar wrote: post #199

I love my MacBook Pro M5 128GB RAM and I love qwen3.6. BUT DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise. Running any kind of sophisticated job on the very laptop you are using is just not viable. Sure you can use it in clamshell mode, but forget touching it while working with AI coding or ag…

Don't forget that your OLED screen will start to color-shift as the heat cooks the panel!

There is no MacBook Pro with OLED (yet).

Re: Qwen 3.6 27B is the sweet spot for local development

#229

Post by ai_fry_ur_brain »

janalsncm wrote: post #100
onion2k wrote: post #20

None of the examples reflect 'real work', at least not what I'd consider real work. Being able to nail a zero-shot greenfield project is relatively easy even for a small model. There's not much context to build up and it can fall back to similar examples in the training data easily. So long as you're not asking it to invent something wholly new it'll probably manage. The real test is whether or not it can work with y…

> Being able to nail a zero-shot greenfield project is relatively easy even for a small model Not really germane to your comment but I hope I don’t sound old when I say I remember a time when spinning up a PoC was a week of work, and a statement like yours was pure science fiction.

Yeah, and we still do take a week for people that actually care.

If I start prompting away the core of a new project I lose interest in the entire thing almost straight away. I hate it. The next day I could care less about it. In fact it just makes me lazy, like a fat person who drives everywhere.

I love typing code and thinking for myself. Im going to continue to do that. I still dont know anyone who's shipped anything truly useful with this garbage tech, let alone with a local 30b param model. So much cope in these comments.

Spending 6k on hardware to run the worlds most mediocre model truly does make you an incredibly stupid person, so Im not really suprised by these comments of people saying these tiny models are helping them so much.

Its like a special needs kid all of sudden got the ability to code, of course they'd be impressed by basically all the code it produces.

Re: Qwen 3.6 27B is the sweet spot for local development

#230

Post by diseasedyak »

I have 24GB of VRAM (via a RTX 4090) and run Qwen3.6-35b:iq4, so it's importance-aware quantization and isn't nearly as dumb as it sounds like, fitting the 35b into 18 GB so you have some left over. So far I've had no issues, other than it taking a while for things like image gen, which I found out if you're gonna do with any alacrity, just have a cloud model do it.

For anything else local, including writing some automation scripts and such, it works great.

Post reply on HN