Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local models is good now

vickiboykis.com

Post reply on HN
Print view View original

511–520 of 651 posts

Re: Running local models is good now

#511

Post by hypfer »

linuxhansl wrote: post #460
hypfer wrote: post #15

After having been a happy user of Qwen3.6-27B for a few weeks, due to being away from the hardware, I'm currently forced to use Claude Sonnet 4.6 It is such a downgrade. I don't understand how that's even possible. The thing has so many strongly-held opinions I did not ever ask it for, talking just way too much and generally feeling somehow dumber. Of course, being significantly larger, it will encode more knowledge,…

How qwen3.6:27b compare to qwen3.6:35b-a3b (MoE) in your experience (if you tried). I find the dense models are way too slow on my H/W.

I don't really like them tbh.

They are a lot faster, yes, but at least for my needs, they feel a lot dumber too.

It's a shame that the gemma4 that _would_ fit 24GB VRAM is MoE. :(

Re: Running local models is good now

#512

Post by suncemoje »

indoordin0saur wrote: post #347
lelanthran wrote: post #300

Earlier quoted context omitted.

> Not a big deal if your app is for sharing cat videos, but if you're medical or wealth management or a government contractor or the like enterprise clients really like to see good data security policies. If this mattered to them, they wouldn't be running so much in the cloud or in proprietary software that they have no ability to air-gap. If companies ever cared about this, Windows would not be dominant on the deskt…

There are a lot of government jobs I know of that are absolutely air-gapped. Your computer has basically no internet access, everything is stored on-prem. Hedge funds also tend to be extremely locked down, from what I saw when I interviewed. With certain data sets either having strict encryption-in-transit or a being stored in a quirky on-prem service. I can't imagine they're going to be dumping their data into Claud…

> There are a lot of government jobs I know of that are absolutely air-gapped. Your computer has basically no internet access, everything is stored on-prem.

I wonder if that's because they don't know better or because of a lack of trust or costs?

Re: Running local models is good now

#513

Post by fmap »

frollogaston wrote: post #260
rmunn wrote: post #7

This is the kind of thing that Anthropic et al should be worried about. As it becomes easier and easier to run local models, the ceiling of what they'll be able to charge will get lower and lower. Not that nobody will be willing to pay $$$$$ per month, but a lot of people are going to multiply the per-month charge by 12 or 24 and say "Could I set up a local model for less than that, and have it pay for itself within…

Anthropic isn't just renting out compute, they're renting out a closed model that's better than anything you can download for free. So they're rightfully focused on preventing others from distilling their model.

It's in Anthropic's best interest to focus the conversation on "distillation".

Imo the more interesting thing to focus on is that there are now several more labs with the expertise and capabilities to train trillion parameter models. That's a serious technical accomplishment and the main reason why open models are catching up to Anthropic and OpenAI (and local models are typically distillations of much larger models).

Who cares that they got some small amount of training data out of Claude. The crux is that the big US labs are not special, they just have a first mover advantage that's slowly shrinking as incremental progress becomes harder.

Re: Running local models is good now

#514

Post by rimliu »

bityard wrote: post #154
rmunn wrote: post #7

This is the kind of thing that Anthropic et al should be worried about. As it becomes easier and easier to run local models, the ceiling of what they'll be able to charge will get lower and lower. Not that nobody will be willing to pay $$$$$ per month, but a lot of people are going to multiply the per-month charge by 12 or 24 and say "Could I set up a local model for less than that, and have it pay for itself within…

The general consensus is that local models will continue to improve drastically, but hosted models will as well. There will _always_ be a pretty big gulf of capability between what you can do with a desk full of hardware at home vs a few racks of hardware in a datacenter. That seems to be the real "moat" of hosted models at this point in time: access to capital. What's interesting/exciting is that local models are _a…

Nothing will improve drastically anymore. And when big ones run out of money to burn, who will train your local models?

Re: Running local models is good now

#515

Post by NamlchakKhandro »

c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

Pi mono is king. Everything else is hypetrash.

If I can't customise it then I won't waste my time using it it getting use to it.

Claude code is trash, it's customisability is extremely shallow, open code, codex, copilot, Kiro, etc etc... all trash. Yes even open code..

If open code was so awesome then open claw would have been based on it... But it wasn't. That's should tell you everything you need to know.

Re: Running local models is good now

#516

Post by hypfer »

hypfer wrote: post #31
derethanhausen wrote: post #26

Earlier quoted context omitted.

I would not generalize based on experiences with Sonnet. The flagship models (Opus being the claude equivalent) are dramatically better.

Opus in my experience is equally unpleasant "character"-wise, but at least it actually gets stuff done more often, so it's at least slightly more earned at that. It's still a neurotic cargo-culting dogmatic idiot, but one that at least sometimes does produce deliverables instead of only bottom-tier HN-esque opinions. Hmm. I think I might just fundamentally disagree with Anthropic about the idea of what a "tool" shoul…

This morning I have been blessed with an example of the exact behavior that is so infuriating.

> But re-reading the comment:

> "In the real world however, it does not. Hence, in the future we might fail this check even if it works within this limited check."

> The comment says "we might fail this check even if it works" — implying the original intent was to always fail (return 1) as a conservative stance, leaving room to tighten it later. The new code does the opposite of what the comment describes.

> So either:

> 1) The comment needs updating to reflect the new intent, or

> 2) The logic is wrong and should still return 1 unconditionally if the original intent was preserved.

> Which is it — did you intentionally want to succeed when X works, making the comment outdated? Or should the return 1 be restored?

> User: did you see the "in the future"?

> Yes, fair point — "in the future we might fail this check" is forward-looking, not a statement of current intent. The comment is saying the door is left open to tightening the check later, not that it should currently always fail. So the current code is correct and the comment still holds. All good.

___

It reads/comprehends 5% of what is on screen but acts like it knows it all, bossing you around. Why is it like this. What's wrong with Anthropic and what's wrong with people _not_ getting incredibly angry at that unearned smartass.

Re: Running local models is good now

#517

Post by EnPissant »

c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

When running on a GPU, dense models are shaping up to be the best way due to two things:

- Maximum intelligence per VRAM (you dont have much)

- Dense models can benefit from MTP to get an almost 2x speedup in decode (ie, a 27b dense model with mtp decodes at about the same speed as a MoE model with 14b active param model would). This is important because local llm rarely has parallel streams to batch together.

When running on large unified memory like Strix Halo or Spark Dgx, MoE models are usually best:

- You can get similar intelligence as a smaller dense model with fewer active params (to compensate for the slower memory) by throwing ram at the problem.

Re: Running local models is good now

#518

Post by agile-gift0262 »

segmondy wrote: post #100

It's more than good. As of today, it's great. Those models listed in the blog are horrible compared to what you can run today, There's absolutely no reason to run those, you have Qwen3.6, Gemma4, and plenty other sized comparable models. If you're resourceful, you can even run SOTA models. KimiK2.7, MiMo-V2.5/V2.5-Pro, MiniMax2.5/2.7/3, DeepSeekV3.1/v3.2/V4-Flash/V4Pro, GLM5.1, Step3.7-Flash, Qwen3.5-397B, Qwen3.5-12…

> Qwen3.5-122B

do you find Qwen3.5-122B to be SOTA-level? I moved from it to Qwen3.6-27B (both Q8), and I prefer 3.6-27B, and it leaves me room to spare for other small models

Re: Running local models is good now

#520

Post by EnPissant »

girvo wrote: post #447
htrp wrote: post #434

Earlier quoted context omitted.

I've seen the same, Sparks are great at non time-sensitive tasks. if you can set up a agentic loop that does not require human intervention, you can design around the memory bandwidth limitations

The other benefit is that speculative decoding literally trades compute to make up for low bandwidth, so MTP/EAGLE/DFlash are unreasonably effective on the GB10 IMO, as long as your use case fits it. I’m getting 40tk/s decode with 1000+tk/prefill with a 198B-A11B model on mine

I thought MTP wasn't very useful on MoE models because the expert overlap for 2 tokens was too small.
Post reply on HN