Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local models is good now

vickiboykis.com

Post reply on HN
Print view View original

561–570 of 651 posts

Re: Running local models is good now

#561

Post by MaKey »

hughw wrote: post #97
StevenWaterman wrote: post #24

Earlier quoted context omitted.

Yep, I daily drive Qwen3.6-27B (including for work), have done pretty much since it came out. IMO it's the only (small-ish, local) model worth using, if you can run it. It might not be as good as Opus at "add X large feature" but I don't want that in a model. I want to do the thinking while it does the typing. And Qwen 3.6 27B is perfectly good at that (while in my experience models like the 35A3B and gemma are signi…

Just this morning I tweaked my single 3090 setup too: OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_CONTEXT_LENGTH=180000 and that fits in 23GB. [edited for format]

Friends don't let friends use Ollama: https://sleepingrobots.com/dreams/stop-using-ollama/

Re: Running local models is good now

#564

Post by nullc »

I'm a little mystified at people taking about qwen 3.6 27b/ gemma 31b being slow in one breath and then saying they're using a 16GB gpu in the next.

You do need to use sutable hardware.

I get 50tok/s from Qwen 3.6 27b with Q8 & MTP (I can get more aggregate tok/s in parallel rather than using MOE, but don't have enough memory for too many full sized contexts) and 100 tok/s with 35B-A3b Q8 (no MTP as it's not that useful with MOE) on a single workstation gpu that I spent 3k on a couple years ago.

These speeds are somewhat faster than what I've seen from commercial SOTA models, they're plenty fast for many applications.

Re: Running local models is good now

#565

Post by nullc »

EnPissant wrote: post #517
c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

When running on a GPU, dense models are shaping up to be the best way due to two things: - Maximum intelligence per VRAM (you dont have much) - Dense models can benefit from MTP to get an almost 2x speedup in decode (ie, a 27b dense model with mtp decodes at about the same speed as a MoE model with 14b active param model would). This is important because local llm rarely has parallel streams to batch together. When r…

> This is important because local llm rarely has parallel streams to batch together.

I think most people using agent-like usage could easily run any number of parallel streams pretty often, but you run out of vram for multiple KV caches, unfortunately.

Re: Running local models is good now

#566

Post by towledev »

pornel wrote: post #344

[meta] I wonder why people have such wildly different bar for what is "good" agentic coding? In a way, it's absolutely amazing that we've went from "Playing 'Set a Timer' on Apple Music" intelligence to something that may pass the Turing Test, but in practical terms the small models are still far from what I'd call "good" for more than a tech demo. To me, 7B models are just a fuzzy echo of Wikipedia. Gemma models at…

> may pass the Turing Test

Why do you say 'may'? Just curious. Surely you've got something

Re: Running local models is good now

#567

Post by angry_octet »

rsanek wrote: post #429
angry_octet wrote: post #394

Programmers are used to paying nothing for tools. A basic laptop (SSD, multi core, 16GB of RAM) is hugely powerful if you are building in C/C++/Rust, even python. But all of a sudden it's no good, and we're back to using someone else's computer, hiring our tools every day. Worse, we get a different model every day, and maybe we aren't allowed to borrow the good tools some days because some mafioso are shaking down th…

Who's paying the $50k? I don't see how it makes sense to pay that much for a home-grown setup when I could pay <$5k/year total for both of the two best frontier models at effectively unlimited usage.

This is for clients paying millions for deliverables, with high stakes deadlines. Expertise in the domain is in short supply. Data and model control is very important, so relying on AIaaS was already risky. You can LLM enable engineers without big AIaaS risks.

A 16C Ryzen 128GB with 96GB Blackwell is ~US$16k, quite reasonable for a worker billed at $300k. In fact so reasonable it's worth having AI enabled backend for lots of things.

Re: Running local models is good now

#568

Post by chrsw »

c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

The very understandable desire to not have to rely on huge, centralized companies or powers for tokens has clouded people's judgment on how well these local models actually perform. They've improving, which is great, but for real work I use the best models available right now because they're so much better than local models.

Re: Running local models is good now

#569

Post by xdertz »

rmunn wrote: post #7

This is the kind of thing that Anthropic et al should be worried about. As it becomes easier and easier to run local models, the ceiling of what they'll be able to charge will get lower and lower. Not that nobody will be willing to pay $$$$$ per month, but a lot of people are going to multiply the per-month charge by 12 or 24 and say "Could I set up a local model for less than that, and have it pay for itself within…

AI usage is very spiky and good models require very expensive hardware. Running locally would just result in it sitting idle ~90% of the time. I think renting will always be cheaper, for comparable performance at least.

Re: Running local models is good now

#570

Post by embedding-shape »

cheschire wrote: post #461
pornel wrote: post #344

[meta] I wonder why people have such wildly different bar for what is "good" agentic coding? In a way, it's absolutely amazing that we've went from "Playing 'Set a Timer' on Apple Music" intelligence to something that may pass the Turing Test, but in practical terms the small models are still far from what I'd call "good" for more than a tech demo. To me, 7B models are just a fuzzy echo of Wikipedia. Gemma models at…

Haves and have nots. We aren’t wealthy enough to have the hardware that would make this good. The people who have the money to buy a spare maxed out Mac mini just don’t get it. I see lots of folks with RTX 6000’s in threads like these. Or any RTX card that ends in “90”. Cloud AI is what allows the proles to participate in the broader AI conversation, but not these AI conversations.

As one of those folks with a 6000 Pro who tries out basically every local model I can get my hands on; local models still aren't ready to be relied on for larger-scale software engineering. They're quite capable when it comes to writing and editing code though.
Post reply on HN