Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local models is good now

vickiboykis.com

Post reply on HN
Print view View original

311–320 of 651 posts

Re: Running local models is good now

#311

Post by KaoruAoiShiho »

hypfer wrote: post #31
derethanhausen wrote: post #26

Earlier quoted context omitted.

I would not generalize based on experiences with Sonnet. The flagship models (Opus being the claude equivalent) are dramatically better.

Opus in my experience is equally unpleasant "character"-wise, but at least it actually gets stuff done more often, so it's at least slightly more earned at that. It's still a neurotic cargo-culting dogmatic idiot, but one that at least sometimes does produce deliverables instead of only bottom-tier HN-esque opinions. Hmm. I think I might just fundamentally disagree with Anthropic about the idea of what a "tool" shoul…

Fable largely fixed the annoying chatterness so sucks that it's gone now.

Re: Running local models is good now

#312

Post by ryan_glass »

aftbit wrote: post #165
c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

IMO running local models "well" still requires an expensive hardware investment. You really want 96GB of VRAM on a modern Blackwell arch to run these models with decent KV cache. Trying to run them on a unified memory Mac, an AI Max AMD processor, or a DGX Spark-alike is really just asking for trouble. Prefill kills perf. If you throw the right GPUs at the problem, they become much better - but still not quite in the…

For a fraction of the price of 96GB vram, I built a desktop based on a supermicro server mobo and EPYC 9 series CPU, with just under 400GB rdimm ram (approx $4500 all in but this was before the ram price hike). Works really well for serving larger local modals at a decent enough speed (I consider anything more than 10 tokens/second usable and value accuracy over speed).

Re: Running local models is good now

#313

Post by ptx »

> Security: I run every Pi session in a Docker container and give it permissions only to bash so that it can’t run Python code or do web browsing

How does that work? The script in the post references the file "docker-compose.sandbox.yml", but I don't anything about what that file does.

The post that this one links to, that it's based on, says that Pi doesn't do proper sandboxing.

Presumably bash can still execute other binaries, otherwise it would be fairly useless. What stops it from executing Python? Or opening a network connection and downloading Python?

Re: Running local models is good now

#314

Post by sgt101 »

garciasn wrote: post #155
myaccountonhn wrote: post #132

Earlier quoted context omitted.

Its not going to be as good as Claude, but if you know what you're doing, it may be good enough to get your work done.

A highly skilled carpenter may be able to 'get work done' by banging nails in with a heavy-bottomed cocktail glass, doesn't mean it's not painful to do so when it is continuously breaking and leaving shards of glass all over the workshop for you to find every day for the rest of your life until you clean up the mess you made using the wrong tool for the job.

If someone comes into the workshop and takes all the tools (hello Donald) then having a cocktail glass to hand might be a bit of a lucky break.

(geddit?)

Re: Running local models is good now

#315

Post by [deleted] »

rapind wrote: post #226
saghm wrote: post #136

Earlier quoted context omitted.

This is basically my experience as well. I have a moderately recent but high spec desktop (Radeon 6900 XT with 16 GB VRAM, Ryzen 9 7900X 12-core, 64 GB system RAM), and I tried out some recommended models with ollama a month or two ago. Anything not geared specifically towards coding seemed to struggled with actually making tool calls instead of just stating the actions they would take without making them (and trying…

> The best "free" experience I've found is using OpenCode with Big Pickle. I have absolutely zero interest in free. I honestly don't think I'm even remotely in the same demographic as people using free tiers / models. I want to pay. I don't want my data used for training. I want it to be open. I want it to be consistently up (more than Claude!). I want it to be fast. I don't want it to be subsidized as that's just an…

[deleted]

Re: Running local models is good now

#316

Post by swatcoder »

frollogaston wrote: post #304
swatcoder wrote: post #299

Earlier quoted context omitted.

Same with many and their shop tools in other trades. Most hobbyists and many professionals could end up far ahead financially by leveraging makerspaces, tool rentals, and co-op shops or even by hiring out a professional to prep certain intermediates for them, but they get psychological value -- as well as flexibility, reliability, and resale opportunity -- from having their own well-outfitted shop. And they can affor…

Yeah but the local model doesn't have those advantages for the coding use cases, at least not yet. In theory you could post-train one on your codebase or something, but nobody cares to do that when any vanilla coding agent service can read and understand the whole thing better than a locally tuned free model. I was already being very generous towards the Mac in pretending it does the same thing as the paid service. A…

The shifting sands of commercial models or pay-per-use managed models are just really not appealing to a lot of people.

Most come with huge privacy concerns, total costs and availability are impossible to forecast very far out, and the specific behavior of frontier models in particular is not something anybody can rely on as those are subscription products that are subject behavior on their publisher's whims (whether from changing system prompts, new "safeguards", retired models, forced "updates", new regulations, etc).

It's quite hard to put a price on all that, and as more people find local models productive enough or develop curiosity to explore models, training, or harness-crafting in their own ways, the marginal cost of buying some shop hardware just sort of disappears into the budget noise for plenty enough people.

Re: Running local models is good now

#317

Post by famouswaffles »

embedding-shape wrote: post #145
zozbot234 wrote: post #108

Earlier quoted context omitted.

The issues around training diffusion models are well known among researchers. They're likely to not be feasibly scalable far beyond the 26B size of DiffusionGemma itself, and their lower quality compared to an equally-sized auto-regressive model (the usual one-token-at-a-time flow) is also a matter of broad consensus.

> They're likely to not be feasibly scalable far beyond the 26B size of DiffusionGemma itself I think people used to say the same about the 8B text-diffusion models too when they came out, like LLaDA. LLaDA2.0 seemingly claims 100B total / 6.1B active MoE diffusion (DiffusionGemma is also MoE). Not saying you're wrong about the current consensus, but it has a way of changing over time, might be a bit early to claim i…

Difficulty of scaling is not the only issue. Nobody is going to be particularly invested in scaling an architecture that has:

- consistently proven behind their auto-regressive counterparts in quality. Look at the dgemma benchmarks - pretty steep dropoffs and the more difficult the benchmark the worse the dropoff. That's not a good look and it's not like its some artifact of google's release. Every dllm is like this.

- And whose inference benefits are negated at scale. Transformers are still cheaper if you want to serve lots of users.

>"DiffusionGemma's speedup is designed for local and low-concurrency inference. In high-QPS cloud serving, autoregressive models can be deployed to saturate compute efficiently, so DiffusionGemma's parallel decoding offers diminishing returns and can result in higher serving costs"

Put yourself in the shoes of all the labs, even open source ones. Why would you put much effort into this ?

Re: Running local models is good now

#319

Post by MrLeap »

echelon wrote: post #310
spockz wrote: post #249

Earlier quoted context omitted.

For what it is worth, I’m on a similar machine. (9070XT,5900X) and found a lot of performance improvement over ollama by compiling llama.cpp and running with —no-mmap and —perf. The context is still quite small though. With online models I use contexts of at least 200k which is useful for longer running/more complicated commands. Locally I haven’t gone much further than 8k. That is sufficient for small changes on sma…

I would rather we give up the idea of running open models on RTX cards and instead focus on running much bigger open models on H200s. 1. The hardware will eventually catch up. 2. This keeps the delta between frontier models smaller. 3. We can still fine tune and own the weights. 4. The models will be more useful, faster, and reliable. RTX is hobbyist tier, not professional tier. Gated cloud models from hyperscalers t…

There's a lot more professionals that have RTX cards than H200s. You're inevitably see more development and experimentation on things actual humans have lmao.

Re: Running local models is good now

#320

Post by freehorse »

c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

> You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes

This is sadly also my experience. I wish we had some MoE models with a higher ratio of active parameters per total. My experience is that the newer MoE models that can run in a 64b laptop have too few active parameters to be useful outside narrower, specific tasks. Mixtral 8x7b was a 14b active parameter (56b total) MoE model a few years ago and was probably the best model one could run in that range for some time, but it is too old now.

I have been using the qwen 27b and it is great, but running a dense model like this in a macbook is a bit suboptimal, and i wish I could run sth faster than 15 tok/s.

Post reply on HN