Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local models is good now

vickiboykis.com

Post reply on HN
Print view View original

411–420 of 651 posts

Re: Running local models is good now

#411

Post by pizzafeelsright »

devilsdata wrote: post #404
c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

Just to piggyback onto this comment; has anyone tried running multiple of these in conjunction? For example, having a Python script that has one of these orchestrate others, and offloads certain tasks to better/more powerful models, or even cloud models?

yes but then that defeats the purpose of 'local'

and if remaining local, the hardware required to run multiple poor models could be better spent running better models.

I have attempted to orchestrate using different models, loading and unloading, but the speed is not there and by the time mistakes are discovered considering the lack of quick iteration the results become worthless unless the task is trivial.

Re: Running local models is good now

#412

Post by diddid »

trouve_search wrote: post #183
adam_arthur wrote: post #135

Earlier quoted context omitted.

Gemma 4 is particularly good at pipeline/automation tasks. It outperforms all the Qwen models (even 100B+) for rule following/automation style tasks in my experience. Its image interpretation is also very good, and out-benchmarks Opus. Qwen seems to ignore instructions and consistently outputs incorrect formats (when token generation format is not explicitly constrained) But yes, on the DGX Spark Gemma 31B Q4 with MT…

On a 5090, gemma4 26B runs at 350TPS with the command below [1] and gemma4 31B is around 150TPS with a similar command. I'm really surprised how much slower a DGX spark is for the same price. 1. Here's my command. PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \ vllm serve cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit \ --dtype auto \ --gpu-memory-utilization 0.95 \ --kv-cache-dtype fp8 \ --enable-chunked-prefill \ --enable…

With the 5090 you need to buy the rest of the computer though, and the Dgx spark will run 1/4th as slow but use 1/5th the electricity. And the spark would be able to run things the 5090 just couldn’t, like the Qwen3.5 122b. Which is all just to say that for llm workflows there is no easy answer. And if you media generation it gets even more complicated.

Re: Running local models is good now

#413

Post by mohamedkoubaa »

codethief wrote: post #287
TkTech wrote: post #119

Earlier quoted context omitted.

For many companies (country-dependent) that's not really why they use cloud services vs purchasing. It's tax shenanigans and business process overhead. OpEx vs CapEx, and a small (%) bump in the huge AWS bill no one will even notice or a $30k+ invoice for hardware that has to go through rigorous review and 3 departments. Same reason people pay for things through the AWS marketplace (like Vanta) instead of having to g…

Good point. Maybe there'll be companies that maintain your on-premise GPU cluster just like there are companies that service the coffee machine in your office?

This is far more likely than everyone racking their own servers.

Re: Running local models is good now

#414

Post by dofm »

johndough wrote: post #402
dofm wrote: post #393

Earlier quoted context omitted.

Which law is that? Not doubting you — just want to read it!

Article 53 of the AI Act: https://ai-act-law.eu/article/53/ The definition of a "genral-purpose AI model" is described in more detail in the "Guidelines on the scope of obligations for providers of general-purpose AI models under the AI Act": https://ec.europa.eu/newsroom/dae/redirection/document/11834...

Thanks, v. interesting.

Re: Running local models is good now

#415

Post by mohamedkoubaa »

otabdeveloper4 wrote: post #372
sathackr wrote: post #30

Earlier quoted context omitted.

The opposite of that has been happening for 20 years now with cloud compute. It won't happen with AI models either. It's almost ingrained in the American business model now. Outsource everything. Nobody wants to manage a room full of servers when they can spend 2-3x as much and outsource that headache along with the responsibility for it. Same will happen with AI. Whether that means paying Anthropic that premium or p…

> in the American business model AI company valuations won't survive if they're only for the "American business model".

Exactly. American businesses aren't even particularly efficient or well run

Re: Running local models is good now

#417

Post by mohamedkoubaa »

CamperBob2 wrote: post #197
sathackr wrote: post #30

Earlier quoted context omitted.

The opposite of that has been happening for 20 years now with cloud compute. It won't happen with AI models either. It's almost ingrained in the American business model now. Outsource everything. Nobody wants to manage a room full of servers when they can spend 2-3x as much and outsource that headache along with the responsibility for it. Same will happen with AI. Whether that means paying Anthropic that premium or p…

outsource that headache along with the responsibility for it You know what gives me headaches? When I'm in the middle of a session and the model gets rug-pulled out from under me because somebody at the model provider didn't pay the Trump bill that month. Or when someone at the model provider decides that the curve-fitting algorithm in my graphics package looks a little too much like Skynet for comfort. Or when they…

What about inference suggests it naturally belongs in the cloud?

Re: Running local models is good now

#418

Post by [deleted] »

ngxson wrote: post #172
phainopepla2 wrote: post #166

Earlier quoted context omitted.

Why not just use DS V4 Flash for the small stuff? Very fast and extremely cheap.

The dsv4 flash is 158B params in total. It is possible to run locally but will require all my system RAM. Also, a lot of my day-to-day tasks perform the same on both small and bigger models: summarize a web page, draft a response, translations, quick web search, etc.

[deleted]

Re: Running local models is good now

#419

Post by mohamedkoubaa »

wuliwong wrote: post #44
rmunn wrote: post #7

This is the kind of thing that Anthropic et al should be worried about. As it becomes easier and easier to run local models, the ceiling of what they'll be able to charge will get lower and lower. Not that nobody will be willing to pay $$$$$ per month, but a lot of people are going to multiply the per-month charge by 12 or 24 and say "Could I set up a local model for less than that, and have it pay for itself within…

These local models can do some of the work the non-frontier models can do but for me, that's not worth much. If I am just using Sonnet 4.6, I can pretty much work all day on the $20/month plan. And Sonnet is still a way more powerful model than a one you could self host on an M2 mac. If things change to token usage billing for everyone, maybe I'll be singing a different tune but on a subscription, I don't think it ma…

What about when the gravy train stops and Sonnet is priced with some marine above the cost to provide it?

Re: Running local models is good now

#420

Post by mohamedkoubaa »

bityard wrote: post #154
rmunn wrote: post #7

This is the kind of thing that Anthropic et al should be worried about. As it becomes easier and easier to run local models, the ceiling of what they'll be able to charge will get lower and lower. Not that nobody will be willing to pay $$$$$ per month, but a lot of people are going to multiply the per-month charge by 12 or 24 and say "Could I set up a local model for less than that, and have it pay for itself within…

The general consensus is that local models will continue to improve drastically, but hosted models will as well. There will _always_ be a pretty big gulf of capability between what you can do with a desk full of hardware at home vs a few racks of hardware in a datacenter. That seems to be the real "moat" of hosted models at this point in time: access to capital. What's interesting/exciting is that local models are _a…

It the model is as good as composer, has a decent harness around it, and isn't incompetent at tool calls - it'll be useful at least as a sub agent for most workflows in perpituity.
Post reply on HN