Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local models is good now

vickiboykis.com

Post reply on HN
Print view View original

501–510 of 651 posts

Re: Running local models is good now

#501

Post by larodi »

fc417fc802 wrote: post #449
aamoscodes wrote: post #238

Earlier quoted context omitted.

You can pay, and also use deepseek-v4-flash. OpenRouter even lets you "block" or limit your usage to providers that don't train on data. Since the weights are open, other companies are already serving the model on non-DeepSeek owned hardware: https://openrouter.ai/deepseek/deepseek-v4-flash

> OpenRouter even lets you "block" or limit your usage to providers that don't train on data. More than that, they have various zero data retention options and provide a convenient json list of them.

The fact OpenRouter strips https to reroute screams danger already.

Re: Running local models is good now

#502

Post by dofm »

zwaps wrote: post #477
johndough wrote: post #402

Earlier quoted context omitted.

Article 53 of the AI Act: https://ai-act-law.eu/article/53/ The definition of a "genral-purpose AI model" is described in more detail in the "Guidelines on the scope of obligations for providers of general-purpose AI models under the AI Act": https://ec.europa.eu/newsroom/dae/redirection/document/11834...

Does not apply to oss models

Does it not apply to hosting and running them for money? How would it not?

Re: Running local models is good now

#503

Post by dofm »

nl wrote: post #480
johndough wrote: post #381

Earlier quoted context omitted.

I see that OVH offers Qwen3.5-397B-A17B, which is a bit surprising to me. I thought that EU providers had to comply with the AI act where you have to provide opt-out and information about the training data once the model is sufficiently large (over 10^23 FLOPs, likely the case here), but providing information is not possible since people who train those models only give vague information at best. Does anyone know if…

OVH is acting as a "Deployer", not a "Provider", which have special meaning under the AI Act. There are much less (almost no) disclosure regulations on the deployer. https://ethicalogic.com/articles/gpai-guide-roles-public-dat...

Pretty convenient, it must be noted, for a market that does not have any meaningful home grown models.

Re: Running local models is good now

#504

Post by wraptile »

sathackr wrote: post #30
rmunn wrote: post #7

This is the kind of thing that Anthropic et al should be worried about. As it becomes easier and easier to run local models, the ceiling of what they'll be able to charge will get lower and lower. Not that nobody will be willing to pay $$$$$ per month, but a lot of people are going to multiply the per-month charge by 12 or 24 and say "Could I set up a local model for less than that, and have it pay for itself within…

The opposite of that has been happening for 20 years now with cloud compute. It won't happen with AI models either. It's almost ingrained in the American business model now. Outsource everything. Nobody wants to manage a room full of servers when they can spend 2-3x as much and outsource that headache along with the responsibility for it. Same will happen with AI. Whether that means paying Anthropic that premium or p…

> It won't happen with AI models either.

AI is definitely different. Cloud compute is incredibly convenient to the point where even if AWS is more expensive it's just so _nice_. LLM models are much more abstract and while I can't easily swap AWS for Hetzner to save 80% of my costs I can absolutely get close to that for many of LLM tasks, even today.

I suspect Anthropic and gang all know that that's why they are buying up dev tools and shifting towards long-running agents because that's where they can get AWS's "nicesness" that they can charge for.

Re: Running local models is good now

#505

Post by jnaina »

Running Qwen3-30B-A3B-Instruct-2507-AWQ-4bit on an Olares One with NVIDIA GeForce RTX 5090 Mobile GPU (24GB GDDR7 VRAM) and an Intel Core Ultra 9 275HX processor.

Plenty fast for coding work and for sharing with my OpenClaw setup.

Currently in the process of adding another external GPU (RTX 4090 with pipeline parallelism) via thunderbolt 5 to the Olares One box, for higher quantization, possibly 8-bit, larger context, better concurrency, more kv cache.

Re: Running local models is good now

#507

Post by fc417fc802 »

larodi wrote: post #501
fc417fc802 wrote: post #449

Earlier quoted context omitted.

> OpenRouter even lets you "block" or limit your usage to providers that don't train on data. More than that, they have various zero data retention options and provide a convenient json list of them.

The fact OpenRouter strips https to reroute screams danger already.

What do you mean? Are you objecting that they communicate with the provider on your behalf? But how else would you design such a system?

Plumbing you straight through would require nonstandard certificate juggling and they wouldn't be able to implement their core service of providing a standardized API nor could they transparently route your request to the fastest / cheapest / whatever provider on the fly nor could they implement transparent fallback nor could they implement their policy of not billing you if the response from the provider is invalid.

Also the chosen provider could fingerprint your network stack if you communicated directly. The routing service is acting as a proxy and for most providers fully anonymizes requests (it does send a stable uid to some of them though).

Re: Running local models is good now

#509

Post by locknitpicker »

c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

> I don't know about good, I use a lot of local models and they're still pretty painful to run locally

You are somehow assuming cloud-based models are not painful.

I can tell you my past experience. I was using GPT 5.5 and Claude Opus interchangeably and I prompted them to implement a feature. I paid attention to the agent window and it was literally screwing up implementations, causing tests to fail, and going into test-fail-fix loops to clean up after itself. After a few minutes, it finally called it done. That run cost $0.60.

I went to review the code and only half of the source files complied with the instruction files. I prompted the model to clarify why it failed to comply with the instruction file. The model outputs "you are right, I should have complied with the instruction files. That prompt cost $0.30.

I prompted the model to proceed and apply the instruction file prompts. It went ahead and applied changes. Success. It cost $0.16.

I reviewed the code again. Only half of the sloppy code was touched up. I prompted it to fix the whole mess, not just a couple of files. It complied. One coin less in my purse.

So, around a third of the cost of a feature is spent on the model cleaning the mess it left in it's wake.

And this was a tiny feature with a plan, a solid set of instruction files.

Very expensive.

Are costs going down? I doubt so. OpenAI seems to still be spending 3 times it's revenue already.

In comparison, local models sound very good.

Re: Running local models is good now

#510

Post by jnaina »

iagooar wrote: post #115

I love running two models locally: qwen3.6 27B 8bit (dense) and qwen3.6 35B 4bit (MoE). The 27B is the smarter, more reliable one - but it is slower. The 35B is faster, still very smart but below 27B, a bit less reliable. The reason is the MoE - Mixture of Experts architecture, which only activates a subset of parameters, making the model much much faster. I run the 27B on a MacBook Pro M5 Max + 40 GPU cores + 128GB…

how are you connecting the 35B model to your mailbox, for email classification?
Post reply on HN