Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local models is good now

vickiboykis.com

Post reply on HN
Print view View original

471–480 of 651 posts

Re: Running local models is good now

#471

Post by jhonof »

fragmede wrote: post #464
rapidfl wrote: post #462

Earlier quoted context omitted.

if we get to the stage where the AI hardware is a more of a commodity and usability becomes 10x simpler, then people may buy their own hardware and run local models. Everybody owns a car, washer, TV, etc today. Maybe one could finance a server-box/trailer costing $20k, trade it in every 7 years for a newer model, etc. Many people are going to own a $20k Optimus.

The car, TV, washer, and whatever humanoid robot finds product market fit physically need to be in my house, or close to it, in order for them to be useful to me. Thanks to the Internet, the data center doesn't need to be, like at all. Economy of scale says that renting a slice of time on the most expensive GPU supercomputer out there is going to be faster and also probably cheaper since I'd only be getting a slice w…

Why don't you just buy a chromebook or equiv and do that now then?

Re: Running local models is good now

#472

Post by aeonfox »

starshadowx2 wrote: post #386
rmunn wrote: post #7

This is the kind of thing that Anthropic et al should be worried about. As it becomes easier and easier to run local models, the ceiling of what they'll be able to charge will get lower and lower. Not that nobody will be willing to pay $$$$$ per month, but a lot of people are going to multiply the per-month charge by 12 or 24 and say "Could I set up a local model for less than that, and have it pay for itself within…

Earlier I was thinking it's maybe comparable to paying for Netflix vs torrenting and running Plex or something. For the majority of normal, mainstream users I feel like most would just pay for the thing that is already setup and ready for them. There'll still be all the more techy or determined types who will do it themselves, I just wonder what the percentages of both groups will be.

> I feel like most would just pay for the thing that is already setup and ready for them

Nothing stopping turnkey OSS AI hardware being productised, including niceties like opt-in automated updates. If the trend continues of models becoming smaller and more capable for everyday use, it also derisks against obsolescence.

Re: Running local models is good now

#473

Post by sosodev »

kristopolous wrote: post #442
sosodev wrote: post #35

I think this is overselling their capabilities. I've used Gemma 4 and Qwen 3.6 quite a bit on my strix halo home server. They're great models and the dense variants are significantly better, but they're still very far behind the frontier. If you boot up Gemma 4 MoE and OpenCode/Pi and expect to perform anything like Claude Code or Codex you're going to be very disappointed.

You need to switch out the prompts and work with it differently. I posted this yesterday https://github.com/day50-dev/petsitter I use it with https://github.com/day50-dev/simple-llm-cli And modify the "tricks" until my evals get to good numbers. It's a model by model basis. This is what the larger firms are doing - they have custom prompts per model

Petsitter's default tricks doesn't seem to do much for Qwen3.6, right? JSON mode could be useful I suppose, but that's not really going to make it better at writing code. Do you have any other example tricks? I'm having a hard time understanding how I would apply them.

Re: Running local models is good now

#475

Post by FridgeSeal »

echelon wrote: post #310
spockz wrote: post #249

Earlier quoted context omitted.

For what it is worth, I’m on a similar machine. (9070XT,5900X) and found a lot of performance improvement over ollama by compiling llama.cpp and running with —no-mmap and —perf. The context is still quite small though. With online models I use contexts of at least 200k which is useful for longer running/more complicated commands. Locally I haven’t gone much further than 8k. That is sufficient for small changes on sma…

I would rather we give up the idea of running open models on RTX cards and instead focus on running much bigger open models on H200s. 1. The hardware will eventually catch up. 2. This keeps the delta between frontier models smaller. 3. We can still fine tune and own the weights. 4. The models will be more useful, faster, and reliable. RTX is hobbyist tier, not professional tier. Gated cloud models from hyperscalers t…

Ah yes, because of all the people at home with computers who have…checks notes…datacentre GPU’s lying around.

Re: Running local models is good now

#476

Post by echelon »

what wrote: post #470
echelon wrote: post #362

Earlier quoted context omitted.

If you're doing professional work in coding or video, you can easily saturate a single H200. This is what RunPod-type services are for. For instance, ComfyUI is an abomination that can't do half of what Nano Banana and Seedance 2.0 can do. And you have to sit around and wait 10x longer for single results. I can rent an H200 for $3.50 an hour. That's INSANELY cheap. I do not understand this split between hosted APIs a…

> I’m a contrarian that says things that rile up the anti-AI folks That’s hardly contrarian here, lol.

Are we experiencing the same website?

I swear, two thirds of the folks here just make comments that dunk on AI. They underestimate it, hate it, hate those that use it, etc. It's the "old angry man yells at cloud" trope.

I've had so many consecutive days of "-4" karma posts that HN is blocking me from commenting. And the comment retorts I get from these folks are absolute gems that will undoubtedly age like milk.

Re: Running local models is good now

#477

Post by zwaps »

johndough wrote: post #402
dofm wrote: post #393

Earlier quoted context omitted.

Which law is that? Not doubting you — just want to read it!

Article 53 of the AI Act: https://ai-act-law.eu/article/53/ The definition of a "genral-purpose AI model" is described in more detail in the "Guidelines on the scope of obligations for providers of general-purpose AI models under the AI Act": https://ec.europa.eu/newsroom/dae/redirection/document/11834...

Does not apply to oss models

Re: Running local models is good now

#478

Post by azeirah »

aftbit wrote: post #165
c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

IMO running local models "well" still requires an expensive hardware investment. You really want 96GB of VRAM on a modern Blackwell arch to run these models with decent KV cache. Trying to run them on a unified memory Mac, an AI Max AMD processor, or a DGX Spark-alike is really just asking for trouble. Prefill kills perf. If you throw the right GPUs at the problem, they become much better - but still not quite in the…

Qwen 3.6 27B performs similarly to sonnet 4.5 (note I said 4.5, not 4.6) when it comes to coding. It runs amazingly well on my PC with a 7900xtx.

It's worse at general tasks, but in the precise domain of coding I actually prefer to use it over my claude subscription because it has 0 latency (and no privacy concerns whatsoever).

Re: Running local models is good now

#479

Post by jen20 »

darkmarmot wrote: post #251
rapind wrote: post #226

Earlier quoted context omitted.

> The best "free" experience I've found is using OpenCode with Big Pickle. I have absolutely zero interest in free. I honestly don't think I'm even remotely in the same demographic as people using free tiers / models. I want to pay. I don't want my data used for training. I want it to be open. I want it to be consistently up (more than Claude!). I want it to be fast. I don't want it to be subsidized as that's just an…

Hard to guarantee it's private if you don't keep it local... I don't have a lot of trust for companies in this space.

I trust AWS in this space. I'm 100% sure that they will be precisely honoring the terms of service for Bedrock (I've never looked to see whether they claim to train on your data though).

Re: Running local models is good now

#480

Post by nl »

johndough wrote: post #381
rlkf wrote: post #368

Earlier quoted context omitted.

> Basically I want Hetzner and OVH to run open model clouds You can run Qwen3 on OVH already: https://www.ovhcloud.com/en/public-cloud/ai-endpoints/catalo... >

I see that OVH offers Qwen3.5-397B-A17B, which is a bit surprising to me. I thought that EU providers had to comply with the AI act where you have to provide opt-out and information about the training data once the model is sufficiently large (over 10^23 FLOPs, likely the case here), but providing information is not possible since people who train those models only give vague information at best. Does anyone know if…

OVH is acting as a "Deployer", not a "Provider", which have special meaning under the AI Act.

There are much less (almost no) disclosure regulations on the deployer.

https://ethicalogic.com/articles/gpai-guide-roles-public-dat...

Post reply on HN