Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local models is good now

vickiboykis.com

Post reply on HN
Print view View original

551–560 of 651 posts

Re: Running local models is good now

#551

Post by itissid »

markussss wrote: post #542
saghm wrote: post #136

Earlier quoted context omitted.

This is basically my experience as well. I have a moderately recent but high spec desktop (Radeon 6900 XT with 16 GB VRAM, Ryzen 9 7900X 12-core, 64 GB system RAM), and I tried out some recommended models with ollama a month or two ago. Anything not geared specifically towards coding seemed to struggled with actually making tool calls instead of just stating the actions they would take without making them (and trying…

My system is quite similar to your, my GPU is a 6950 XT and CPU a Ryzen 5 2600x, same amount of RAM, and I feel your pain. It sounds very similar to my experience from a few months ago. When it comes to tool calling, there are multiple possible issues; some models have borked templates bundled with the model file, some models are not trained on tool calling, some agent harnesses doesn't support the tool call output f…

Interesting. Making low latency correct tool calls correctly is pretty important in voice AI cascading models(STT LLM TTS). Realtime Models are still 2x the cost and there are only 2 providers openai and google that are in the race. For cost control it has to be cascading models

For llms Sadly the only model right now that fits the bill for LLM is GPT 4.1 and it’s standard in my stack because thinking models have unacceptable latency(>=1 sec) even though they are good at tool calling. The main issue with 4.1 is that it can make still mistakes and prompt prose has to be tuned quite a bit.

I wonder if any local models can be tuned to match the response time and tool calling while supporting many languages.

Re: Running local models is good now

#552

Post by onel »

c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

Agree with this. Open models are the future but currently they are a pain to run locally.

As painful as it is to admit, the future might be cloud inference from a trusted provider.

Re: Running local models is good now

#553

Post by [deleted] »

zozbot234 wrote: post #116
c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

Maybe we shouldn't be running these models on laptops with their thermally constrained form factor, and we shouldn't expect quick inference on a par with a large cloud-based platform either, at least not for near-SOTA model quality. It's still worth it to avoid becoming massively reliant on centralized services.

[deleted]

Re: Running local models is good now

#554

Post by girvo »

EnPissant wrote: post #520
girvo wrote: post #447

Earlier quoted context omitted.

The other benefit is that speculative decoding literally trades compute to make up for low bandwidth, so MTP/EAGLE/DFlash are unreasonably effective on the GB10 IMO, as long as your use case fits it. I’m getting 40tk/s decode with 1000+tk/prefill with a 198B-A11B model on mine

I thought MTP wasn't very useful on MoE models because the expert overlap for 2 tokens was too small.

Still helps, and Step 3.5/3.7 were specifically trained for MTP (in a weird triple layer/triple head fashion with a kind of unique architecture)

With the currently-in-PR implementation it doubles decode performance for all the tasks I've been testing it against, at in the worst case is still a 35% uplift, so on a box with heaps of compute and not much memory bandwidth, it's worth it in practice

Re: Running local models is good now

#555

Post by npodbielski »

jtbaker wrote: post #252
aftbit wrote: post #165

Earlier quoted context omitted.

IMO running local models "well" still requires an expensive hardware investment. You really want 96GB of VRAM on a modern Blackwell arch to run these models with decent KV cache. Trying to run them on a unified memory Mac, an AI Max AMD processor, or a DGX Spark-alike is really just asking for trouble. Prefill kills perf. If you throw the right GPUs at the problem, they become much better - but still not quite in the…

> Trying to run them on a unified memory Mac > but still not quite in the realm of Sonnet or DeepSeek 4 Flash these are not mutually exclusive anymore. DS4 has set the bar for me these days. https://github.com/antirez/ds4

Anybody tried it on Strix Halo?

Re: Running local models is good now

#556

Post by munksbeer »

jaapz wrote: post #549
suncemoje wrote: post #536

Earlier quoted context omitted.

Then I'm interested if there are any facts as to what ZDR actually means?

It can still mean Zero Data Retention - i just comes down to whether you trust the company to actually do what they promise. The fact that they've trained models on data that wasn't theirs does not make me trust them a lot when they make this claim.

When discussing this, may I ask (I know you are probably bored of the actual arguments), what does "trained models on data that wasn't theirs" actually mean in practice?

Again, I know these arguments have been done to death, but every human who reads source code that wasn't written by them, or views art that wasn't created by them, and practices against this art, is training their brain on data "that wasn't theirs".

They are frequently making a living doing so.

Is this distinction the scale, or is there actually a different more strict definition that we should be using as a common language to talk about this? As in, I should not even be reading certain source code if it is not licensed appropriate, or I will be in breach because I'm training myself illegally? And the same question for art, etc?

Re: Running local models is good now

#557

Post by Der_Einzige »

dofm wrote: post #503
nl wrote: post #480

Earlier quoted context omitted.

OVH is acting as a "Deployer", not a "Provider", which have special meaning under the AI Act. There are much less (almost no) disclosure regulations on the deployer. https://ethicalogic.com/articles/gpai-guide-roles-public-dat...

Pretty convenient, it must be noted, for a market that does not have any meaningful home grown models.

What are you talking about? Le Chaton Fat is the best 30T model no the planet!

Re: Running local models is good now

#558

Post by palisade »

pornel wrote: post #344

[meta] I wonder why people have such wildly different bar for what is "good" agentic coding? In a way, it's absolutely amazing that we've went from "Playing 'Set a Timer' on Apple Music" intelligence to something that may pass the Turing Test, but in practical terms the small models are still far from what I'd call "good" for more than a tech demo. To me, 7B models are just a fuzzy echo of Wikipedia. Gemma models at…

Those who have never known anything better are okay with much less. For example, anyone who used Fable when it came out are saying that it is very difficult to go back to lesser models now. Even our strongest aren't good enough in comparison.

Re: Running local models is good now

#559

Post by teknologist »

I found a tool that makes it easy to run Salvatore Sanfilippo's (Redis creator) ds4.c on a Mac: https://github.com/notatestuser/ds4-control

His program uses quantization, but is very optimised and has builds that can fit into 96GB of memory with great results.

DS4 Flash is usually my go-to for a lot of things these days, and I don't have to worry about a cloud model stopping or telling me it's concerned about my usage.

Re: Running local models is good now

#560

Post by angry_octet »

fragmede wrote: post #459
rsanek wrote: post #429

Earlier quoted context omitted.

Who's paying the $50k? I don't see how it makes sense to pay that much for a home-grown setup when I could pay <$5k/year total for both of the two best frontier models at effectively unlimited usage.

> best frontier models at effectively unlimited usage. It would've been easy to spend $5k on Fable in the short week it was available. If that's the direction things are going (we can assume GPT-6 to be if similar class) $5k's not going to get you "best frontier models at effectively unlimited usage".

I know some orgs that are already spending more on tokens than developer salary. That's what unlimited use leads to. Teams of agents running on expensive models, agents designing and running test suites with barely any oversight.
Post reply on HN