Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local models is good now

vickiboykis.com

Post reply on HN
Print view View original

391–400 of 651 posts

Re: Running local models is good now

#391

Post by dofm »

echelon wrote: post #310
spockz wrote: post #249

Earlier quoted context omitted.

For what it is worth, I’m on a similar machine. (9070XT,5900X) and found a lot of performance improvement over ollama by compiling llama.cpp and running with —no-mmap and —perf. The context is still quite small though. With online models I use contexts of at least 200k which is useful for longer running/more complicated commands. Locally I haven’t gone much further than 8k. That is sufficient for small changes on sma…

I would rather we give up the idea of running open models on RTX cards and instead focus on running much bigger open models on H200s. 1. The hardware will eventually catch up. 2. This keeps the delta between frontier models smaller. 3. We can still fine tune and own the weights. 4. The models will be more useful, faster, and reliable. RTX is hobbyist tier, not professional tier. Gated cloud models from hyperscalers t…

Pressure on small model quality and design is absolutely what is needed. There are still gains to be made.

Re: Running local models is good now

#392

Post by girvo »

layer8 wrote: post #289
wincy wrote: post #250

Earlier quoted context omitted.

If I could just save up $6000 I could sell off my RTX 5090 for $4,000 and buy an RTX 6000 Blackwell Pro Workstation. I can fit models into the 32GB of vram but my context window ends up being tiny for any halfway capable model.

Isn’t the RTX 6000 Blackwell Pro Workstation over $13000 now?

And rising. It's depressing.

Re: Running local models is good now

#393

Post by dofm »

johndough wrote: post #381
rlkf wrote: post #368

Earlier quoted context omitted.

> Basically I want Hetzner and OVH to run open model clouds You can run Qwen3 on OVH already: https://www.ovhcloud.com/en/public-cloud/ai-endpoints/catalo... >

I see that OVH offers Qwen3.5-397B-A17B, which is a bit surprising to me. I thought that EU providers had to comply with the AI act where you have to provide opt-out and information about the training data once the model is sufficiently large (over 10^23 FLOPs, likely the case here), but providing information is not possible since people who train those models only give vague information at best. Does anyone know if…

Which law is that?

Not doubting you — just want to read it!

Re: Running local models is good now

#394

Post by angry_octet »

Programmers are used to paying nothing for tools. A basic laptop (SSD, multi core, 16GB of RAM) is hugely powerful if you are building in C/C++/Rust, even python. But all of a sudden it's no good, and we're back to using someone else's computer, hiring our tools every day. Worse, we get a different model every day, and maybe we aren't allowed to borrow the good tools some days because some mafioso are shaking down the manufacturer.

Most other trades need to invest significantly in tools. If you want good tooling, you really want 64GB of GPU memory (e.g. 2x 5090) and 96GB of RAM. If I'm paying $200k for an expert engineer then $50k every other year for tooling seems pretty reasonable.

Re: Running local models is good now

#395

Post by girvo »

ozim wrote: post #303
adam_arthur wrote: post #135

Earlier quoted context omitted.

Gemma 4 is particularly good at pipeline/automation tasks. It outperforms all the Qwen models (even 100B+) for rule following/automation style tasks in my experience. Its image interpretation is also very good, and out-benchmarks Opus. Qwen seems to ignore instructions and consistently outputs incorrect formats (when token generation format is not explicitly constrained) But yes, on the DGX Spark Gemma 31B Q4 with MT…

I was expecting DGX Spark to run Gemma 31b Q4 much faster. I was expecting it would run Q8 in 50 tok/s. I guess that’s good I stopped thinking about buying it because I would be disappointed.

I love my Spark-alike, but they really aren't inference boxes IMO. They're experimentation boxes. A couple of 3080 20GB's for cheap from China, a 5090, an RTX Pro 6000 if you can swing the horrible cost: those are better choices IMO

That said, I'm still running Step 3.7 Flash at ~40tk/s decode, 1000tk/s+ prefill on mine and its both very capable and fast enough

I got Gemma 31b to run on this at ~22tk/s decode at FP8 using MTP

Re: Running local models is good now

#396

Post by iwontberude »

citizenpaul wrote: post #309
c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

They are still terrible at tool usage which loses 99% of the effectiveness of the agent. I've had to concede and use paid frontier models that can use tools or its not worth using agents....copy...paste....copy....paste....

Your models aren’t big enough and they are forgetting about the tools. Try a larger model. If you can’t, then your rig was too underpowered anyways.

Re: Running local models is good now

#397

Post by girvo »

ggerganov wrote: post #83
hypfer wrote: post #15

After having been a happy user of Qwen3.6-27B for a few weeks, due to being away from the hardware, I'm currently forced to use Claude Sonnet 4.6 It is such a downgrade. I don't understand how that's even possible. The thing has so many strongly-held opinions I did not ever ask it for, talking just way too much and generally feeling somehow dumber. Of course, being significantly larger, it will encode more knowledge,…

I haven't spent a dime on cloud inference, so cannot make a direct comparison like you. But I can 100% attest to the fact that Qwen3.6-27B is a very capable local model for coding tasks. Over the last month and a half I've been using it almost daily, either on my M2 Ultra or on my RTX 5090 box. I use it for small mundane tasks at ggml-org [0] - nothing really impressive, but definitely a helpful tool for a maintainer…

> Currently, I have a very lightweight harness - the pi agent with everything stripped (`pi -nc --offline`) and a short system prompt [1] to align it a bit with my style

This really is the secret to getting the most out of these models IMO. Pi is so damned good. I have a strongly tuned Pi for running Step 3.7 Flash (IQ4_XS) and Qwen 3.6 27B (FP8)

Also, thank you for llama.cpp mate :)

Re: Running local models is good now

#398

Post by [deleted] »

hypfer wrote: post #60
indoordin0saur wrote: post #47

Earlier quoted context omitted.

Very curious what hardware you're running this on!

The same 24GB VRAM RTX 4090 I bought to play Cyberpunk 2077 with. Works perfectly fine in llama.cpp throwing 70+t/s at me with 128k q8 K/V context when using the IQ4_NL quant + MTP at q4 MTP K/V. Also leaving this here because you might find it useful: https://hypfer.github.io/will-it-fit-llama-cpp/

[deleted]

Re: Running local models is good now

#399

Post by girvo »

ggerganov wrote: post #153
celrod wrote: post #124

Earlier quoted context omitted.

What quant do you run it at? 32GB seems like cutting it close on the rtx 5090 if going 8b, but other commenters are saying 4b lobotomizes the model.

As a baseline, I run all models in Q8 [0] because I want to be confident that when I observe a problem, the root cause is not due to the quantization. However, in this specific case, I use Q8 on the mac and Q4 on the RTX machine because the latter does not fit the full context at Q8. So far, I don't have conclusive evidence that the Q4 quantization affects the quality in a significant way for this model and the tasks…

27B seems surprisingly resiliant to quantisation. Though my evals showed there was some impact to coding ability from 8 bit to 4 bit, it was less than I would've expected: and it was on task types that you've said above that you don't really do with these!

Re: Running local models is good now

#400

Post by Rzor »

hypfer wrote: post #60
indoordin0saur wrote: post #47

Earlier quoted context omitted.

Very curious what hardware you're running this on!

The same 24GB VRAM RTX 4090 I bought to play Cyberpunk 2077 with. Works perfectly fine in llama.cpp throwing 70+t/s at me with 128k q8 K/V context when using the IQ4_NL quant + MTP at q4 MTP K/V. Also leaving this here because you might find it useful: https://hypfer.github.io/will-it-fit-llama-cpp/

Can you fix MTP-GEMMA-4-26B-A4B-IT? It says the weights are 0.5 GB in size.

edit: nvm, I'm confusing models.

Post reply on HN