Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A guide to local coding models

aiforswes.com

Post reply on HN
Print view View original

331–340 of 363 posts

Re: A guide to local coding models

#331

Post by SamDc73 »

nl wrote: post #176
SamDc73 wrote: post #145

Earlier quoted context omitted.

This older HN thread shows R1 running on a ~$2k box using ~512 GB of system RAM, no GPU, at ~3.5-4.25 TPS: https://news.ycombinator.com/item?id=42897205 If you scale that setup and add a couple of used RTX 3090s with heavy memory offloading, you can technically run something in the K2 class.

Is 4 TPS actually useful for anything? That's around 350,000 tokens in a day. I don't track my Claude/Codex usage, but Kilocode with the free Grok model does and I'm using between 3.3M and 50M tokens in a day (plus additional usage in Claude + Codex + Mistral Vibe + Amp Coder) I'm trying to imagine a use case where I'd want this. Maybe running some small coding task overnight? But it just doesn't seem very useful.

I only run small models (70b at my hardware gets me around 10-20 TOPS) for just random things (personal assistant kind of thing) but not for coding tasks.

For coding related tasks I consume 30-80M tokens per day and I want something as fast as it gets

Re: A guide to local coding models

#332

Post by SamDc73 »

BoredPositron wrote: post #204
SamDc73 wrote: post #145

Earlier quoted context omitted.

This older HN thread shows R1 running on a ~$2k box using ~512 GB of system RAM, no GPU, at ~3.5-4.25 TPS: https://news.ycombinator.com/item?id=42897205 If you scale that setup and add a couple of used RTX 3090s with heavy memory offloading, you can technically run something in the K2 class.

Stop recommending 3090s they are all but obsolete now. Not having native bf16 is a showstopper.

Even with something like a 5090, I’d still run Q4_K_S/Q4_K_M because they’re far more resource-efficient for inference.

Also, the 3090 supports NVLink, which is actually more useful for inference speed than native BF16 support.

Maybe if you're training bf16 matters?

Re: A guide to local coding models

#333

Post by BoredPositron »

SamDc73 wrote: post #332
BoredPositron wrote: post #204

Earlier quoted context omitted.

Stop recommending 3090s they are all but obsolete now. Not having native bf16 is a showstopper.

Even with something like a 5090, I’d still run Q4_K_S/Q4_K_M because they’re far more resource-efficient for inference. Also, the 3090 supports NVLink, which is actually more useful for inference speed than native BF16 support. Maybe if you're training bf16 matters?

That's a smart thing todo considering a 5090 has native tensor cores for 4bit precision...

Re: A guide to local coding models

#334

Post by qayxc »

BoredPositron wrote: post #294
qayxc wrote: post #274

Earlier quoted context omitted.

Actual benchmarks show otherwise. > The tensor performance of the 3090 is also abysmal. I for one compared my 50-series card's performance to my 3090 and didn't see "abysmal performance" on the older card at all. In fact, in actual real-world use (quantised models only, no one runs big fp32 models locally), the difference in performance isn't very noticeable at all. But I'm sure you'll be able to provide actual numbe…

To be clear, we are not discussing small toy models but to be fair I also don't use consumer cards. Benchmarks are out there (phoronix, runpod, hugginface or from Nvidias own presentation) and they say it's at least 2x on high and nearly 4x on low precision, which is comparable to the uplift I see on my 6000 cards, if you don't see the performance uplift everyone else sees there is something wrong with your setup and…

> To be clear, we are not discussing small toy models but to be fair I also don't use consumer cards.

> if you don't see the performance uplift everyone else sees there is something wrong with your setup and I don't have the time to debug it.

Read these two statements and think about what might be the issue. I only run what you call "toy models" (good enough for my purposes), so of course your experience is fundamentally different from mine. Spending 5 figures on hardware just to run models locally is usually a bad investment. Repurposing old hardware OTOH is just fine to play with local models and optimise them for specific applications and workflows.

Re: A guide to local coding models

#335

Post by satvikpendem »

brulard wrote: post #257
satvikpendem wrote: post #155

Earlier quoted context omitted.

No? First of all you can limit how much of the unified RAM goes into VRAM, and second, many applications don't need that much RAM. Even if you put 108 GB to VRAM and 16 to applications, you'll be fine.

How about the rest of the resources? CPU/GPU? Would your work not be affected by inference running?

AI doesn't really use much CPU. In a simple answer, no your work would not be affected.

Re: A guide to local coding models

#336

Post by satvikpendem »

kasey_junk wrote: post #269
satvikpendem wrote: post #160

Earlier quoted context omitted.

Sure, you're like me, you're not a vibe coder by the actual definition then. Still, the general trend I see is that a lot of actual vibe coders do try to get their product working, code quality be damned. Personally, same as you, I stopped vibe coding and actually started writing a lot of architecture and code myself first then allowing the LLM to fill in the features so to speak.

The issue is that your claim was that if you are using up tokens you are probably vibe coding. But I’ve not found that to be true at all. My actually engineered processes where I care the most is where I push tokens the hardest. Mostly because I’m using llms in many places in the sdlc. When I’m vibing it’s just a single agent sort of puttering along. It uses much fewer tokens.

> The issue is that your claim was that if you are using up tokens you are probably vibe coding.

I said "by and large" ie generally speaking. As I mentioned before, the exception does not invalidate the trend. I assume HN is more heavily weighted towards non-vibe-coders using up tokens like me and you but again, that's the exception to what I see online elsewhere.

Re: A guide to local coding models

#337

Post by sellmesoap »

nl wrote: post #142
sheepscreek wrote: post #131

Earlier quoted context omitted.

Not for Codex. Not even for Gemini/Antigravity! I am truly shocked by how much mileage I can get out of them. I recently bought the $200/mo OpenAI subscription but could barely use 10% of it. Now for over a month, I use codex for at least 2 hrs every day and have yet to reach the quota. With Gemini/Antigravity, there’s the added benefit of switching to Claude Code Opus 4.5 once you hit your Gemini quota, and Google i…

I do the same and agree this works well. It's worth noting that the Claude subscription seems notably less than the others. Also there are good free options for code review.

My first try at LLM coding was with Claude, got back confusing results for a hello world++ type test and ran out of credits in a couple of hours, asked for a refund all the same day. I'm slowly teaching myself prompt engineering on qwen3-coder, it goes in circles much like claude was, but at least it's doing that at the cost of electricity at the wall, I already had a GPU.

Re: A guide to local coding models

#338

Post by estimator7292 »

yoan9224 wrote: post #307

The cost analysis here is solid, but it misses the latency and context window trade-offs that matter in practice. I've been running Qwen2.5-Coder locally for the past month and the real bottleneck isn't cost - it's the iteration speed. Claude's 200k context window with instant responses lets me paste entire codebases and get architectural advice. Local models with 32k context force me to be more surgical about what I…

I'd only consider the GPU cost if you intend to chuck it in a dumpster after three years. Why not factor in the cost of your CPU and amortize your RAM and disks?

Those aren't useful numbers.

Re: A guide to local coding models

#339

Post by nl »

zarzavat wrote: post #198
nl wrote: post #176

Earlier quoted context omitted.

Is 4 TPS actually useful for anything? That's around 350,000 tokens in a day. I don't track my Claude/Codex usage, but Kilocode with the free Grok model does and I'm using between 3.3M and 50M tokens in a day (plus additional usage in Claude + Codex + Mistral Vibe + Amp Coder) I'm trying to imagine a use case where I'd want this. Maybe running some small coding task overnight? But it just doesn't seem very useful.

3.5-50M tokens a day? What are you doing with all those tokens? Yesterday I asked Claude to write one function. I didn't ask it to do anything else because it wouldn't have been helpful.

https://github.com/nlothian/Vibe-Prolog chews a lot of tokens.

Have a bunch of other side projects as well as my day job.

It's pretty easy to get through lots of tokens.

Re: A guide to local coding models

#340

Post by ashirviskas »

littlestymaar wrote: post #250
Workaccount2 wrote: post #12

I'm curious what the mental calculus was that a $5k laptop would competitively benchmark against SOTA models for the next 5 years was. Somewhat comically, the author seems to have made it about 2 days. Out of 1,825. I think the real story is the folly of fixating your eyes on shiny new hardware and searching for justifications. I'm too ashamed to admit how many times I've done that dance... Local models are purely fo…

> Local models are purely for fun, hobby, and extreme privacy paranoia I always find it funny when the same people who were adamant that GPT-4 was game-changer level of intelligence are now dismissing local models that are both way more competent and much faster than GPT-4 was.

Moon lander computers were also game changers. Does not mean I should be impressed by the compute of a 30 year old calcualator that is 100x more powerful/efficient in 2025 when we have stuff a few orders of magnitude better.

For simple compute, its usefulness curve is a log scale. 10x faster may only be 2x more useful. For LLMs (and human intelligence) its more quadratic, if not inverse log (140IQ human can do maths that you cannot do with 2x 70IQ humans. And I know, IQ is not a good/real metric, but you get the point)

Post reply on HN