Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A guide to local coding models

aiforswes.com

Post reply on HN
Print view View original

301–310 of 363 posts

Re: A guide to local coding models

#301

Post by Lapel2742 »

calgoo wrote: post #229
smcleod wrote: post #40

Earlier quoted context omitted.

I suspect Ollama is at least partly moving away open source as they look to raise capitol, when they released their replacement desktop app they did so as closed source. You're absolutely right that people should be using llama.cpp - not only is it truly open source but it's significantly faster, has better model support, many more features, better maintained and the development community is far more active.

Only issue I have found with llama.cpp is trying to get it working with my amd GPU. Ollama almost works out of the box, in docker and directly on my Linux box.

>Only issue I have found with llama.cpp is trying to get it working with my amd GPU.

I had no problems with ROCm 6.x but couldn't get it to run with ROCm 7.x. I switched to Vulkan and the performance seems ok for my use cases

Re: A guide to local coding models

#302

Post by groguzt »

bilater wrote: post #289

If you are using local models for coding you are midwiting this. Your code should be worth more than a subscription. The only legit use case for local models is privacy. I don't know why anyone would want to code with an intern level model when they can get a senior engineer level model for a couple of bucks more. It DOESN'T MATTER if you're writing a simple hello world function or building out a complex feature. Jus…

"senior engineer level model" is the biggest cope I've ever seen

Re: A guide to local coding models

#303

Post by jgalt212 »

Aurornis wrote: post #297
jgalt212 wrote: post #291

Earlier quoted context omitted.

I will use a local coding model for our proprietary / trade secrets internal code when Google uses Claude for its internal code and Microsoft starts using Gemini for internal code. The flip side of this coin is I'd be very excited if Jane Street or DE Shaw were running their trading models through Claude. Then I'd have access to billions of dollars of secrets.

> I'd be very excited if Jane Street or DE Shaw were running their trading models through Claude. Then I'd have access to billions of dollars of secrets. Using Claude for inference does not mean the codebase gets pulled into their training set. This is a tired myth that muddies up every conversation about LLMs

> This is a tired myth that muddies up every conversation about LLMs

Many copyright holders, and the courts would beg to differ.

Re: A guide to local coding models

#304

Post by bilater »

pcl wrote: post #293
bilater wrote: post #289

If you are using local models for coding you are midwiting this. Your code should be worth more than a subscription. The only legit use case for local models is privacy. I don't know why anyone would want to code with an intern level model when they can get a senior engineer level model for a couple of bucks more. It DOESN'T MATTER if you're writing a simple hello world function or building out a complex feature. Jus…

Or if you want to do development work while offline.

Good to have fallbacks but in reality most ppl ( at least in the west) will have internet 99% of the time.

Re: A guide to local coding models

#305

Post by bilater »

Aurornis wrote: post #297
jgalt212 wrote: post #291

Earlier quoted context omitted.

I will use a local coding model for our proprietary / trade secrets internal code when Google uses Claude for its internal code and Microsoft starts using Gemini for internal code. The flip side of this coin is I'd be very excited if Jane Street or DE Shaw were running their trading models through Claude. Then I'd have access to billions of dollars of secrets.

> I'd be very excited if Jane Street or DE Shaw were running their trading models through Claude. Then I'd have access to billions of dollars of secrets. Using Claude for inference does not mean the codebase gets pulled into their training set. This is a tired myth that muddies up every conversation about LLMs

lol yeah its weird to me why even ppl on HN can't wrap their heads around stateless calls.

Re: A guide to local coding models

#306

Post by a96 »

satvikpendem wrote: post #31
a_victorp wrote: post #6

Earlier quoted context omitted.

If you ever do it, please make a guide! I've been toying with the same notion myself

Jeff Geerling has (not quite but sort of) guides: https://news.ycombinator.com/item?id=46338016

Also worth looking is stuff from Donato Capitella : https://github.com/kyuz0 https://www.youtube.com/@donatocapitella https://llm-chronicles.com/ etc

Re: A guide to local coding models

#307

Post by yoan9224 »

The cost analysis here is solid, but it misses the latency and context window trade-offs that matter in practice. I've been running Qwen2.5-Coder locally for the past month and the real bottleneck isn't cost - it's the iteration speed. Claude's 200k context window with instant responses lets me paste entire codebases and get architectural advice. Local models with 32k context force me to be more surgical about what I include.

That said, the privacy argument is compelling for commercial projects. Running inference locally means no training data concerns, no rate limits during critical debugging sessions, and no dependency on external API uptime. We're building Prysm (analytics SaaS) and considered local models for our AI features, but the accuracy gap on complex multi-step reasoning was too large. We ended up with a hybrid: GPT-4o-mini for simple queries, GPT-4 for analysis, and potentially local models for PII-sensitive data processing.

The TCO calculation should also factor in GPU depreciation and electricity costs. A 4090 pulling 450W at $0.15/kWh for 8 hours/day is ~$200/year just in power, plus ~$1600 amortized over 3 years. That's $733/year before you even start inferencing. You need to be spending $61+/month on Claude to break even, and that's assuming local performance is equivalent.

Re: A guide to local coding models

#308

Post by lostmsu »

jwr wrote: post #203

I am still hoping, but for the moment… I have been trying every 30-80B model that came out in the last several months, with crush and opencode, and it's just useless. They do produce some output, but it's nowhere near the level that claude code gets me out of the box. It's not even the same league. With LLMs, I feel like price isn't the main factor: my time is valuable, and a tool that doesn't improve the way I work…

I did the same with recent stuff and so far gpt-oss-120b on high was the best with gpt-oss-20b on high close second.

Re: A guide to local coding models

#309

Post by SpaceManNabs »

I love that this article added a correction and took ownership in it. This encourages more people to blog stuff and then get more input for parts they missed.

The best way to get the correct answer on something is posting the wrong thing. Not sure where I got this from, but I remember it was in the context of stackoverflow questions getting the correct answer in the comments of a reply :)

Props to the author for their honesty and having the impetus to blog about this in the first place.

Post reply on HN