Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A guide to local coding models

aiforswes.com

Post reply on HN
Print view View original

281–290 of 363 posts

Re: A guide to local coding models

#281

Post by KronisLV »

baconner wrote: post #184
alok-g wrote: post #174

Earlier quoted context omitted.

I have heard about this approach elsewhere too. Could you please provide some more details on the set up steps and usage approach. I would like to replicate. Thanks.

There are a couple of decent approaches to having a planning/reviewer model set (eg. claude, codex, gemini) and an execution model (eg. glm 4.6, flash models, etc) workflow that I've tried. All three of these will let you live in a single coding cli but swap in different models for different tasks easily. - claude code router - basically allows you to swap in other models using the real claude code cli and set up som…

RooCode and KiloCode also have an Orchestrator mode that can create sub-tasks and you can specify which model to use for what - and since they report their results back after finishing a task (implement X, fix Y), the context of the more expensive model doesn’t get as polluted. Probably one of the most user friendly ways to do that.

A simpler approach without subtasks would be to just use the smart model for Ask/Plan/whatever mode and the dumb but cheap one for the Code one, so the smart model can review the results as well and suggest improvements or fixes.

Re: A guide to local coding models

#282

Post by sesm »

bjt12345 wrote: post #222

Here's my take on it though... Just as we had the golden era of the internet in the late 90s, when the WWW was an eden of certificate-less homepages with spinning skulls on geocities without ad tracking, we are now in the golden era of agentic coding where massive companies make eye watering losses so we can use models without any concerns. But this won't last and Local Llamas will become a compelling idea to use, pa…

Unfortunately, GPUs die in datacenters very quickly, and GPU manufacturers don't care about hardware longevity.

Re: A guide to local coding models

#283

Post by lostmsu »

deaux wrote: post #110
_delirium wrote: post #78

Earlier quoted context omitted.

As a third option, I've found I can do a few hours a day on the $20/mo Google plan. I don't think Gemini is quite as good as Claude for my uses, but it's good enough and you get a lot of tokens for your $20. Make sure to enable the Gemini 3 preview in gemini-cli though (not enabled by default).

Huge caveat: For the $20/mo subscription Google hasn't made clear if they train on your data. Anthropic and OAI on the other hand either clearly state they don't train on paid usage or offer very straightforward opt-outs. https://geminicli.com/docs/faq/ > What is the privacy policy for using Gemini Code Assist or Gemini CLI if I’ve subscribed to Google AI Pro or Ultra? > To learn more about your privacy policy and te…

Are you sure about OpenAI? I thought they actually do retain your agent chats (training I am less concerned about personally).

Anthropic has an option to opt out of training and delete the chats from their cloud in 30 days.

Re: A guide to local coding models

#284

Post by jszymborski »

kelvie wrote: post #170
jszymborski wrote: post #144

I just got a RTX 5090, so I thought I'd see what all the fuss was about these AI coding tools. I've previously copy pasted back and forth from Claude but never used the instruct models. So I fired up Cline with gpt-oss-120b, asked it to tell me what a specific function does, and proceeded to watch it run `cat README.md` over and over again. I'm sure it's better with other the Qwen Coder models, but it was a pretty fu…

gpt-oss-120b doesn't fit on a 5090 without offloading or crazy quants -- or did you mean you ran it via openrouter or something?

I'm running the MXFP4 [0] quants at like 10-13 toks/sec. It is actually really good, I'm starting to think its a problem with Cline since I just tried it with Qwen3 and the same thing happened. Turns out Cline _hates_ empty files in my projects, although they aren't required for this to happen.

[0] https://huggingface.co/blog/RakshitAralimatti/learn-ai-with-...

Re: A guide to local coding models

#285

Post by whitehexagon »

a_victorp wrote: post #6
nzeid wrote: post #2

I appreciate the author's modesty but the flip-flopping was a little confusing. If I'm not mistaken, the conclusion is that by "self-hosting" you save money in all cases, but you cripple performance in scenarios where you need to squeeze out the kind of quality that requires hardware that's impractical to cobble together at home or within a laptop. I am still toying with the notion of assembling an LLM tower with a f…

If you ever do it, please make a guide! I've been toying with the same notion myself

SimonW used to have more articles/guides on local LLM setup, at least until he got the big toys to play with, but well worth looking through his site. Although if you are in parts of Europe, the site is blocked at weekends, something to do with the great-firewall of streamed sports.

https://simonwillison.net/

Indeed, his self hosting inspired me to get Qwen3:32B ollama working locally. Fits nicely on my M1 pro 32GB (running Asahi). Output is a nice read-along speed and I havent felt the need for anything more powerful.

I'd be more tempted with a maxed out M2 Ultra as an upgrade, vs tower with dedicated GPU cards. The unified memory just feels right for this task. Although I noticed the 2nd hand value of those machine jumped massively in the last few months.

I know that people turn their noses up at local LLM's, but it more than does the job for me. Plus I decided a New Years Resolution of no more subscriptions / Big-AdTech freebies.

Re: A guide to local coding models

#286

Post by [deleted] »

qayxc wrote: post #274
BoredPositron wrote: post #253

Earlier quoted context omitted.

If you run fp32 maybe but no sane person does that. The tensor performance of the 3090 is also abysmal. If you run bf16 or fp8 stay away from obsolete cards. Its barely usable for llms and borderline garbage tier on video and image gen.

Actual benchmarks show otherwise. > The tensor performance of the 3090 is also abysmal. I for one compared my 50-series card's performance to my 3090 and didn't see "abysmal performance" on the older card at all. In fact, in actual real-world use (quantised models only, no one runs big fp32 models locally), the difference in performance isn't very noticeable at all. But I'm sure you'll be able to provide actual numbe…

[deleted]

Re: A guide to local coding models

#287

Post by embedding-shape »

> because GPT-OSS frequently gave me “I cannot fulfill this request” responses when I asked it to build features.

This is something that frequently comes up and whenever I ask people to share the full prompts, I'm never able to reproduce this locally. I'm running GPT-OSS-120B with the "native" weights in MXFP4, and I've only seen "I cannot fulfill this request" when I actually expect it, not even once had that happen for a "normal" request you expect to have a proper response for.

Has anyone else come across this when not using the lower quantizations or 20b (So GPT-OSS-120B proper in MXFP4) and could share the exact developer/system/user prompt that they used that triggered this issue?

Just like at launch, from my point of view, this seems to be a myth that keeps propagating, and no one can demonstrate a innocent prompt that actually triggers this issue on the weights OpenAI themselves published. But then the author here seems to actually have hit that issue but again, no examples of actual prompts, so still impossible to reproduce this issue.

Re: A guide to local coding models

#288

Post by embedding-shape »

rynn wrote: post #65
maranas wrote: post #9

Cline + RooCode and VSCode already works really well with local models like qwen3-coder or even the latest gpt-oss. It is not as plug-and-play as Claude but it gets you to a point where you only have to do the last 5% of the work

What are you working on that you’ve had such great success with gpt-oss? I didn’t try it long because I got frustrated waiting for it to spit out wrong answers. But I’m open to trying again.

> What are you working on that you’ve had such great success with gpt-oss?

I'm doing programming on/off (mostly use Codex with hosted models) with GPT-OSS-120B, and with reasoning_effort set to high, it gets it right maybe 95% of the times, rarely does it get anything wrong.

Re: A guide to local coding models

#289

Post by bilater »

If you are using local models for coding you are midwiting this. Your code should be worth more than a subscription.

The only legit use case for local models is privacy.

I don't know why anyone would want to code with an intern level model when they can get a senior engineer level model for a couple of bucks more.

It DOESN'T MATTER if you're writing a simple hello world function or building out a complex feature. Just use the f*ing best model.

Re: A guide to local coding models

#290

Post by andix »

snoman wrote: post #56
andix wrote: post #38

I wouldn't run local models on the development PC. Instead run them on a box in another room or another location. Less fan noise and it won't influence the performance of the pc you're working on. Latency is not an issue at all for LLMs, even a few hundred ms won't matter. It doesn't make a lot of sense to me, except when working offline while traveling.

Less of a concern these days with hardware like a Mac Studio or Nvidia dgx which are accessible and aren’t noisy at all.

I'm not fully convinced that those devices don't create noise at full power. But one issue still remains: LLMs eating up compute on the device you're working on. This will always be noticeable.
Post reply on HN