Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

news.ycombinator.com

Post reply on HN
Print view View original

561–570 of 620 posts

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#561

Post by moezd »

amarshall wrote: post #285
moezd wrote: post #128

Not yet. Without pure Apple game or decent GPUs, even with a lot of RAM and threads, all you get is about 30-50 tokens/second, and that's thinking turned off. Without these optimizations your model will have a field day with your MCPs, skills and agent descriptions and you will watch the paint dry before seeing the first output token. Local model serving means you have to fight for every token in your context window,…

Thinking doesn’t change output speed. Anthropic’s models are ~ 40–60 t/s median output speed.

Do you have access to Anthropic model weights to run them locally?

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#562

Post by SeriousM »

throw10920 wrote: post #530
robertlagrant wrote: post #452

Earlier quoted context omitted.

How are you sandboxing your Pi coding harness? Directly only mounting certain folders, using capabilities to kill the network and not giving it all your shell env vars, that sort of thing? Or do you use a tool?

And, is the sandboxing for security (avoid RCE on the host) or merely guardrails for the models? I've wanted the latter quite a bit for Pi, because weaker models like Deepseek V4 have extreme issues with obeying prompts (e.g. I'll instruct it to find a bug but not fix it, and it'll "helpfully" try to fix it anyway), so having a "read-only mode" actually backed by the OS would be very useful.

Haha, yes! Last time I asked it for options how to tackle a task and only do the research without touching any code. With xhigh rasoning, it echoed the options that many times until it was convinced that option A is the better choice and started implementing it.

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#563

Post by thrownaway561 »

slvnx wrote: post #507
thrownaway561 wrote: post #229

I just use DeepSeekV4 Fast... It's cheap as hell. Currently my monthly usage has been 67M Ouput 51M Input Total $0.83 dollar. I honestly don't understand why people just don't use DeepSeek.

Do you use deepcode, or which cli and/or coding agent you use it with?

No... I just use the github copilot chat

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#565

Post by fransje26 »

whartung wrote: post #149

Will the inevitable M5 releases from Apple change this equation in any meaningful way? I'm waiting to swap out my last gen Intel iMac with a new M5 mini of some kind, with the eye to hopefully be able to run some models locally. I envision a mini (heh) arms race to simply swapping out an M(X-1) for an M(X) annually as this field shakes out.

> Will the inevitable M5 releases from Apple change this equation in any meaningful way?

No. Apple is also running out of RAM, so you will not have the RAM you need.

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#566

Post by yesb »

jodoherty wrote: post #125

I use pi with an RTX Pro 6000 Blackwell to run Gemma 4 31b to do all my agentic coding. I find it useful. This side project highlights a similar approach to how I scope and tackle projects at work now: https://git.theodohertyfamily.com/wg-wrap.git/tree/README.md https://git.theodohertyfamily.com/wg-wrap.git/tree/CASE_STUD... You have to apply a lot of careful architecture and TDD to your approach. Eliminate technical…

Tried out the wg-wrap tool, might come in handy from time to time. Neat that it was made with a local model.

Some issues:

1. `wg-wrap healthcheck` was all green even though unprivileged user namespaces was restricted via AppArmor (Debian). That check doesn't seem to work

2. DNS doesn't work (no domains resolve) if the config lists multiple servers e.g. DNS = 1.1.1.1, 1.0.0.1

3. Peer endpoints don't support domain names, only IP addresses

4. Minor: the tool doesn't add an implied /32 cidr prefix for single ip configs (common from some VPN providers).

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#567

Post by girvo »

gwerbin wrote: post #498
girvo wrote: post #293

Earlier quoted context omitted.

Right. Tokens/s decode isn't the most important thing to me: wall clock time for task completion is. And tracking all of that, on my GB10-based Asus box, Step 3.7 Flash at IQ4_XS beats Qwen 3.6 27B despite the latter having MTP, on all of my actual coding task evaluations in real codebases. Qwen seems better at one-shotting things based on vague prompts to an acceptable degree, but thats literally not what I use thes…

Do you think the choice of quantization matters that much for other models? I've seen a lot of discussion about different quantization and FP formats but I feel totally unequipped to make an informed decision about what to try. What's your evaluation setup like? It sounds like maybe the best thing to do is have a realistic evaluation that resembles your actual intended workload and workflow, and then just try everyth…

>What's your evaluation setup like? It sounds like maybe the best thing to do is have a realistic evaluation that resembles your actual intended workload and workflow, and then just try everything.

That is quite literally what I have setup :)

I have a few codebases I've written over the years that I attempt a suite of specific tasks: code analysis/bug finding, bug fixing, adding features, that kind of thing. I keep track of the results, including wall clock time

>Do you think the choice of quantization matters that much for other models

It hugely matters. Lots more than r/LocalLlama would have you believe, sadly. Some model architectures can handle more aggressive quantisation than others, and it's hard to know ahead of time.

Step handles it surprisingly well (sparse MoE models seem to generally, when the particular layers are chosen to be quantised carefully). Qwen 3.6 27B handles it okay, but FP8 was better... except annoyingly Qwen's official FP8 has worse KLD/perplexity numbers/accuracy than it otherwise should. RedHat's one was better in my testing, though not by a huge amount.

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#568

Post by girvo »

ttoinou wrote: post #497
girvo wrote: post #491

Earlier quoted context omitted.

It isn’t though, I’ve run both through a bunch of coding evals. You nearly certainly didn’t have the right sampling parameters or quantised the KV cache? Ds4 is impressive for what it is, but it loops and over thinks even more, burning massive wall clock time to not even get great outcomes. It’s also limited to a slow speed on my Spark

I tried a bunch of stuff with step 3.5 and step 3.7 maybe not as much as you. Could you tell me what parameters and launched you’re using ? Antirez ds4 flash q2-q4 works almost out of the box for me

To be fair: if you're happy with ds4 then IMO stick with it!

Step 3.7 is notably better than 3.5

1. Use the official StepFun GGUF, IQ4_XS - theirs is better tuned in my experience than the other quants

2. Temp 1.0 top_p 0.95 sampling parameters for reasoning/agentic coding

3. It's really quite important that you don't quantise the KV cache: it made a surprising amount of difference to the looping and over thinking I found, at least for the quantised version of the model. I'm using the full F16 for K, and Q8 for V

4. Note that it now supports `reasoning_effort: low|medium|high` in your chat_template_kwargs; this is super useful :)

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#569

Post by jodoherty »

yesb wrote: post #566
jodoherty wrote: post #125

I use pi with an RTX Pro 6000 Blackwell to run Gemma 4 31b to do all my agentic coding. I find it useful. This side project highlights a similar approach to how I scope and tackle projects at work now: https://git.theodohertyfamily.com/wg-wrap.git/tree/README.md https://git.theodohertyfamily.com/wg-wrap.git/tree/CASE_STUD... You have to apply a lot of careful architecture and TDD to your approach. Eliminate technical…

Tried out the wg-wrap tool, might come in handy from time to time. Neat that it was made with a local model. Some issues: 1. `wg-wrap healthcheck` was all green even though unprivileged user namespaces was restricted via AppArmor (Debian). That check doesn't seem to work 2. DNS doesn't work (no domains resolve) if the config lists multiple servers e.g. DNS = 1.1.1.1, 1.0.0.1 3. Peer endpoints don't support domain nam…

Hey, thanks for the feedback!

If I get some time to circle back, I'll be sure to incorporate these into some new tests and address them.

I want to set up a qemu-system emulator based testing approach so I can incorporate things like AppArmor and SELinux into end to end tests that include different environment configurations.

Part of that will be setting up software defined networking so I can have things like DNS and wireguard VPN servers in a box and then test and evaluate the wg-wrap behavior at the packet level.

Post reply on HN