Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

news.ycombinator.com

Post reply on HN
Print view View original

451–460 of 620 posts

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#451

Post by anaisbetts »

chakspak wrote: post #173
lambda wrote: post #155

Earlier quoted context omitted.

This is very similar to my setup. Pi in a container (I do let it have network access, just no access to creds or anything, only the one directory that I'm working on at the time and my ~/.pi directory), talking to llama.cpp in another container. I'm on a Strix Halo 128 GiB unified memory laptop. I've never used the frontier models in earnest, I don't believe in using proprietary tools for my programming, so I can't r…

Hopefully this isn't off-topic, but your setup sounds just like mine, Strix Halo and (I'm assuming) llama.cpp on ROCm, and I'm finding that the Qwen hybrid models don't handle prompt caching and instead re-process the context in full on every turn. I'm wondering if you were able to solve this and how?

If you're hitting this you have a bug, this is not related to the model. Either your harness is editing the messages between turns incorrectly (i.e. it is not append-only), or sometimes this is because of llama.cpp bugs, but bet on the former. Setting up something like Tailscale's Aperture will let you capture the requests and then you can diff them.

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#452

Post by robertlagrant »

Greenpants wrote: post #112

I have! I care about data privacy and LLMs being free. I'm using the Pi coding harness but containerized and sandboxed, to make sure it's running completely offline. On my Mac Studio with 128GB RAM (or MacBook with 36GB RAM) I'm using Qwen3.6 35b, with only 3b active parameters so that it runs really fast. I've done a complete redesign for my website's homepage and blog with Django + Wagtail. The latter is interestin…

How are you sandboxing your Pi coding harness? Directly only mounting certain folders, using capabilities to kill the network and not giving it all your shell env vars, that sort of thing? Or do you use a tool?

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#453

Post by arjie »

mtone wrote: post #390
arjie wrote: post #347

Earlier quoted context omitted.

Vouched your comment. Very cool. What are you running on to get 190 tok/s? I get 400 tok/s at c=4 but c=1 is slower than you.

I am using the `voipmonitor/vllm:lucifer` docker from the RTX6K discord community discussed at the same link the other commenter posted. It is based around this PR https://github.com/vllm-project/vllm/pull/43477

Ah I’m on the same PR just behind. Thank you.

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#454

Post by arjie »

CamperBob2 wrote: post #386
arjie wrote: post #347

Earlier quoted context omitted.

Vouched your comment. Very cool. What are you running on to get 190 tok/s? I get 400 tok/s at c=4 but c=1 is slower than you.

Not OP, but I am seeing up to 260 tokens/second output at c=1 with the recipe at https://github.com/local-inference-lab/rtx6kpro/blob/master/... using 4x 6k cards. Average is more like 200. There may be a way to get the 2-bit quantized version running even faster on a pair of them.

Thank you. Useful to know. Clipped on top by reduce, I assume.

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#455

Post by stared »

Greenpants wrote: post #112

I have! I care about data privacy and LLMs being free. I'm using the Pi coding harness but containerized and sandboxed, to make sure it's running completely offline. On my Mac Studio with 128GB RAM (or MacBook with 36GB RAM) I'm using Qwen3.6 35b, with only 3b active parameters so that it runs really fast. I've done a complete redesign for my website's homepage and blog with Django + Wagtail. The latter is interestin…

Why I do like Qwen 3.6 35B A3B, I have found that the difference improvement of Qwen 3.6 27B is massive. Sure, it is 3x slower (https://github.com/stared/benching-local-llms-on-apple-silic...), but for the total development time it felt that still 27B is faster to get the goal.

Is it that in your case is it different?

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#456

Post by incrudible »

femto113 wrote: post #367
fjdjshsh wrote: post #343

Earlier quoted context omitted.

>I'm still a AI skeptic What does this mean in June 2026 wrt coding? To me it sounds like being a "rice cooker skeptic". Some people don't like using rice cookers, some do.

For me the distinction is that your rice only needs to be edible once, while your code may need to last for decades. Using AI to code anything I could comfortably throw away if needed is a lot less fraught than letting it make choices that I and anybody who inherits the code is gonna have to live with, especially if by outsourcing those choices I reduce my understanding of the implications of those choices.

I think the idea that code should last decades is now questionable, if not problematic. If we can now produce code at 10x the rate, that means we can have 10x more code (probably not desirable) or we can have 10x as many revisions. Whoever inherits the code can have it rewritten to their liking and understanding. Nothing helps better in understanding a system than to rebuild it, even if just by handholding an LLM.

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#458

Post by c16 »

My experience so far has been Qwen3.5:32b-a3b-coder via Claude Code on a MBP 64gb M4, and a MBP 32gb M5. Just found about qwen3.6 so downloading that currently.

on 64GB M4 I find it's able to do things fairly well. The few times I run out of tokens, I hop over to that and I'm mostly unimpeded. I compare it to the Haiku models, where you have to go in and be surgical about your changes, or like others have said, guide a junior.

on 32GB M5, I find that it works, but around the 30% ctx threshold it slows down quite substantially, so more need to be surgical in your requests. I'll often just have my IDE open and Claude. But maybe I've been too comfortable talking to Sonnet/Opus and so forget I need to be more deliberate in my requests.

My finding here is that the harness is a big part of the problem. CC seems to be very good with Qwen in my experience. Better than OpenCode.

I also run DeepSeek for some other non-structured data tasks and to generate a to-do out of that. That's not coding, so won't go into that, other than to say it's very competent as a small model left to run in the background and automate small parts of my life and process.

tl;dr it's totally doable on a 32gb mbp using ollama, but be precise in your requests and guidance.

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#459

Post by hparadiz »

bandrami wrote: post #428
hparadiz wrote: post #187

Earlier quoted context omitted.

I am right there with you. Mind-boggling. It's a indistinguishable from magic technology!! I tried running some basic tasks through Qwen with Opencode on a 10 year old dual Xeon server for shits and giggles. I gave it a simple task like "use ffprobe first but convert this webm to mp4" and it was able to complete the task with zero network calls outside my network. On 10 year old hardware. It took about 3 minutes to c…

> You're gonna be googling the CLI switches for at least 10 minutes So there's this really amazing program called "man"

Yea there's something called a phone book too.

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#460

Post by ttoinou »

girvo wrote: post #293
electronsoup wrote: post #189

Earlier quoted context omitted.

> It gets into loops quite often, and surprisingly often gets the edit tool call wrong I find that running better quantization, like Q8 tend to prevent this even though its a bit slower to run, it saves overall time with less churn Using 3.6-27b is even slower again than 3.6-35b, but I find the accuracy really pays off

Right. Tokens/s decode isn't the most important thing to me: wall clock time for task completion is. And tracking all of that, on my GB10-based Asus box, Step 3.7 Flash at IQ4_XS beats Qwen 3.6 27B despite the latter having MTP, on all of my actual coding task evaluations in real codebases. Qwen seems better at one-shotting things based on vague prompts to an acceptable degree, but thats literally not what I use thes…

I tried Step 3.7 Flash on my mac 128GB and it seemed very dumb. antirez ds4 flash is much better !
Post reply on HN