Let's say I have a server with an h200 gpu at home. What's the best open model for coding I can run on it today? And is it somewhat competitive with commercial models like sonnet 4.5?
Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
121–130 of 196 posts
Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
#122Earlier quoted context omitted.
gpt-oss-120b keeps stopping for me in Codex. (Also in Crush.) I have to say "continue" constantly.
See https://news.ycombinator.com/item?id=45773874 (TLDR, you need to hard-code some inference parameters to be the right ones, otherwise you'd get really bad behaviour + prompting to get the workflow right)
Thanks. Did you need to modify Codex's prompt?
Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
#123What are folks motivation for using local coding models? Is it privacy and there's no cloud host you trust? I love local models for some use cases. However for coding there is a big gap between the quality of models you can run at home and those you can't (at least on hardware I can afford) like GLM 4.6, Sonnet 4.5, Codex 5, Qwen Coder 408. What makes local coding models compelling?
I don't run any locally, but when I was thinking about investing in a setup, it would just be to have the tool offline. I haven't found the online subscription models to be sufficiently and frequently useful enough beyond occasional random tedious implementations that I'd consider investing in either online or offline LLMs long-term, and I've reverted back to normal programming for the most part, since it just keeps…
Something to consider is using a middleman like openrouter, you can buy some credits and then use them at whatever provider through them - no subscription just payg. For a few ad hoc things you can put a few bucks in and not worry about some monthly thing.
Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
#124What are folks motivation for using local coding models? Is it privacy and there's no cloud host you trust? I love local models for some use cases. However for coding there is a big gap between the quality of models you can run at home and those you can't (at least on hardware I can afford) like GLM 4.6, Sonnet 4.5, Codex 5, Qwen Coder 408. What makes local coding models compelling?
Another reason along with the others is that the output quality of the top commercial models varies wildly with time. They start strong and then deteriorate. The providers keep changing the model and/or its configuration without changing the name. With a local open weights model, you can learn each model's strengths and it can't be taken away with an update.
Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
#125Let's say I have a server with an h200 gpu at home. What's the best open model for coding I can run on it today? And is it somewhat competitive with commercial models like sonnet 4.5?
That's still very limiting when comparing to commercial models. To be truly competitive with commercial offerings the bar is closer to 4-8x that for one node .
That said, maybe a quantized version of GLM 4.5 Air, but if we're talking no hardware constraints I find some of the responses from LongCat-Chat-Flash to be favorable over Sonnet when playing around with LMArena.
Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
#126Models gpt-oss-120b, Meta Llama 3.2, or Gemma (just depends on what I’m doing) Hardware - Apple M4 Max (128 GB RAM) paired with a GPD Win 4 running Ubuntu 24.04 over USB-C networking Software - Claude Code - RA.Aid - llama.cpp For CUDA computing, I use an older NVIDIA RTX 2080 in an old System76 workstation. Process I create a good INSTRUCTIONS.md for Claude/Raid that specifies a task & production process with a task…
What does the GPD Win 4 do in this scenario? Is there a step w/ Agent Organizer that decides if a task can go to a smaller model on the Win 4 vs a larger model on your Mac?
Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
#127Qwen3:32b on MBP M1 Pro 32GB running Asahi linux. Mainly command line for some help with armv8 assembly, and some SoC stuff (this week explaining I2C protocol). I couldnt find any good intro on the web-of-ads. It's not much help with Zig, but then nothing seems to keep up with Zig at the moment. I get a steady stream of tokens, slightly slower than my reading pace, which I find is more than fast enough. In fact I´d o…
Qwen 3 Coder is a 30B model but an MoE with 3B active parameters, so it should be much faster for you. Give it a try.
It's not as smart as dense 32B for general tasks, but theoretically should be better for the sort of coding tasks from StackExchange.
Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
#128If you're going to get a MacBook, get the Pro, it has a built-in fan, you don't want the heat just sitting there on the MacBook Air. Same with the Mac mini, get the studio instead, it has a fan, the Mini does not. I don't know about you but I wouldn't want my brand new laptop / desktop to be heating up the entire time I'm coding with 0 cool off. If you go the Mac route, I recommend getting TG Pro, the default fan set…
Correction, if you're going to get a Mac then get a Max or Ultra, with as much memory as possible, the increase in RAM bandwidth will make running larger models viable.
Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
#129FWIW I bought the M4 max with 128GB and it is useful for local LLMs for OCR, I don't find it as useful for coding (ala Codex / Claude Code) with local LLMs. I find that even with GPT 5 / Claude 4.5 Sonnet that trust is low, and local LLMs can lower that just enough to not be as useful. The heat is also a factor - Apple makes great hardware, but I don't believe it is designed for continuous usage the way a desktop is.
Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
#130Let's say I have a server with an h200 gpu at home. What's the best open model for coding I can run on it today? And is it somewhat competitive with commercial models like sonnet 4.5?
If you do, damn bro
I played around with renting H200s and coding with aider and gpt-oss 120b. It was impressive but not at the level of claude. I decided buying $30k worth of tokens made far more sense than buying 30k worth of one GPU.