Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 196 posts

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#81

Post by jetsnoc »

CubsFan1060 wrote: post #60
jetsnoc wrote: post #49

Models gpt-oss-120b, Meta Llama 3.2, or Gemma (just depends on what I’m doing) Hardware - Apple M4 Max (128 GB RAM) paired with a GPD Win 4 running Ubuntu 24.04 over USB-C networking Software - Claude Code - RA.Aid - llama.cpp For CUDA computing, I use an older NVIDIA RTX 2080 in an old System76 workstation. Process I create a good INSTRUCTIONS.md for Claude/Raid that specifies a task & production process with a task…

What is the Agent Organizer you use?

It’s a Claude agent prompt. I don’t recall who originally shared it, so I can’t yet attribute the source, but I’ll track that down shortly and add proper attribution here.

Here’s the Claude agent markdown:

https://github.com/lst97/claude-code-sub-agents/blob/main/ag...

Edit: Updated from the old Pastebin link to the GitHub version. Attribution found: lst97 on GitHub

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#82

Post by Xenograph »

embedding-shape wrote: post #31
simonw wrote: post #30

I'd be very interested to hear from anyone who's finding local models that work well for coding agents (Claude Code, Codex CLI, OpenHands etc). I haven't found a local model that fits on a 64GB Mac or 128GB Spark yet that appears to be good enough to reliably run bash-in-a-loop over multiple turns, but maybe I haven't tried the right combination of models and tools.

I've had good luck with GPT-OSS-120b (reasoning_effort set to "high") + Codex + llama.cpp all running locally, but I needed to do some local patches to Codex as they don't allow configuring and setting the right values for temperature and top_p for GPT-OSS. Also heavy prompting via AGENTS.md was needed to get it to have similar workflow to GPT-5, it didn't seem to pick up that by itself, so I'm assuming GPT-5 been tr…

Would love for you to share the Codex patches you needed to make and the AGENTS.md prompting, if you're open to it.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#83

Post by garethsprice »

scosman wrote: post #46

What are folks motivation for using local coding models? Is it privacy and there's no cloud host you trust? I love local models for some use cases. However for coding there is a big gap between the quality of models you can run at home and those you can't (at least on hardware I can afford) like GLM 4.6, Sonnet 4.5, Codex 5, Qwen Coder 408. What makes local coding models compelling?

It's fun for me. This is a good enough reason to do anything.

I learn a lot about how LLMs work and how to work with them.

I can also ask my dumbest questions to a local model and get a response faster, without burning tokens that count towards usage limits on the hosted services I use for actual work.

Definitely a hobby-category activity though, don't feel you're missing out on some big advantage (yet, anyway) unless you feel a great desire to set fire to thousands of dollars in exchange for spending your evenings untangling CUDA driver issues and wondering if that weird smell is your GPU melting. Some people are into that sort of thing, though.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#85

Post by firefax »

james2doyle wrote: post #22
firefax wrote: post #8

I've been using Ollama, Gemma3:12b is about all my little air can handle. If anyone has suggestions on other models, as an experiment I tried asking it to design me a new latex resumé and it struggled for two hours with the request to put my name prominently at the top in a grey box with my email and phone number beside it.

I was playing with the new IBM Granite models. They are quick/small and they do seem accurate. You can even try them online in the browser because they are small enough to be loaded via the filesystem: https://huggingface.co/spaces/ibm-granite/Granite-4.0-Nano-W... Not only are they a lot more recent than gemma, they seem really good at tool calling, so probably good for coding tools. I haven’t personally tried it my…

Interesting. Is there a way to load this into Ollama? Doing things in browser is a cool flex, but my interest is specifically in privacy respecting LLMs -- my goal is to run the most powerful one I can on my personal machine, with the end goal being those little queries I used to send to "the cloud" can be done offline, privately.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#86

Post by erikig »

Hardware: MacBook Pro M4 Max, 128GB

Platform: LMStudio (primarily) & Ollama

Models:

- qwen/qwen3-coder-30b A3B Instruct 8-bit MLX

- mlx-community/gpt-oss-120b-MXFP4-Q8

For code generation especially for larger projects, these models aren't as good as the cutting edge foundation models. For summarizing local git repos/libraries, generating documentation and simple offline command-line tool-use they do a good job.

I find these communities quite vibrant and helpful too:

- https://www.reddit.com/r/LocalLLM/

- https://www.reddit.com/r/LocalLLaMA/

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#87

Post by andai »

embedding-shape wrote: post #23

> Which model(s) are you running (e.g., Ollama, LM Studio, or others) I'm running mainly GPT-OSS-120b/20b depending on the task, Magistral for multimodal stuff and some smaller models I've fine-tuned myself for specific tasks.. All the software is implemented by myself, but I started out with basically calling out to llama.cpp, as it was the simplest and fastest option that let me integrate it into my own software wi…

gpt-oss-120b keeps stopping for me in Codex. (Also in Crush.)

I have to say "continue" constantly.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#88

Post by lacoolj »

fm2606 wrote: post #57
lreeves wrote: post #2

I sometimes still code with a local LLM but can't imagine doing it on a laptop. I have a server that has GPUs and runs llama.cpp behind llama-swap (letting me switch between models quickly). The best local coding setup I've been able to do so far is using Aider with gpt-oss-120b. I guess you could get a Ryzen AI Max+ with 128GB RAM to try and do that locally but non-nVidia hardware is incredibly slow for coding usage…

gpt-oss-120b is amazing. I created a RAG agent to hold most of GCP documentation (separate download, parsing, chunking, etc). ChatGPT finished a 50 question quiz in 6 min with a score of 46 / 50. gpt-oss-120b took over an hour but got 47 / 50. All the other local LLMs I tried were small and performed way worse, like less than 50% correct. I ran this on an i7 with 64gb of RAM and an old nvidia card with 8g of vram. ED…

you can run the 120b model on an 8GB GPU? or are you running this on CPU with the 64GB RAM?

I'm about to try this out lol

The 20b model is not great, so I'm hoping 120b is the golden ticket.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#89

Post by johnisgood »

scosman wrote: post #58
johnisgood wrote: post #55

Earlier quoted context omitted.

What setup would you (or other people) recommend for a local model, and which model, if I want something like Claude Sonnet 4.5 (or actually, earlier versions, which seemed to be better)? Anyone could chime in! I just want to have working local model that is at least as good as Sonnet 4.5, or 3.x.

Nothing open is quite as good as Sonnet 4.5 and Codex 5. GLM 4.6, MiniMax M2, Deepseek v3.2, Kimi K2 and Qwen Coder 3 are close. But those are hundreds of billions of parameters, so running locally is very very expensive.

That is unfortunate. I will never be able to afford such hardware that could run them. :(

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#90

Post by jetsnoc »

altcognito wrote: post #50
jetsnoc wrote: post #49

Models gpt-oss-120b, Meta Llama 3.2, or Gemma (just depends on what I’m doing) Hardware - Apple M4 Max (128 GB RAM) paired with a GPD Win 4 running Ubuntu 24.04 over USB-C networking Software - Claude Code - RA.Aid - llama.cpp For CUDA computing, I use an older NVIDIA RTX 2080 in an old System76 workstation. Process I create a good INSTRUCTIONS.md for Claude/Raid that specifies a task & production process with a task…

What sorts of token/s are you getting with each model?

Model performance summary:

  **openai/gpt-oss-120b** — MLX (MXFP4), ~66 tokens/sec @ Hugging Face: `lmstudio-community/gpt-oss-120b-MLX-8bit`

  **google/gemma-3-27b** — MLX (4-bit), ~27 tokens/sec @ Hugging Face: `mlx-community/gemma-3-27b-it-qat-4bit`

  **qwen/qwen3-coder-30b** — MLX (8-bit), ~78 tokens/sec @ Hugging Face: `Qwen/Qwen3-Coder-30B-A3B-Instruct`
Will reply back and add Meta Llama performance shortly.
Post reply on HN