Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

news.ycombinator.com

Post reply on HN
Print view View original

171–180 of 196 posts

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#171

Post by Balinares »

dennemark wrote: post #135

I have AMD Strix Halo (395) on my work laptop (HP Ultrabook G1A) as well as at home with Framework Desktop. On both i have setup lemonade-server on system start. At work i use Qwen3 Coder 30B-3A with continue.dev. It serves me well in 90% of cases. At home i have 128GB RAM. I try a bit GPT120B. I host Open WebUI on it and connect via https and wireguard to it, so i can use it as PWA on my phone. I love not needing to…

Does Qwen3 Coder do a good job invoking its tools as appropriate for you? Under continue.dev at least, I've found I need to remind it constantly.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#172

Post by nickthegreek »

mwambua wrote: post #84

Tangential question. What do people use for search? What search engines provide the best quality to cost ratios? Also are there good solutions for searching through a local collection of documents?

just setup searxng yesterday and mcp for it in lm studio to be able to search the net for answers to simple queries. small ibm granite worked surpassingly well, while oss20b seemed to be looping searches.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#173

Post by windexh8er »

embedding-shape wrote: post #168
windexh8er wrote: post #155

Earlier quoted context omitted.

I've been really impressed by OpenCode [0]. The limitations of all the frontier TUI is removed and it is feature complete and performant compared to Codex or Claude Code. [0] https://opencode.ai/

> OpenCode will be available on desktop soon Anyone happen to know what that means exactly? The install instructions at the top seems to indicate it already is available on desktop?

It's a terminal only (TUI) tool today. They're releasing a graphical (GUI) version in the future.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#174

Post by embedding-shape »

windexh8er wrote: post #173
embedding-shape wrote: post #168

Earlier quoted context omitted.

> OpenCode will be available on desktop soon Anyone happen to know what that means exactly? The install instructions at the top seems to indicate it already is available on desktop?

It's a terminal only (TUI) tool today. They're releasing a graphical (GUI) version in the future.

> It's a terminal only (TUI) tool today.

But to use that TUI you need a desktop, or at least a laptop I guess, but that distinction doesn't make sense. Are they referring to the GUI being the "Desktop Version"? Never heard it put that way before if so.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#175

Post by gkfasdfasdf »

fm2606 wrote: post #57
lreeves wrote: post #2

I sometimes still code with a local LLM but can't imagine doing it on a laptop. I have a server that has GPUs and runs llama.cpp behind llama-swap (letting me switch between models quickly). The best local coding setup I've been able to do so far is using Aider with gpt-oss-120b. I guess you could get a Ryzen AI Max+ with 128GB RAM to try and do that locally but non-nVidia hardware is incredibly slow for coding usage…

gpt-oss-120b is amazing. I created a RAG agent to hold most of GCP documentation (separate download, parsing, chunking, etc). ChatGPT finished a 50 question quiz in 6 min with a score of 46 / 50. gpt-oss-120b took over an hour but got 47 / 50. All the other local LLMs I tried were small and performed way worse, like less than 50% correct. I ran this on an i7 with 64gb of RAM and an old nvidia card with 8g of vram. ED…

What were you using for RAG? Did you build your own or some off the shelf solution (e.g. openwebui)

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#176

Post by toogle »

I built a custom code-completion server: https://github.com/toogle/mlx-dev-server>.

The key advantage is that it cancels generation when you continue typing, so invalidated completions don’t waste time. This makes completion latency predictable (about 1.5 seconds for me).

My setup: - MacBook Pro (M3 Max) - Neovim - https://github.com/huggingface/llm.nvim

Models I typically use: - mlx-community/DeepSeek-Coder-V2-Lite-Instruct-4bit-mlx - mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#177

Post by Greenpants »

mark_l_watson wrote: post #164
Greenpants wrote: post #159

I got a personal Mac Studio M4 Max with 128GB RAM for a silent, relatively power-efficient yet powerful home server. It runs Ollama + Open WebUI with GPT-OSS 120b as well as GLM4.5-Air (default quantisations). I rarely ever use ChatGPT anymore. Love that all data stays at home. I connect remotely only via VPN (my phone enables this automatically via Tasker). I'm 50% brainstorming ideas with it, asking critical questi…

I like your method of adding TODOs in your code, then using a model - I am going to try that. I only have a 32G M2 Mac so I have to use Ollama Cloud to run some of the larger models but that said I am surprised by what I can do ‘all local’ and it really is magical running all on my own hardware, when I can.

The TODOs really help me get my logic sorted out first in pseudocode. Glad to inspire someone else with it!

I've read that GPT-OSS:20b is still a very powerful model, I believe it fits in your Mac's RAM as well and could still be quite fast to output. For me personally, only the more complex questions require a better model than local ones. And then I'm often wondering if LLMs are the right tool to solve the complexity.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#179

Post by suprjami »

kabes wrote: post #158
suprjami wrote: post #131

Earlier quoted context omitted.

If you have ~$25k to buy a H200 then don't buy one. Rent them out much cheaper and keep renting newer models when your H200 becomes an outdated paperweight. Assuming you ran inference for the full working day, you'd need to run your H200 for almost 2 years to break even. Realistically you don't run inference full time so you'll never realise the value of the card before it's obsolete.

The company I work for is in the defense industry and by contract can't send any code outside their own datacenter. So cloud-rented H200's are a no-go and obviously commercial LLM's as well. so breaking even is not the goal here.

In that case I suggest you buy cheaper desktop cards instead of a H200. Two or three 5090s will let you run decent models at very good speed.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#180

Post by fm2606 »

rovr138 wrote: post #142
fm2606 wrote: post #57

Earlier quoted context omitted.

gpt-oss-120b is amazing. I created a RAG agent to hold most of GCP documentation (separate download, parsing, chunking, etc). ChatGPT finished a 50 question quiz in 6 min with a score of 46 / 50. gpt-oss-120b took over an hour but got 47 / 50. All the other local LLMs I tried were small and performed way worse, like less than 50% correct. I ran this on an i7 with 64gb of RAM and an old nvidia card with 8g of vram. ED…

> I created a RAG agent to hold most of GCP documentation (separate download, parsing, chunking, etc) If you share the scripts to gather the GCP documentation this, that'd be great. Because I have had an idea to do something like this, and the part I don't want to deal with is getting the data

I tried scripts but got blocked. I used wget to download tthem
Post reply on HN