Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

news.ycombinator.com

Post reply on HN
Print view View original

131–140 of 196 posts

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#131

Post by suprjami »

kabes wrote: post #121

Let's say I have a server with an h200 gpu at home. What's the best open model for coding I can run on it today? And is it somewhat competitive with commercial models like sonnet 4.5?

If you have ~$25k to buy a H200 then don't buy one. Rent them out much cheaper and keep renting newer models when your H200 becomes an outdated paperweight.

Assuming you ran inference for the full working day, you'd need to run your H200 for almost 2 years to break even. Realistically you don't run inference full time so you'll never realise the value of the card before it's obsolete.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#132

Post by Xenograph »

dust42 wrote: post #29

On a Macbook pro 64GB I use Qwen3-Coder-30B-A3B Q4 quant with llama.cpp. For VSCode I use continue.dev as it allows to set my own (short) system prompt. I get around 50token/sec generation and prompt processing 550t/s. When giving well defined small tasks, it is as good as any frontier model. I like the speed and low latency and the availability while on the plane/train or off-grid. Also decent FIM with the llama.cpp…

Have you tried continue.dev's new open completion model [1]? How does it compare to llama.vscode FIM with qwen?

[1] https://blog.continue.dev/instinct/

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#133

Post by gunalx »

lacoolj wrote: post #88
fm2606 wrote: post #57

Earlier quoted context omitted.

gpt-oss-120b is amazing. I created a RAG agent to hold most of GCP documentation (separate download, parsing, chunking, etc). ChatGPT finished a 50 question quiz in 6 min with a score of 46 / 50. gpt-oss-120b took over an hour but got 47 / 50. All the other local LLMs I tried were small and performed way worse, like less than 50% correct. I ran this on an i7 with 64gb of RAM and an old nvidia card with 8g of vram. ED…

you can run the 120b model on an 8GB GPU? or are you running this on CPU with the 64GB RAM? I'm about to try this out lol The 20b model is not great, so I'm hoping 120b is the golden ticket.

I have in many cases had better results with the 20b model, over the 120b model. Mostly because it is faster and I can iterate prompts quicker to choerce it to follow instructions.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#134

Post by Xenograph »

embedding-shape wrote: post #97
Xenograph wrote: post #82

Earlier quoted context omitted.

Would love for you to share the Codex patches you needed to make and the AGENTS.md prompting, if you're open to it.

Basically just find the place where the inference call happens, add top_k, top_p and temperature to hard-coded numbers (0, 1.0 and 1.0 for GPT-OSS) and you should be good to go. If you really need it, I could dig out patch from it, but it should be really straightforward today, and my patch might be conflicting with the current master of codex, I've diverged for other reasons since I did this.

That makes sense, wasn't sure if it was as simple as tweaking those two numbers or not, thanks for sharing!

If there's any insight you can share about your AGENTS.md prompting, it may also be helpful for others!

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#135

Post by dennemark »

I have AMD Strix Halo (395) on my work laptop (HP Ultrabook G1A) as well as at home with Framework Desktop.

On both i have setup lemonade-server on system start. At work i use Qwen3 Coder 30B-3A with continue.dev. It serves me well in 90% of cases.

At home i have 128GB RAM. I try a bit GPT120B. I host Open WebUI on it and connect via https and wireguard to it, so i can use it as PWA on my phone. I love not needing to think about where my data goes. But i would like to allow parallel requests, so i need to tinker a bit more. Maybe llama-swap is enough.

I just need to see how to deal with context length. My models stop or go into infinite loop after some messages. But then i often start a new chat.

Lemonade-server runs with llama.cpp, vllm seems to be scaling better thoug, but is not so easy to set up.

Unsloth GGUFs are great resource for models.

Also for Strix Halo check out kyuz0 repositorIES! Also has image gen. I didnt try those yet. But the benchmarks are awesome! Lots to learn from. Framework forum can be useful, too.

https://github.com/kyuz0/amd-strix-halo-toolboxes Also nice: https://llm-tracker.info/ It links to some benchmark site with models by size. I prefer such resources, since it is quite easy to see which one fit in my RAM (even though i have this silly thumbrule Billion Token ≈ GB RAM).

Btw. even a AMD HX 370 with non soldered RAM can get some nice t/s for smaller models. Can be helpful enough when disconnected from internet and you dont know how to style a svg :)

Thanks for opening up this topic! Lots of food :)

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#136

Post by ThatPlayer »

lacoolj wrote: post #88
fm2606 wrote: post #57

Earlier quoted context omitted.

gpt-oss-120b is amazing. I created a RAG agent to hold most of GCP documentation (separate download, parsing, chunking, etc). ChatGPT finished a 50 question quiz in 6 min with a score of 46 / 50. gpt-oss-120b took over an hour but got 47 / 50. All the other local LLMs I tried were small and performed way worse, like less than 50% correct. I ran this on an i7 with 64gb of RAM and an old nvidia card with 8g of vram. ED…

you can run the 120b model on an 8GB GPU? or are you running this on CPU with the 64GB RAM? I'm about to try this out lol The 20b model is not great, so I'm hoping 120b is the golden ticket.

With MoE models like gpt-oss, you can run some layers on the CPU (and some on GPU): https://github.com/ggml-org/llama.cpp/discussions/15396

Mentions 120b is runnable on 8GB VRAM too: "Note that even with just 8GB of VRAM, we can adjust the CPU layers so that we can run the large 120B model too"

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#137

Post by banku_brougham »

egberts1 wrote: post #112

Ollama, 16-CPU Xenon E6320 (old), 1.9Ghz, 120GB DDRAM4, 240TB RAID5 SSDs, on Dell Precision T710 ("The Beast"). NO GPU. 20b (n oooooot f aah st at all). Pure CPU bound. Tweaked for 256KB chunking into RAG. Ingested election laws of 50 states, territories and Federal. Goal. Mapping out each feature of the election and deal with (in)consistent terminologies sprouted by different university-trained public administration…

bravo, this is a great use of talent for society

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#138

Post by vinhnx »

jdthedisciple wrote: post #39
vinhnx wrote: post #36

> Which model(s) are you running (e.g., Ollama, LM Studio, or others) and which open-source coding assistant/integration (for example, a VS Code plugin) you’re using? Open-source coding assistant: VT Code (my own coding agent -- github.com/vinhnx/vtcode) Model: gpt-oss-120b remote hosted via Ollama cloud experimental > What laptop hardware do you have (CPU, GPU/NPU, memory, whether discrete GPU or integrated, OS) and…

Wait ollama cloud has a free tier? Sounds too good. Where's the catch? And is it private?

Yeah, Ollama recently announces Cloud, as I think it still in beta and free, usage is generously too, enough to build and hack on. But I'm not sure about data training, I don't see the settings to turn this off..

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#139

Post by vinhnx »

bradfa wrote: post #43
jdthedisciple wrote: post #39

Earlier quoted context omitted.

Wait ollama cloud has a free tier? Sounds too good. Where's the catch? And is it private?

The catch is ollama cloud is likely to increase prices and/or decrease usage limit levels soon. Free tier has more restrictions than their $20/mo tier. They claim to not store anything ( https://ollama.com/cloud ) but you'll have to clarify what you mean by "private" (your model likely runs on shared hardware with other users).

I agree. "Free" usage could mean tradeoff. But for side-project and experiments, to accesss open source model like gpt-oss, as my machine can not run, I think I will accept it.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#140

Post by trailbits »

giancarlostoro wrote: post #70

If you're going to get a MacBook, get the Pro, it has a built-in fan, you don't want the heat just sitting there on the MacBook Air. Same with the Mac mini, get the studio instead, it has a fan, the Mini does not. I don't know about you but I wouldn't want my brand new laptop / desktop to be heating up the entire time I'm coding with 0 cool off. If you go the Mac route, I recommend getting TG Pro, the default fan set…

The default context window using ollama or lmstudio is small, but you can easily quadruple the default size while running gpt-oss-20b on a 24GB Mac.
Post reply on HN