Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jamesob's guide to running SOTA LLMs locally

github.com

Post reply on HN
Print view View original

161–170 of 193 posts

Re: Jamesob's guide to running SOTA LLMs locally

#161

Post by rldjbpin »

no matter your luck with hardware or your sysadmin skills, doing local inference for just yourself and/or to emulate typical usage (e.g. your coding workflow and deep research, etc.) is just very inefficient in current model architecture.

to me, this is a "truck" approach to city driving as a single person who does not do furniture hauling every weekend. the sense of privacy and freedom is nice but online inference is more "economical" as multi-user load is more effectively served than going solo.

maybe new architectures would make it effective to do text inference locally [1], till then great on you if you can spend car money on your setup. hope it is a great learning experience as well.

[1] https://deepmind.google/models/gemma/diffusiongemma/

Re: Jamesob's guide to running SOTA LLMs locally

#162

Post by aka-rider »

3eb7988a1663 wrote: post #38

Related - what is the best isolation system available? Do I have to go full, fat VMs or can I get by with a Firecracker-like thing? Seemingly every available option has some subtle-gotchas about how easy it is to blow off your foot and effectively have no security at all. I use VMs because I actually trust that security is a foundational principle of the technology, not a well-if-you-use-these-20-flags-and-squint kin…

On MacOS you have a seatbelt sandbox built-in. On Linux - docker with SELinux or similar utility over namespaces.

You need to model attack vector first.

`rm -rf` - restricted write

`curl malware.sh | sh` restrict execution from writeable dirs (seatbelt/SELinux)

Restricted write to sensitive directories would most likely neuter most malware.

Credentials leak - cleanup environment, deny reading .ssh, .aws, other, and don't allow LLMs anywhere near production systems.

I made a small utility for MacOS https://github.com/aka-rider/leash

But it may be as well a bash script

Re: Jamesob's guide to running SOTA LLMs locally

#163

Post by titanomachy »

mips_avatar wrote: post #118
titanomachy wrote: post #98

Earlier quoted context omitted.

The bandwidth argument is compelling, do we have benchmarks for these models? I’m curious what it translates to in tokens per second

I benchmarked mine for a deep research workload I was running. Concurrency 1 is the speed you'd get if you're chatting with one agent, 2x3090 (has an nvlink bridge though it didn't seem to matter hugely for inference) Qwen 3.6 27b int4: Concurrency 1: 68 tok/s output Concurrency 32: 363 tok/s output Prompt processing speed: 1520 tok/s Qwen 3.6 35ba3b int4: Concurrency 1: 150 tok/s output Concurrency 32: 1083 tok/s ou…

Yeah that’s a huge difference. I don’t think I want to interactively use any model with 18 tps.

Re: Jamesob's guide to running SOTA LLMs locally

#164

Post by aka-rider »

odo1242 wrote: post #44
Aurornis wrote: post #16

I play with local LLMs a lot. I've spent more on hardware than I should. I'm friends with a local group of people who have spent a lot more than I have. The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like…

This is similar to my experience with (8-bit quantized, non-MOE, 26b) Qwen locally on my computer. It’s really good for small tasks, but the first time I tried to do a major task with it it straight up forgot what agent harness it was in and started using the wrong format for tool calls lol (If you’re curious, it was running in Pi, but somehow convinced itself it was running in Claude instead and started trying to ca…

Model+harness combination means a lot. That's why all major labs are making their own. All models have quircks harnesses know about "you are reading the same file 3rd time you are in the loop, step back"

I tried all frontier Chinese models, and Qwen is the one running the best in ClaudeCode, my personal theory, it's because Qwen was distilled from Opus.

Re: Jamesob's guide to running SOTA LLMs locally

#165

Post by fweimer »

simonw wrote: post #48
jacobgold wrote: post #47

> "~$40k At this price level, you get the next step up in model intelligence. Something pretty close to Claude Opus." That is equivalent to 16.8 years of Claude Opus 4.8 or Codex GPT 5.5 at $200/mo. I'm a huge fan of running local models, but they're still wildly expensive, lower quality, and possibly dangerous (if backdoored). I sincerely wish this wasn't the case.

That $200/month is already more like $4,000/month if you have to pay full API pricing - "enterprise" companies for example. That drops the equivalent to 10 months. (I'd be surprised if that local rig really can drive the equivalent of $4,000/month of API spend though, given that a local rig can run prompts in parallel a lot less effectively than Anthropic's many data centers.)

I think the decode phase of inference typically uses local compute resources poorly due to the very small batch size. If you can run many inference tasks in parallel, this will make local inference more competitive to centralized inference, not less.

Re: Jamesob's guide to running SOTA LLMs locally

#166

Post by petu »

gizajob wrote: post #142
whalesalad wrote: post #46

why in gods name is a RTX PRO 6000 $13,000? supply and command?

About £4000 on eBay uk right now. Because if they were any lower we’d all be buying six each.

Nvidia reuses numbers for workstation cards, so there's multiple vastly different 'RTX 6000' cards:

  Quadro RTX 6000 (Turing / 2080 Ti / 24GB @ 672.0 GB/s)
  RTX A6000 (Ampere / 3090 / 48GB @ 768.0 GB/s)
  RTX 6000 Ada (4090 / 48GB @ 960.0 GB/s)
  RTX PRO 6000 Blackwell (5090 / 96GB @ 1.79 TB/s)
For £4000 you were likely looking at RTX 6000 Ada listing.

Re: Jamesob's guide to running SOTA LLMs locally

#167

Post by NBJack »

aka-rider wrote: post #160
Aurornis wrote: post #16

I play with local LLMs a lot. I've spent more on hardware than I should. I'm friends with a local group of people who have spent a lot more than I have. The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like…

I run Qwen3.6 on RTX4090, and it does amazing job for the most parts. For coding task, one needs to break the session among multiple calls I made https://github.com/aka-rider/orqestra but it's possible to do the same in almost any modern harness directly. The main idea is: - separate session that burns context on reading code and calling tools (context7, etc) -> markdown report "here are relevant patrts of code, docs…

I'm afraid prompts and clever arrangements of data don't really negate the parent post warnings. It's great if it works for you and your projects. Unfortunately, I can almost guarantee your approach will break down once you get a project large enough or switch to a less popular language.

My favorite example is Godot; most local models just can't get it through their thick AI skull that code alone won't be enough to generate working solutions. They must accept a more complex harness, or you must provide much more info that eats the precious available context on every run.

Re: Jamesob's guide to running SOTA LLMs locally

#168

Post by nullbio »

Aurornis wrote: post #16

I play with local LLMs a lot. I've spent more on hardware than I should. I'm friends with a local group of people who have spent a lot more than I have. The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like…

The models will improve and the hardware will remain useful. It's likely a good investment regardless, if you have the money to spend. Plus your business won't be stolen by Anthropic.

Re: Jamesob's guide to running SOTA LLMs locally

#169

Post by nullbio »

jacobgold wrote: post #47

> "~$40k At this price level, you get the next step up in model intelligence. Something pretty close to Claude Opus." That is equivalent to 16.8 years of Claude Opus 4.8 or Codex GPT 5.5 at $200/mo. I'm a huge fan of running local models, but they're still wildly expensive, lower quality, and possibly dangerous (if backdoored). I sincerely wish this wasn't the case.

None of the leading models are backdoored, that's nonsense. I've still never heard of a single backdoored model, and if one was found, it would be quickly eradicated from HF. This is a non issue.

Re: Jamesob's guide to running SOTA LLMs locally

#170

Post by djx22 »

in my experience running models that have been heavily quantized(q4) or altered to some extent has never made me say “wow, this is so amazing”. On the contrary, the model ended up in the thrash bin after a few prompts. I have an RTX 6000 PRO with 96GB, and what I can run comfortably is Qwen 3.6 27B or MoE, Gemma 4 31B. This is as far as it goes when you run the model at full precision and maximum context length. They perform well and you can use them for coding, doing research on the internet and what have you. So if you do the math and you see yourself spending more than the $2400/year to Anthropic, then it might make sense to get one of these cards but accept the quality drop. Otherwise, will humans even be coding in 5 years from now?
Post reply on HN