Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jamesob's guide to running SOTA LLMs locally

github.com

Post reply on HN
Print view View original

71–80 of 193 posts

Re: Jamesob's guide to running SOTA LLMs locally

#71

Post by CobaltFire »

mips_avatar wrote: post #65
datadrivenangel wrote: post #5

"A great way to go is 2x RTX 3090s for a total of 48GB VRAM total. You can then run Qwen3.6-27B, which is an awesome model." Just want to note that for $3k you can get an M5 macbook pro with 48gb of shared memory, and it will not be a giant box. Also, consider committing to spending that money on a cloud hosting provider, which will be at least somewhat cheaper if not significantly cheaper. It is awesome being able t…

The cool thing about the 3090s is the RAM bandwidth. Token generation is mostly bottlenecked on memory bandwidth. Dual 3090s have 1.87 TB/s memory bandwidth (0.936 TB/s each), vs the M5 Macbook pro with only 0.3 TB/s (max chip has up to 0.63 TB/s but it's a $10k machine at that config). This translates to qwen 27b actually working fast enough for useful work on dual 3090s and being painfully slow on Macbook Pros. Als…

$4.8k for 48GB Max (what the parent said). Half of your quote.

Even a 128GB is $6.8k today. Still only 2/3 your quote.

Bandwidth is relevant (I have both a 5090 and an M4 Max 128GB Studio, so have direct comparison right here), but quote the cost appropriately!

Re: Jamesob's guide to running SOTA LLMs locally

#72

Post by nabakin »

turova wrote: post #21

For qwen3.6-27b you can also run the q4 variant with full ~250K context on one 3090. It's fast enough to not be frustrating so the speed gains with 2x 3090s wouldn't be worth it to me. Running a q6 on 2x 3090s at half the speed with a smaller context is an option, but you're really not going to compete with SOTA models there anyway so unless you already have 2x 3090s, I would say 1 is the best investment given curren…

Are you running qwen3.6-27b on one 3090 with your KV cache at q4? Ime there is significant long-context recall accuracy degradation at that precision. I prefer putting the KV cache at q8 and working with the 120k context

Re: Jamesob's guide to running SOTA LLMs locally

#73

Post by searealist »

CuriouslyC wrote: post #53
Aurornis wrote: post #45

Earlier quoted context omitted.

SSD streaming throughput is too slow to be usable. GLM-5.2 has 40B active parameters at a time. At Q4 that's 20GB. The best PCIe 5 SSDs can get 15GB/sec when everything goes well. Every expert load would take more than a second. If you had enough RAM and enough SSDs in parallel you might get a couple tokens per second on a good day. If you left this machine running 24 hours straight, you might be able to get 200,000…

You can improve that with speculative preload. I'm sure models could be designed and tuned around efficient SSD offloading to keep throughput pretty high.

It would apply equally to GPU or RAM inference as those are also bandwidth constrained on decode, so people already try to optimize for it.

Re: Jamesob's guide to running SOTA LLMs locally

#74

Post by satvikpendem »

datadrivenangel wrote: post #5

"A great way to go is 2x RTX 3090s for a total of 48GB VRAM total. You can then run Qwen3.6-27B, which is an awesome model." Just want to note that for $3k you can get an M5 macbook pro with 48gb of shared memory, and it will not be a giant box. Also, consider committing to spending that money on a cloud hosting provider, which will be at least somewhat cheaper if not significantly cheaper. It is awesome being able t…

To summarize a video I saw recently [0] rebutting your arguments, MacBooks can get crazy slow when running local models or even just Claude Code and Codex due to their poor implementation, to the point that the laptop itself becomes unusable.

There are other arguments for running an ssh-able box in a closet somewhere too as with KVMs you can give an agent remote control over the machine itself such that it has vastly more capabilities than if it were controlling its own machine it's running on, as well as not needing to keep the MacBook open all the time just to have the agent finish running.

[0] https://youtu.be/9tGrhrVKCrE

Re: Jamesob's guide to running SOTA LLMs locally

#76

Post by satvikpendem »

echelon wrote: post #49
jacobgold wrote: post #47

> "~$40k At this price level, you get the next step up in model intelligence. Something pretty close to Claude Opus." That is equivalent to 16.8 years of Claude Opus 4.8 or Codex GPT 5.5 at $200/mo. I'm a huge fan of running local models, but they're still wildly expensive, lower quality, and possibly dangerous (if backdoored). I sincerely wish this wasn't the case.

Stop trying to run them locally, folks. You don't own your fiber connection. So why try to own another rapidly depreciating, expensive, and annoying asset? Rent cloud GPUs! You get to participate in the ownership, data control, price control, and hacking culture without having to Frankenstein some hobbyist box that costs a ton, is distilled down to functional uselessness, and is a PITA to maintain.

If I'm gonna rent cloud GPUs I might as well just use a subsidized cloud agent like Claude or Codex. As for depreciation, that is true, but the bet is that models get better for a certain parameter count faster than your hardware becomes obsolete, such as Gemma models for example at the same 30 billion parameter count being much better than some years ago.

Re: Jamesob's guide to running SOTA LLMs locally

#78

Post by mips_avatar »

CobaltFire wrote: post #71
mips_avatar wrote: post #65

Earlier quoted context omitted.

The cool thing about the 3090s is the RAM bandwidth. Token generation is mostly bottlenecked on memory bandwidth. Dual 3090s have 1.87 TB/s memory bandwidth (0.936 TB/s each), vs the M5 Macbook pro with only 0.3 TB/s (max chip has up to 0.63 TB/s but it's a $10k machine at that config). This translates to qwen 27b actually working fast enough for useful work on dual 3090s and being painfully slow on Macbook Pros. Als…

$4.8k for 48GB Max (what the parent said). Half of your quote. Even a 128GB is $6.8k today. Still only 2/3 your quote. Bandwidth is relevant (I have both a 5090 and an M4 Max 128GB Studio, so have direct comparison right here), but quote the cost appropriately!

You need the 128gb ram config to get the 614 GB/s bandwidth (which is $6999), you could skip out on upgrading the storage to save money but at that point I think most people upgrade the storage too at which point it's $8-10k + tax.

Re: Jamesob's guide to running SOTA LLMs locally

#79

Post by igorbark »

3eb7988a1663 wrote: post #38

Related - what is the best isolation system available? Do I have to go full, fat VMs or can I get by with a Firecracker-like thing? Seemingly every available option has some subtle-gotchas about how easy it is to blow off your foot and effectively have no security at all. I use VMs because I actually trust that security is a foundational principle of the technology, not a well-if-you-use-these-20-flags-and-squint kin…

im not sure that there is a plug n play set up that will work for everyone, because as with any security boundary, each layer of hardening has a usability trade-off. i definitely feel you about the uncertainty of it all, how do you actually know everything is tight?

personally, i think either a VM or microVM is the way to go. these things are actually designed as security boundaries, as opposed to containers. and as compared to bubblewrap, you can just give the agent a whole FS to work with and run it in yolo mode, whereas with bubblewrap you have to manually bootstrap the availability of each individual dev tool and make sure its config dirs and package caches and etc are mounted in a secure way and still will probably hit perm errors all the time. and there's just way less isolation.

also, something that has limited support in harnesses but IMO would make a lot of sense is running the harness process in the host, but having all the tool calls and file system interactions delegated to the VM. that way you keep all your session data and auth keys on the main machine where it can never get into context. otoh it makes your harness part of the security boundary, so that's the trade-off.

there's also all the usability questions around how to actually get data in/out of the VM. i have a script which can push local git repos into the VM and then pull from them as a remote, so the VM can't initiate any connection with the host doesn't need to hold git credentials. but ig for someone who wants their agent to push straight to GitHub that's a waste of effort.

options i've tried or seen for the VM itself: - qemu + libvirt: takes some doing to wrangle it together, but very battle tested and configurable - crun-vm is a PoC of higher level integration layer between podman and qemu, which is a really cool way to go about it. seems maybe abandoned but i just think it's neat and very existing tools/standards oriented rather than starting a new project and brand so i mention - libkrun is a newer entrant, and several ppl have built wrappers around it: - microsandbox - smolvm (posted/discussed on here recently) - krunvm

this is all Linux oriented, it's all i know.

Re: Jamesob's guide to running SOTA LLMs locally

#80

Post by SwellJoe »

I recently wrote up how I run local LLMs, because several folks had asked (https://swelljoe.com/post/how-i-run-local-llms/) and I think even my setup, which I spent maybe $4200 on, half on a Strix Halo and half on upgrades for my desktop, would be too expensive to justify today. I bought before prices went through the roof, and only did so because I like to tinker with hardware...not because I expected it to ever pay for itself vs. buying subsidized tokens from the big guys or the cheap tokens from efficient providers like DeepSeek.

Buying four $13000 GPUs and several thousand dollars worth of supporting hardware seems crazy. This supply shortage has to end eventually, and I can buy billions of DeepSeek, MiMo, and GLM tokens, and use $100 or $200 a month subscriptions for the big guys in the meantime for the difference in price once that happens. And, you can't even run the full-sized GLM on that hardware, it is quantized and so is your KV cache; the degradation is small, but not non-existent. You're not running a model that's equal to what you get when you buy GLM tokens from Z.ai.

My recommendation for self-hosting is this: If you already have a 24GB or 32GB GPU, or two, or a recent Mac with 32GB or more, run the appropriate quantization of Qwen 3.6 27B or Gemma 4 31B. If your hardware is older and too slow for that, use the MoE, but know it'll be dumber. Use the tiny model for the stuff that doesn't need deep smarts: Research (give it a Brave or Exa MCP for web search), summarization, simple Python scripts for basic tasks, simple websites or web apps, categorization of stuff (I used Gemma 4 to review my past writing for friendliness and helpfulness), etc. It can also be a sub-agent for bigger agents (for those same kinds of tasks). Gemma 4 12B is an incredibly good model for its size, particularly for vision tasks, and in the 4-bit quantization (7GB on disk) it runs on anything, even a modern tablet or phone.

And, if you don't already have a big GPU or unified memory Mac, just wait. Use the cheap tokens every AI company wants to sell you, for now. A Claude or Codex or Gemini subscription is a good deal. Tokens from DeepSeek are a good deal, especially with Reasonix agent (which maximizes caching, which DeepSeek is uniquely good at, and cached tokens are uniquely cheap at DeepSeek). GLM is Good Enough and has a cheap coding plan. MiMo has the cheapest tokens for a 1T+ model in the game, though DeepSeek and GLM are better models, MiMo is fine.

When prices come down, I'll be speccing out a beast to run the big models, too. But, I'm not paying 4x for RAM and GPU and storage, and y'all shouldn't either. That's crazy. Computer prices go down over time. It is the law.

Post reply on HN