Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jamesob's guide to running SOTA LLMs locally

github.com

Post reply on HN
Print view View original

61–70 of 193 posts

Re: Jamesob's guide to running SOTA LLMs locally

#61

Post by dofm »

elsombrero wrote: post #41
ZiiS wrote: post #39

Earlier quoted context omitted.

Full fat VMs with GPU passthough I trust a lot less then CPU ones.

from my understanding, you can run the inference server (llama.cpp/vllm/whatever) and the agent/harness in different contexts, event different machines. The risky part is in the agent/harness and what tools it has access to. You don't need to give GPU passthrough to the VM running the agent/harness. There is still a risk of a prompt messing with the inference server, but I think that's a much lower risk compared to a…

Right. All my experiments are naïve, I am sure, but I run the LLM on the host and expose it via OpenAI API to the VMs.

This approach requires that you trust the llama.cpp codebase, essentially. It might be reasonable not to.

I suppose in principle there is the risk of a prompt exploit corrupting the inference server.

Re: Jamesob's guide to running SOTA LLMs locally

#62

Post by vient »

Aurornis wrote: post #16

I play with local LLMs a lot. I've spent more on hardware than I should. I'm friends with a local group of people who have spent a lot more than I have. The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like…

Wonder if AMD MI350P release will affect setups like this. From what I've heard, the price will be pretty similar to RTX PRO 6000 while having 50% more VRAM which is additionally an HBM3E instead of GDDR7.

Re: Jamesob's guide to running SOTA LLMs locally

#64

Post by maxxxml »

api wrote: post #13

Apple M series chips deserve a mention as another option, especially since you get a whole Mac laptop or desktop workstation too. They have unified memory and respectable inference performance, and for some variations can be cheaper than video cards, especially if you get an older-gen high-end M series with a lot of RAM used or refurbished. I've read that Apple has plans once the RAM bottleneck passes to offer more R…

MLX is super underrated right now, tons of performance unlocked as of recent. Love to see it!

Re: Jamesob's guide to running SOTA LLMs locally

#65

Post by mips_avatar »

datadrivenangel wrote: post #5

"A great way to go is 2x RTX 3090s for a total of 48GB VRAM total. You can then run Qwen3.6-27B, which is an awesome model." Just want to note that for $3k you can get an M5 macbook pro with 48gb of shared memory, and it will not be a giant box. Also, consider committing to spending that money on a cloud hosting provider, which will be at least somewhat cheaper if not significantly cheaper. It is awesome being able t…

The cool thing about the 3090s is the RAM bandwidth. Token generation is mostly bottlenecked on memory bandwidth. Dual 3090s have 1.87 TB/s memory bandwidth (0.936 TB/s each), vs the M5 Macbook pro with only 0.3 TB/s (max chip has up to 0.63 TB/s but it's a $10k machine at that config).

This translates to qwen 27b actually working fast enough for useful work on dual 3090s and being painfully slow on Macbook Pros. Also if you're running a big model on a macbook pro the UI gets laggy and the keyboard gets hot. Much better to run dual 3090s in your basement and connect to them from your Macbook.

Re: Jamesob's guide to running SOTA LLMs locally

#66

Post by npodbielski »

maxxxml wrote: post #63

What harness is the best for local LLMs? I've been researching optimizing local LLM agent harness performance with context/ tools. Quite the endeavor and would love to learn what users prefer for this type of workflow.

I like vibe and pi. Vibe just looks nice and is good enough. But pi extensibility is just another level. There is also Dirac that is quite OK but seems like full of bugs. Zerostack is the simplest harness I saw. OpenCode is OK too. Rest I did not try.

Re: Jamesob's guide to running SOTA LLMs locally

#67

Post by rsalus »

CuriouslyC wrote: post #53
Aurornis wrote: post #45

Earlier quoted context omitted.

SSD streaming throughput is too slow to be usable. GLM-5.2 has 40B active parameters at a time. At Q4 that's 20GB. The best PCIe 5 SSDs can get 15GB/sec when everything goes well. Every expert load would take more than a second. If you had enough RAM and enough SSDs in parallel you might get a couple tokens per second on a good day. If you left this machine running 24 hours straight, you might be able to get 200,000…

You can improve that with speculative preload. I'm sure models could be designed and tuned around efficient SSD offloading to keep throughput pretty high.

surely the supply of unified memory will rise to meet demand before this is needed

Re: Jamesob's guide to running SOTA LLMs locally

#68

Post by npodbielski »

WithinReason wrote: post #34
datadrivenangel wrote: post #5

"A great way to go is 2x RTX 3090s for a total of 48GB VRAM total. You can then run Qwen3.6-27B, which is an awesome model." Just want to note that for $3k you can get an M5 macbook pro with 48gb of shared memory, and it will not be a giant box. Also, consider committing to spending that money on a cloud hosting provider, which will be at least somewhat cheaper if not significantly cheaper. It is awesome being able t…

I'm running Qwen3.6-27B on a single 24GB GPU at 80 tok/s, you don't even need 2 of them

Yeah but 4 bits very often loops needlessly. Which is not that bad because you do not pay for tokens. But you paid for hardware and you want use it for something useful. Q6 is better but then you have like 40t/s prefill. Which is really tiring. But at least it says sorry when you ask it what is wrong! I heard there is some extension for PI preventing that. I need to look into it. Otherwise I am quite happy.

Re: Jamesob's guide to running SOTA LLMs locally

#69

Post by GTP »

There also exists an in-between possibility, that is, if you get 128GB of vram (there are now multiple options in the market to get that amount with a unified memory architecture) you can run DeepSeek V4 flash at good speed via DwarfStar. I'm not going to spend money on this, but my gut feeling is that this would be the right compromise for a lot of people.
Post reply on HN