Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jamesob's guide to running SOTA LLMs locally

github.com

Post reply on HN
Print view View original

101–110 of 193 posts

Re: Jamesob's guide to running SOTA LLMs locally

#101

Post by nullc »

Aurornis wrote: post #16

I play with local LLMs a lot. I've spent more on hardware than I should. I'm friends with a local group of people who have spent a lot more than I have. The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like…

> The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like 50-55K.

Just two months ago you could get RTX PRO 6000's for about $8500 on ebay, which is the MSRP.

Re: Jamesob's guide to running SOTA LLMs locally

#104

Post by neverm0r3 »

jacobgold wrote: post #47

> "~$40k At this price level, you get the next step up in model intelligence. Something pretty close to Claude Opus." That is equivalent to 16.8 years of Claude Opus 4.8 or Codex GPT 5.5 at $200/mo. I'm a huge fan of running local models, but they're still wildly expensive, lower quality, and possibly dangerous (if backdoored). I sincerely wish this wasn't the case.

I agree with your point, but it should be noted that this assumes consistent prices for LLMs. The OpenAIs and Anthropics of this world are still selling the plans at a subsidised prices with the power of VCs, who are going to want that return some time.

Re: Jamesob's guide to running SOTA LLMs locally

#106

Post by anon373839 »

Der_Einzige wrote: post #93
nabakin wrote: post #72

Earlier quoted context omitted.

Are you running qwen3.6-27b on one 3090 with your KV cache at q4? Ime there is significant long-context recall accuracy degradation at that precision. I prefer putting the KV cache at q8 and working with the 120k context

Use modern samplers and you don’t need to limit yourself to 8bit at half the context window. I could push it down to 1.58 bits and get decently good output easily by simply not using the garbage default top_p and top_k that vendors continue to wrongly recommend.

Where do you find optimal samplers and sampler settings for these models? Very interested in this as I, too, use Q8.

Re: Jamesob's guide to running SOTA LLMs locally

#107

Post by thinkmassive »

CamperBob2 wrote: post #20
Aurornis wrote: post #16

I play with local LLMs a lot. I've spent more on hardware than I should. I'm friends with a local group of people who have spent a lot more than I have. The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like…

All very true. Right now, running GLM 5.2 at its full BF16 quantization level needs 1.5 TB of VRAM. You can't run this locally at a usable speed for less than $250K or so, and frankly I'd be surprised if it could be done for less than $500K. The best NV4FP quant for 5.2 appears to be lukealonso's at https://huggingface.co/lukealonso/GLM-5.2-NVFP4 , and it is capable of good throughput (75-100 tps) without losing much…

Another option is renting cloud GPUs only when you need them. A server with 8x B200 is around $32/hr.

Obviously depends on the use case and threat model, but that hardware is publicly available at far less than $500k upfront.

Re: Jamesob's guide to running SOTA LLMs locally

#108

Post by Aurornis »

nullc wrote: post #101
Aurornis wrote: post #16

I play with local LLMs a lot. I've spent more on hardware than I should. I'm friends with a local group of people who have spent a lot more than I have. The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like…

> The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like 50-55K. Just two months ago you could get RTX PRO 6000's for about $8500 on ebay, which is the MSRP.

> Just two months ago you could get RTX PRO 6000's for about $8500 on ebay, which is the MSRP.

The MSRP was raised to $13,250.

Warranty is very important for expensive cards like this. I don't recommend buying on eBay unless they come with a very big discount.

Re: Jamesob's guide to running SOTA LLMs locally

#110

Post by pulse7 »

Der_Einzige wrote: post #88
Aurornis wrote: post #16

I play with local LLMs a lot. I've spent more on hardware than I should. I'm friends with a local group of people who have spent a lot more than I have. The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like…

Everything in this post is spot on and it is a rare example of a HN person not saying BS about LLMs! That said, modern LLM sampling algorithms like min_p, top_n sigma , etc heavily mitigate the performance penalty you get from doing long context tasks. Problems with long context come from accumulation of small sampling errors over time. My qwen 3.6 27b (the dense one) runs perfectly well on coding tasks at the edge o…

Can you please share you llama.cpp server parameters to turn on modern LLM sampling stack?

Docs [1] say that the top_n_sigma is already in the default sampler list: "(default: penalties;dry;top_n_sigma;top_k;typ_p;top_p;min_p;xtc;temperature)"

[1] https://github.com/ggml-org/llama.cpp/blob/master/tools/serv...

Post reply on HN