Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jamesob's guide to running SOTA LLMs locally

github.com

Post reply on HN
Print view View original

81–90 of 193 posts

Re: Jamesob's guide to running SOTA LLMs locally

#81

Post by rsync »

kgeist wrote: post #9

>$40k gets you almost-Opus GLM 5.2 is "almost Opus," and it needs at least 8xH200s for comfortable inference (so it's closer to $400k than $40k). They suggest using this modified model: >A REAP-pruned (≈22% of experts removed), Int8-mix NVFP4 quantized version of GLM-5.2, ≈594B parameters. I wonder how it behaves in practice outside of benchmarks. Qwen3.6, even at 6-bit quantization, often gets stuck in loops while r…

"GLM 5.2 is "almost Opus," and it needs at least 8xH200s for comfortable inference ..."

What is the behavior if one were to run GLM 5.2 with only a single H200 ?

Would it fail to run at all, or would it just run so slowly as to be unusable ?

I would like to prove out the build, and concept, of a SOTA model locally, but then backfill the rest of the GPUs in 18-24 months when they cost significantly less ...

Re: Jamesob's guide to running SOTA LLMs locally

#82

Post by justincormack »

mwcampbell wrote: post #33
api wrote: post #13

Apple M series chips deserve a mention as another option, especially since you get a whole Mac laptop or desktop workstation too. They have unified memory and respectable inference performance, and for some variations can be cheaper than video cards, especially if you get an older-gen high-end M series with a lot of RAM used or refurbished. I've read that Apple has plans once the RAM bottleneck passes to offer more R…

> once the RAM bottleneck passes Do we have evidence that this will actually happen? Maybe the belief that it won't pass is what requires evidence, but I think there's a widespread feeling right now that things are just getting permanently worse and this is one example.

Micron have sold RAM for the next 4 years at current prices, so there are buyers expecting this to stay the same.

Re: Jamesob's guide to running SOTA LLMs locally

#83

Post by weystrom »

While I think that local LLMs are the future, i think these setups are insane. You shouldn't be trying to push the SOTA, most people underestimate how much you can get out of small LLMs.

Why ask FABLE 5000 to "summarize this email thread" when a tiny model can do the job?

Sure Codex3000 can oneshot your backlog, but why not use a subsidized subscription to do it for now? We're clearly not at the peak of these model's capabilities yet.

Re: Jamesob's guide to running SOTA LLMs locally

#84

Post by saltamimi »

Could someone give me an actual guide for spending as little as possible to get as maximal gains with either SOTA or cheap models as a systems administrator and not someone like a full-stack developer?

The models are so powerful and consequently so expensive and confusing to use, I don't get all of it.

Re: Jamesob's guide to running SOTA LLMs locally

#85

Post by ineptech »

Might as well add my own experience since I just set up a local llm this week. I went with a 32GB card made by Intel called Arc B70, which is cheaper than a 3090 and more has ram, at the cost of a slower memory bus. edited to remove something incorrect, thanks diablod3

I went with this because a) the models I wanted to use are a little too big to fit comfortably in 24gb, plus I need room for a few additional small models for autocomplete and speech recognition, and b) I already had a cheap server to use and dual gpus would've required upgrading the mobo and power supply and probably the case as well.

It was definitely a little tricky to set up. The Intel line requires a driver package called "level zero" to support something called SYCL (Intel's version of CUDA basically, AFAICT) that was tricky to get working. I am running llama.cpp in a docker container, which also required some fiddling to get the container to see the card. You also need a kernel from the last few months.

Once I got it working though, the results are very impressive for a $1k investment. Qwen 3.6 35B at q4 quantization takes about 3/4 of the ram and delivers like 88 tokens/sec. So, if you want a decent-sized model for cheap, this is one way to go.

Re: Jamesob's guide to running SOTA LLMs locally

#86

Post by DiabloD3 »

ineptech wrote: post #85

Might as well add my own experience since I just set up a local llm this week. I went with a 32GB card made by Intel called Arc B70, which is cheaper than a 3090 and more has ram, at the cost of a slower memory bus. edited to remove something incorrect, thanks diablod3 I went with this because a) the models I wanted to use are a little too big to fit comfortably in 24gb, plus I need room for a few additional small mo…

That is incorrect.

They both have GDDR6.

The B70 has 256 bit it bus at a clock speed of 2375mhz (608 GB/s), the 3090 has a 384 bit bus at a clock speed of 2438mhz (936 GB/s).

It isn't slower, it just has less channels, ie, it is less wide.

Re: Jamesob's guide to running SOTA LLMs locally

#87

Post by CobaltFire »

mips_avatar wrote: post #78
CobaltFire wrote: post #71

Earlier quoted context omitted.

$4.8k for 48GB Max (what the parent said). Half of your quote. Even a 128GB is $6.8k today. Still only 2/3 your quote. Bandwidth is relevant (I have both a 5090 and an M4 Max 128GB Studio, so have direct comparison right here), but quote the cost appropriately!

You need the 128gb ram config to get the 614 GB/s bandwidth (which is $6999), you could skip out on upgrading the storage to save money but at that point I think most people upgrade the storage too at which point it's $8-10k + tax.

No? Any M5 Max with the upgraded GPU has the full bandwidth, which includes the 48GB model the original poster mentioned. Same as the M4 Max, where only the trimmed part had a lower bandwidth.

Why are you throwing in extra cost for something thats not necessary? I know multiple people with 128GB Macs and none of us upgraded the storage. Especially not on a Studio (which isn't currently available).

I will say that their $3k number is off. I somehow missed that, and its too low.

Re: Jamesob's guide to running SOTA LLMs locally

#88

Post by Der_Einzige »

Aurornis wrote: post #16

I play with local LLMs a lot. I've spent more on hardware than I should. I'm friends with a local group of people who have spent a lot more than I have. The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like…

Everything in this post is spot on and it is a rare example of a HN person not saying BS about LLMs!

That said, modern LLM sampling algorithms like min_p, top_n sigma , etc heavily mitigate the performance penalty you get from doing long context tasks. Problems with long context come from accumulation of small sampling errors over time.

My qwen 3.6 27b (the dense one) runs perfectly well on coding tasks at the edge of its context window because I run it using modern LLM sampling stack, namely top N sigma of one, using DRY to stop repetitions and XTC as a superior alternative to temperature for diversification.

Yes there will be a paper soon on arxiv and hopefully NeurIPS proceedings talking about this phenomenon because it’s not well appreciated by the academic AI community yet.

Post reply on HN