Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jamesob's guide to running SOTA LLMs locally

github.com

Post reply on HN
Print view View original

91–100 of 193 posts

Re: Jamesob's guide to running SOTA LLMs locally

#91

Post by Der_Einzige »

npodbielski wrote: post #68
WithinReason wrote: post #34

Earlier quoted context omitted.

I'm running Qwen3.6-27B on a single 24GB GPU at 80 tok/s, you don't even need 2 of them

Yeah but 4 bits very often loops needlessly. Which is not that bad because you do not pay for tokens. But you paid for hardware and you want use it for something useful. Q6 is better but then you have like 40t/s prefill. Which is really tiring. But at least it says sorry when you ask it what is wrong! I heard there is some extension for PI preventing that. I need to look into it. Otherwise I am quite happy.

You can fix looping with proper repetition penalties. Turn on the one called “DRY” that PeW invented and got merged into llama cpp

Re: Jamesob's guide to running SOTA LLMs locally

#92

Post by Der_Einzige »

kgeist wrote: post #9

>$40k gets you almost-Opus GLM 5.2 is "almost Opus," and it needs at least 8xH200s for comfortable inference (so it's closer to $400k than $40k). They suggest using this modified model: >A REAP-pruned (≈22% of experts removed), Int8-mix NVFP4 quantized version of GLM-5.2, ≈594B parameters. I wonder how it behaves in practice outside of benchmarks. Qwen3.6, even at 6-bit quantization, often gets stuck in loops while r…

Looping, like most other phenomenons related to LLMs, is a sampling problem and can be easily solved with the DRY penalty. It’s in llamacpp. The same guy who wrote heretic invented the SOTA antilooping and diversification strategies.

Re: Jamesob's guide to running SOTA LLMs locally

#93

Post by Der_Einzige »

nabakin wrote: post #72
turova wrote: post #21

For qwen3.6-27b you can also run the q4 variant with full ~250K context on one 3090. It's fast enough to not be frustrating so the speed gains with 2x 3090s wouldn't be worth it to me. Running a q6 on 2x 3090s at half the speed with a smaller context is an option, but you're really not going to compete with SOTA models there anyway so unless you already have 2x 3090s, I would say 1 is the best investment given curren…

Are you running qwen3.6-27b on one 3090 with your KV cache at q4? Ime there is significant long-context recall accuracy degradation at that precision. I prefer putting the KV cache at q8 and working with the 120k context

Use modern samplers and you don’t need to limit yourself to 8bit at half the context window. I could push it down to 1.58 bits and get decently good output easily by simply not using the garbage default top_p and top_k that vendors continue to wrongly recommend.

Re: Jamesob's guide to running SOTA LLMs locally

#95

Post by ineptech »

DiabloD3 wrote: post #86
ineptech wrote: post #85

Might as well add my own experience since I just set up a local llm this week. I went with a 32GB card made by Intel called Arc B70, which is cheaper than a 3090 and more has ram, at the cost of a slower memory bus. edited to remove something incorrect, thanks diablod3 I went with this because a) the models I wanted to use are a little too big to fit comfortably in 24gb, plus I need room for a few additional small mo…

That is incorrect. They both have GDDR6. The B70 has 256 bit it bus at a clock speed of 2375mhz (608 GB/s), the 3090 has a 384 bit bus at a clock speed of 2438mhz (936 GB/s). It isn't slower, it just has less channels, ie, it is less wide.

Whoops thanks, was going from memory. At any rate, the effect is that it's somewhat slower than the 3090, when using a model small enough to fit entirely in nvram, but can fit models the 3090 can't.

Re: Jamesob's guide to running SOTA LLMs locally

#96

Post by Zambyte »

npodbielski wrote: post #68
WithinReason wrote: post #34

Earlier quoted context omitted.

I'm running Qwen3.6-27B on a single 24GB GPU at 80 tok/s, you don't even need 2 of them

Yeah but 4 bits very often loops needlessly. Which is not that bad because you do not pay for tokens. But you paid for hardware and you want use it for something useful. Q6 is better but then you have like 40t/s prefill. Which is really tiring. But at least it says sorry when you ask it what is wrong! I heard there is some extension for PI preventing that. I need to look into it. Otherwise I am quite happy.

"Very often" sounds like a lot more than I would say. I've been using Qwen 3.6 27b Q4 in Pi (with out any anti-looping extension) daily for weeks now, and I've had it get stuck in an infinite loop maybe 3 or 4 times.

Re: Jamesob's guide to running SOTA LLMs locally

#98

Post by titanomachy »

mips_avatar wrote: post #65
datadrivenangel wrote: post #5

"A great way to go is 2x RTX 3090s for a total of 48GB VRAM total. You can then run Qwen3.6-27B, which is an awesome model." Just want to note that for $3k you can get an M5 macbook pro with 48gb of shared memory, and it will not be a giant box. Also, consider committing to spending that money on a cloud hosting provider, which will be at least somewhat cheaper if not significantly cheaper. It is awesome being able t…

The cool thing about the 3090s is the RAM bandwidth. Token generation is mostly bottlenecked on memory bandwidth. Dual 3090s have 1.87 TB/s memory bandwidth (0.936 TB/s each), vs the M5 Macbook pro with only 0.3 TB/s (max chip has up to 0.63 TB/s but it's a $10k machine at that config). This translates to qwen 27b actually working fast enough for useful work on dual 3090s and being painfully slow on Macbook Pros. Als…

The bandwidth argument is compelling, do we have benchmarks for these models? I’m curious what it translates to in tokens per second

Re: Jamesob's guide to running SOTA LLMs locally

#99

Post by nijave »

Aurornis wrote: post #16

I play with local LLMs a lot. I've spent more on hardware than I should. I'm friends with a local group of people who have spent a lot more than I have. The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like…

Yeah, I really wish articles and comments about " running locally" also reran the same common benchmarks published to compare the results.
Post reply on HN