Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jamesob's guide to running SOTA LLMs locally

github.com

Post reply on HN
Print view View original

31–40 of 193 posts

Re: Jamesob's guide to running SOTA LLMs locally

#31

Post by amelius »

Aurornis wrote: post #17
datadrivenangel wrote: post #5

"A great way to go is 2x RTX 3090s for a total of 48GB VRAM total. You can then run Qwen3.6-27B, which is an awesome model." Just want to note that for $3k you can get an M5 macbook pro with 48gb of shared memory, and it will not be a giant box. Also, consider committing to spending that money on a cloud hosting provider, which will be at least somewhat cheaper if not significantly cheaper. It is awesome being able t…

I have an M5 MacBook Pro and I also have a separate GPU setup for running models. The difference in speed is significant. It's not just token generation speed, but time to first token (prompt processing). The M5 hardware is amazing for what it is, but GPUs are still so much faster. Running the models on the GPU box also means I can use the laptop on my lap instead of turning it into a hot plate.

What is your GPU setup?

Re: Jamesob's guide to running SOTA LLMs locally

#33

Post by mwcampbell »

api wrote: post #13

Apple M series chips deserve a mention as another option, especially since you get a whole Mac laptop or desktop workstation too. They have unified memory and respectable inference performance, and for some variations can be cheaper than video cards, especially if you get an older-gen high-end M series with a lot of RAM used or refurbished. I've read that Apple has plans once the RAM bottleneck passes to offer more R…

> once the RAM bottleneck passes

Do we have evidence that this will actually happen? Maybe the belief that it won't pass is what requires evidence, but I think there's a widespread feeling right now that things are just getting permanently worse and this is one example.

Re: Jamesob's guide to running SOTA LLMs locally

#34

Post by WithinReason »

datadrivenangel wrote: post #5

"A great way to go is 2x RTX 3090s for a total of 48GB VRAM total. You can then run Qwen3.6-27B, which is an awesome model." Just want to note that for $3k you can get an M5 macbook pro with 48gb of shared memory, and it will not be a giant box. Also, consider committing to spending that money on a cloud hosting provider, which will be at least somewhat cheaper if not significantly cheaper. It is awesome being able t…

I'm running Qwen3.6-27B on a single 24GB GPU at 80 tok/s, you don't even need 2 of them

Re: Jamesob's guide to running SOTA LLMs locally

#35

Post by hypfer »

turova wrote: post #21

For qwen3.6-27b you can also run the q4 variant with full ~250K context on one 3090. It's fast enough to not be frustrating so the speed gains with 2x 3090s wouldn't be worth it to me. Running a q6 on 2x 3090s at half the speed with a smaller context is an option, but you're really not going to compete with SOTA models there anyway so unless you already have 2x 3090s, I would say 1 is the best investment given curren…

That math (250k context, Q4 model, 24GB VRAM) only checks out at q4 quant for the K/V cache, which is probably not the best idea.

Re: Jamesob's guide to running SOTA LLMs locally

#36

Post by randomblock1 »

chompychop wrote: post #32

Is Whisper still considered SOTA for STT? Since it came out years ago, I'd have assumed there are better models by now.

No, there are quite a few models which are smaller, more accurate, and faster. For example Parakeet TDT v3 is half the size, way faster, and lower WER. There's also Voxstral, which is much larger but also even more accurate.

But the ecosystem isn't as mature, so Whisper is still a valid option, even now. For example Parakeet uses Nemotron framework (made by Nvdia), normally you need CUDA, so you need to use an ONNX version instead on AMD. Meanwhile Whisper has VLLM and desktop apps like Buzz.

There aren't many benchmarks and they often don't have all the models, since STT doesn't get nearly enough attention as normal LLMs, but this is one of the more complete ones: https://artificialanalysis.ai/speech-to-text/non-streaming

Re: Jamesob's guide to running SOTA LLMs locally

#37

Post by venusenvy47 »

chompychop wrote: post #32

Is Whisper still considered SOTA for STT? Since it came out years ago, I'd have assumed there are better models by now.

I don't have anything to compare against, since I have just started using it. But I was fairly happy with it on my personal recordings from my phone. Also, I ran it on my CPU (Core i7) and it was perfectly usable, as something to run when not using the machine for anything else.

Re: Jamesob's guide to running SOTA LLMs locally

#38

Post by 3eb7988a1663 »

Related - what is the best isolation system available? Do I have to go full, fat VMs or can I get by with a Firecracker-like thing?

Seemingly every available option has some subtle-gotchas about how easy it is to blow off your foot and effectively have no security at all. I use VMs because I actually trust that security is a foundational principle of the technology, not a well-if-you-use-these-20-flags-and-squint kind of deal.

Re: Jamesob's guide to running SOTA LLMs locally

#39

Post by ZiiS »

3eb7988a1663 wrote: post #38

Related - what is the best isolation system available? Do I have to go full, fat VMs or can I get by with a Firecracker-like thing? Seemingly every available option has some subtle-gotchas about how easy it is to blow off your foot and effectively have no security at all. I use VMs because I actually trust that security is a foundational principle of the technology, not a well-if-you-use-these-20-flags-and-squint kin…

Full fat VMs with GPU passthough I trust a lot less then CPU ones.
Post reply on HN