Does anyone know any good data center to home conversion kits for gear?
Jamesob's guide to running SOTA LLMs locally
51–60 of 193 posts
Re: Jamesob's guide to running SOTA LLMs locally
#52Is Whisper still considered SOTA for STT? Since it came out years ago, I'd have assumed there are better models by now.
I'm a big fan of Parakeet v3 - I run it using the MacWhisper app, it's a 494MB model and the quality is excellent.
Re: Jamesob's guide to running SOTA LLMs locally
#53Earlier quoted context omitted.
> The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like 50-55K. > Local setups also often rely on quantization and techniques like REAP to fit the models on their hardware. This seems to ignore the very rea…
SSD streaming throughput is too slow to be usable. GLM-5.2 has 40B active parameters at a time. At Q4 that's 20GB. The best PCIe 5 SSDs can get 15GB/sec when everything goes well. Every expert load would take more than a second. If you had enough RAM and enough SSDs in parallel you might get a couple tokens per second on a good day. If you left this machine running 24 hours straight, you might be able to get 200,000…
You can improve that with speculative preload. I'm sure models could be designed and tuned around efficient SSD offloading to keep throughput pretty high.
Re: Jamesob's guide to running SOTA LLMs locally
#54If you can run sota on a 40k setup, why do openai etc spend maybe 100x that?
Re: Jamesob's guide to running SOTA LLMs locally
#55Related - what is the best isolation system available? Do I have to go full, fat VMs or can I get by with a Firecracker-like thing? Seemingly every available option has some subtle-gotchas about how easy it is to blow off your foot and effectively have no security at all. I use VMs because I actually trust that security is a foundational principle of the technology, not a well-if-you-use-these-20-flags-and-squint kin…
It depends - for what? If your security model is sandboxing an agent to ensure they don't nuke your PC, then there are a lot of options, you can use something like bubblewrap[1] or a microVM like libkrun[2] if your goal is light-weight, up to full Docker if you want the tooling that comes with that.
Re: Jamesob's guide to running SOTA LLMs locally
#56If you can run sota on a 40k setup, why do openai etc spend maybe 100x that?
Obvious one: Because they are serving it to millions of people at the same time, not just one local user
Re: Jamesob's guide to running SOTA LLMs locally
#57I play with local LLMs a lot. I've spent more on hardware than I should. I'm friends with a local group of people who have spent a lot more than I have. The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like…
They do say the cards were purchased when they were cheaper. They debuted at less than nine grand apparently.
Re: Jamesob's guide to running SOTA LLMs locally
#58Local open weight models will definitely be a future trend. Imagine if an Opus-level model could run locally: many more latent use cases would likely emerge, since Opus is priced so high. Perhaps the future will be a multi-model architecture, where frontier models handle planning and local models carry out the concrete execution.
Re: Jamesob's guide to running SOTA LLMs locally
#59I am somewhere in the middle, where I want something with more than 48GB/$2k of VRAM, but less than 384GB/$40k. I'm curious if GMKtec's EVO-X2, with ~96GB of usable VRAM, is still a good solution for something like this for $3,399.
I've been happy with an OEM Spark (128G), enough so that I picked up a second one. Have 2x qwen and 1x gemma (both at 8bit and full context), plus embedding, Re-Ranker, and a 1.7B for little things. Running 6x models, probably going to add STT here soon, want to try talking more than typing.
The caveat is that if you try to use multiple models on the same device at the same time, you thrash and destroy tok/s
Re: Jamesob's guide to running SOTA LLMs locally
#60did he call Qwen a SOTA model?
It can be considered SOTA within is size category. Very useful for many things. You still want access to big models, I recommend OpenCode Go if you want to stay with open models.