For qwen3.6-27b you can also run the q4 variant with full ~250K context on one 3090. It's fast enough to not be frustrating so the speed gains with 2x 3090s wouldn't be worth it to me. Running a q6 on 2x 3090s at half the speed with a smaller context is an option, but you're really not going to compete with SOTA models there anyway so unless you already have 2x 3090s, I would say 1 is the best investment given current prices. It's good enough to do a lot, especially with a well-configured harness.
Jamesob's guide to running SOTA LLMs locally
21–30 of 193 posts
Re: Jamesob's guide to running SOTA LLMs locally
#22I play with local LLMs a lot. I've spent more on hardware than I should. I'm friends with a local group of people who have spent a lot more than I have. The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like…
All very true. Right now, running GLM 5.2 at its full BF16 quantization level needs 1.5 TB of VRAM. You can't run this locally at a usable speed for less than $250K or so, and frankly I'd be surprised if it could be done for less than $500K. The best NV4FP quant for 5.2 appears to be lukealonso's at https://huggingface.co/lukealonso/GLM-5.2-NVFP4 , and it is capable of good throughput (75-100 tps) without losing much…
> It'll almost certainly be worth it, given the abusive behavior we've seen and will continue to see from the major closed-model providers.
The proper financial comparison for GLM-5.2 would be one of the providers on OpenRouter or renting a server as needed. Compare apples to apples.
You will almost certainly never break even compared to paying per token.
Local LLMs at this scale are only worth it if you have extremely strict requirements that data not leave the premises.
Re: Jamesob's guide to running SOTA LLMs locally
#23Earlier quoted context omitted.
All very true. Right now, running GLM 5.2 at its full BF16 quantization level needs 1.5 TB of VRAM. You can't run this locally at a usable speed for less than $250K or so, and frankly I'd be surprised if it could be done for less than $500K. The best NV4FP quant for 5.2 appears to be lukealonso's at https://huggingface.co/lukealonso/GLM-5.2-NVFP4 , and it is capable of good throughput (75-100 tps) without losing much…
> It'll almost certainly be worth it, given the abusive behavior we've seen and will continue to see from the major closed-model providers. The proper financial comparison for GLM-5.2 would be one of the providers on OpenRouter or renting a server as needed. Compare apples to apples. You will almost certainly never break even compared to paying per token. Local LLMs at this scale are only worth it if you have extreme…
Or if you want to hedge against the various tail risks of third-party providers raising prices or denying you service or somehow abusing your data...
Re: Jamesob's guide to running SOTA LLMs locally
#24Earlier quoted context omitted.
All very true. Right now, running GLM 5.2 at its full BF16 quantization level needs 1.5 TB of VRAM. You can't run this locally at a usable speed for less than $250K or so, and frankly I'd be surprised if it could be done for less than $500K. The best NV4FP quant for 5.2 appears to be lukealonso's at https://huggingface.co/lukealonso/GLM-5.2-NVFP4 , and it is capable of good throughput (75-100 tps) without losing much…
> It'll almost certainly be worth it, given the abusive behavior we've seen and will continue to see from the major closed-model providers. The proper financial comparison for GLM-5.2 would be one of the providers on OpenRouter or renting a server as needed. Compare apples to apples. You will almost certainly never break even compared to paying per token. Local LLMs at this scale are only worth it if you have extreme…
Also agreed, it's definitely a sucker's game to run a high-end model locally, by any objective measure.
Still... if it's not your weights, running on your box, you're always going to be behind somebody else's 8-ball. Everybody has to decide for themselves where their priorities lie.
Re: Jamesob's guide to running SOTA LLMs locally
#25>$40k gets you almost-Opus GLM 5.2 is "almost Opus," and it needs at least 8xH200s for comfortable inference (so it's closer to $400k than $40k). They suggest using this modified model: >A REAP-pruned (≈22% of experts removed), Int8-mix NVFP4 quantized version of GLM-5.2, ≈594B parameters. I wonder how it behaves in practice outside of benchmarks. Qwen3.6, even at 6-bit quantization, often gets stuck in loops while r…
You can get 1M context with the lukealonso NVFP4 quant on 8x RTX6000s, which remains coherent and useful through at least 400k. No real need to run 8x H200s unless you just want to. Or unless you need to serve many concurrent users or agents on a regular basis.
Re: Jamesob's guide to running SOTA LLMs locally
#26Earlier quoted context omitted.
> It'll almost certainly be worth it, given the abusive behavior we've seen and will continue to see from the major closed-model providers. The proper financial comparison for GLM-5.2 would be one of the providers on OpenRouter or renting a server as needed. Compare apples to apples. You will almost certainly never break even compared to paying per token. Local LLMs at this scale are only worth it if you have extreme…
Or if you want to hedge against the various tail risks of third-party providers raising prices or denying you service or somehow abusing your data...
> hedge against the various tail risks of third-party providers raising prices
They could 10X the prices and you’d still be better off. It’s also unlikely that prices go up enough to warrant a $100K local investment to prevent paying a couple bucks per million tokens.
> or denying you service
I guess you’re not familiar with OpenRouter? There are many providers there. There are providers outside of OpenRouter. There will always be someone to take your business.
> or somehow abusing your data...
If data security is your concern then you’re better renting a server as needed still.
If you cannot tolerate any data leaving, then local models are the only way. You pay a high premium for it!
Re: Jamesob's guide to running SOTA LLMs locally
#27Earlier quoted context omitted.
> It'll almost certainly be worth it, given the abusive behavior we've seen and will continue to see from the major closed-model providers. The proper financial comparison for GLM-5.2 would be one of the providers on OpenRouter or renting a server as needed. Compare apples to apples. You will almost certainly never break even compared to paying per token. Local LLMs at this scale are only worth it if you have extreme…
Or if you want to hedge against the various tail risks of third-party providers raising prices or denying you service or somehow abusing your data...
Raising prices is not a tail risk, anything a local LLM setup can do for you can be done by any cloud provider, with the same capex as yours (or less), there is no moat here, so it is highy price competitive and will remain so. If you want to speculate on hardware shortages, that is a different business altogether and you need no janky garage setup to profit.
Re: Jamesob's guide to running SOTA LLMs locally
#28You can get amazing local STT using parakeet which can use as little as 600mb of vram. Better or as good as whisper v3 large
[flagged]
Re: Jamesob's guide to running SOTA LLMs locally
#29I play with local LLMs a lot. I've spent more on hardware than I should. I'm friends with a local group of people who have spent a lot more than I have. The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like…
Well you could make a REAP with better input prompts on longer context then. It’ll improve the REAP quality
Re: Jamesob's guide to running SOTA LLMs locally
#30"A great way to go is 2x RTX 3090s for a total of 48GB VRAM total. You can then run Qwen3.6-27B, which is an awesome model." Just want to note that for $3k you can get an M5 macbook pro with 48gb of shared memory, and it will not be a giant box. Also, consider committing to spending that money on a cloud hosting provider, which will be at least somewhat cheaper if not significantly cheaper. It is awesome being able t…
You can also buy a Jetson Orin with 64GB of unified memory.