$2k or $40k? One of those is not "self host."
Depends how much money you made going long on AI stocks.
Jamesob's guide to running SOTA LLMs locally
141–150 of 193 posts
Re: Jamesob's guide to running SOTA LLMs locally
#142why in gods name is a RTX PRO 6000 $13,000? supply and command?
About £4000 on eBay uk right now. Because if they were any lower we’d all be buying six each.
Re: Jamesob's guide to running SOTA LLMs locally
#143I play with local LLMs a lot. I've spent more on hardware than I should. I'm friends with a local group of people who have spent a lot more than I have. The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like…
Absolutely true. All this craze about running coding LLMs locally has been detrimental to local AI where purpose built SLMs could actually be beneficial.
Little tools for NLP, TTS, image processing, audio engineering, signal processing, diffusion plugin for Krita etc. are all great for local setup. I wrote a small piece on it few days back[1].
[1] https://abishekmuthian.com/multiple-20-ai-plans-are-better-t...
Re: Jamesob's guide to running SOTA LLMs locally
#144There also exists an in-between possibility, that is, if you get 128GB of vram (there are now multiple options in the market to get that amount with a unified memory architecture) you can run DeepSeek V4 flash at good speed via DwarfStar. I'm not going to spend money on this, but my gut feeling is that this would be the right compromise for a lot of people.
I just started using it on an m4 max 128 and it's the first time since buying the machine a year ago that it feels like local llm "just works" for reasonably decent coding. Use pi though; claude code has way too much bootstrap context; slows everything way down.
Definitely seconding pi. Also avoid opencode, it doesn't support caching (mutates the system prompt constantly)
Re: Jamesob's guide to running SOTA LLMs locally
#145Earlier quoted context omitted.
> It'll almost certainly be worth it, given the abusive behavior we've seen and will continue to see from the major closed-model providers. The proper financial comparison for GLM-5.2 would be one of the providers on OpenRouter or renting a server as needed. Compare apples to apples. You will almost certainly never break even compared to paying per token. Local LLMs at this scale are only worth it if you have extreme…
Never say never. When the free money party stops, then those token costs are going to have to go up and up. The fact there’s such a glaring disparity between the cost of running AI locally and the pennies it costs to use an online model shows how heavily funded those platforms are right now. This is not and cannot be sustainable.
> When the free money party stops
The Openrouter providers the GP referenced were never at the "free money party". The actual cost of running something like GLM5.2 is well understood and tokens from those providers are not sold at a loss.
Obviously running things locally is more expensive but that all comes down to economies of scale. GLM5.2 is as expensive as it will ever be, barring an increase in demand that forces/allows providers to realise windfall gains disconnected from their underlying costs (always possible, but not the point).
Re: Jamesob's guide to running SOTA LLMs locally
#146Earlier quoted context omitted.
Stop trying to run them locally, folks. You don't own your fiber connection. So why try to own another rapidly depreciating, expensive, and annoying asset? Rent cloud GPUs! You get to participate in the ownership, data control, price control, and hacking culture without having to Frankenstein some hobbyist box that costs a ton, is distilled down to functional uselessness, and is a PITA to maintain.
This comment is like the antithesis of hacker culture. Can’t tell if being ironic or not.
Lobotomized RTX models are playthings.
People building this stuff are "year of linux on desktop"ing open weights AI. It's a huge opportunity cost - not just for you, but for the open source community at large.
You need to double down on big fat honking models that take multiple H200s to run. That's where the real power lies, and that's where our entire community needs to focus our efforts if we want to keep the delta between frontier and the proletariat small.
The more we build for people and enterprises to run big weights in private clouds, the better. That's the real treat to Google, Anthropic, and OpenAI. Your RTX cards don't make a dent in the death star.
Re: Jamesob's guide to running SOTA LLMs locally
#147Earlier quoted context omitted.
> in 18-24 months when they cost significantly less ... going to need you to sit down for this one...
Say more. My expectation is that the current gen of gpus will start being replaced by the next gen, and then it may be possible to get used ones that are still within their useful life at lower prices. My expectation is also that memory vendors are likely to increase production, which will drive those prices down eventually. Maybe not over the next 18-24 months though.
I sat in a meeting 7 weeks ago where senior leaders said they expect token prices to drop significantly over the next 6 months, and we should all be using as much AI as possible; our team goal was set to use more tokens.
This week, we are banned from using anything more expensive than opus 4.6 and encouraged to use sonnet (but not sonnet 5! Thats expensive!) or lower for daily tasks to help manage costs.
Weeks ago, they gave exactly the same justification as you just gave; and it makes sense!
…but maybe not over the next 6 months.
> Maybe not over the next 18-24 months
Maybe not. Probably not, I guess.
A lot of money has been invested on the expectation that the current gen of hardware is going to reap a colossal profit, and the capex to replace it, is vanishing into investor skepticism as we speak.
It seems like most people have a very very low ability to forecast long horizon change in the current environment, but, in general… it seems like until demand drops, the chances of prices dropping is dubious; at best we get a price war with chinese models or a bubble pop; and even then, there are plenty of startups lurking to snap up cheap hardware.
For individuals, the horizon for buying cheap AI capable compute doesn’t seem close, at all, to me.
Re: Jamesob's guide to running SOTA LLMs locally
#148Earlier quoted context omitted.
This comment is like the antithesis of hacker culture. Can’t tell if being ironic or not.
Lobotomized RTX models are playthings. People building this stuff are "year of linux on desktop"ing open weights AI. It's a huge opportunity cost - not just for you, but for the open source community at large. You need to double down on big fat honking models that take multiple H200s to run. That's where the real power lies, and that's where our entire community needs to focus our efforts if we want to keep the delta…
You could be an IBM executive writing about the Apple I.
Re: Jamesob's guide to running SOTA LLMs locally
#149> "~$40k At this price level, you get the next step up in model intelligence. Something pretty close to Claude Opus." That is equivalent to 16.8 years of Claude Opus 4.8 or Codex GPT 5.5 at $200/mo. I'm a huge fan of running local models, but they're still wildly expensive, lower quality, and possibly dangerous (if backdoored). I sincerely wish this wasn't the case.
I agree with your point, but it should be noted that this assumes consistent prices for LLMs. The OpenAIs and Anthropics of this world are still selling the plans at a subsidised prices with the power of VCs, who are going to want that return some time.
VCs need to sit on ice for a few more years if they dont want it to pop
Re: Jamesob's guide to running SOTA LLMs locally
#150You can get amazing local STT using parakeet which can use as little as 600mb of vram. Better or as good as whisper v3 large
Have we reached the capability of a local STT+LLM system being constantly listening for normal speech in a room and being able to understand when the human is addressing the system instead of talking to another human?