31–40 of 48 posts
Training is the expensive part here. It seems much more likely that the training of these models slows down drastically and is written off as a sunk cost, a few companies continue running inference on years-old models, and the free versions go away.
This is addressed in the very first sentence of the article that you obviously didn't read.
I dunno, GPT-OSS and Llama and QWEN and any half dozen of other large open-weight models? I really can't imagine OpenAI or Anthropic turning off inference for a model that my workplace is happy to spend >$200*person/month on. Google still has piles of cash and no reason to turn off Gemini. The thing is, if inference is truly heavily subsidized (I don't think it is, because places like OpenRouter charge less than the…
> we'd probably happily pay >$500 a month for the current frontier models Try $5,000. OpenAI loses hundreds of billions a year, they need a 100x, not 2x.
Perhaps this is a helpful model, rather than worrying about the "billions spent" and whether its inference vs training. How much would it cost you to deploy a model that you and maybe a few coworkers could effectively use? $400k probably to buy all the hardware required to host a top-tier model that could do a few hundred tokens per second for 10 concurrent users? That's $40k per person. Ammortize the hardware over 5…
It can easily serve 10 people or more depending on the overall usage pattern (coding vs everything else).
So now imagine Hardware getting better every year, models getting better too and everything overall gets more efficient.
M1 vs M4 apple increased performance by 100% in 4 years
And there are inferencing optimized chips like groq.
Don't forget kv cache and overall optimizations.
I think your math is off.
And ai is already better than interns. An intern costs you at least 1k per month probably 2k.
For me the math works just fine.
Perhaps this is a helpful model, rather than worrying about the "billions spent" and whether its inference vs training. How much would it cost you to deploy a model that you and maybe a few coworkers could effectively use? $400k probably to buy all the hardware required to host a top-tier model that could do a few hundred tokens per second for 10 concurrent users? That's $40k per person. Ammortize the hardware over 5…
The only reason hardware is so expensive now is to scalp the hyperscalers. Once that demand crashes, the supply will skyrocket and prices will crash.
> Self-hosting an AI with your own hardware is probably just as cost-prohibitive, even if you don't value your time. In part because a ton of people will get this idea at the same time, impacting hardware prices even more. And the operating costs of AI seem significant. Would it even be possible to setup your own AI and achieve the same productivity level? I know this is probably an annoying question, but… has the au…
I have an old dual Xeon server from about 2015. 32 2.4GHz cores and 128GB of RAM. It runs models painfully slow (and loud) but they run just fine. My modern Ryzen system from last year works out of the box with full AMD GPU support.
I have yet to find a situation where ollama doesn't work at all out of the box. It literally just turns on and goes. Maybe slow, maybe without GPU, but by god you'll have an LLM running
I think there is too much focus on the article and too little focus on that the host is some sort of dyi solar powered server.
Ads, obviously
I think there is too much focus on the article and too little focus on that the host is some sort of dyi solar powered server.
For the valuable kick start usecase it pays off. It cant do all the magic bootsrraps but for baseline technical questions its perfect. Will put in a rag search eventuallly.
Im not optimistic any use case will come to substantiate todays valuations. But the intertwined fascist businesses is going to stunt a lot of people trying to chain their product to 3rd parties.