151–160 of 495 posts
That's why Microsoft is not doing the deal with OpenAI, that's why Claude was fiddling with token limits just a couple of weeks ago.
It's a huge bubble, and the only winner at this moment is Nvidia.
This whole article is built off using DeepSeek R1, which is a huge premise that I don't think is correct. DeepSeek is much more efficient and I don't think it's a valid way to estimate what OpenAI and Anthropic's costs are. https://www.wheresyoured.at/deep-impact/ Basically, DeepSeek is _very_ efficient at inference, and that was the whole reason why it shook the industry when it was released.
We also don't know the per-token cost for OpenAI and Anthropic models, but I would be highly surprised if it was significantly more expensive than open models anyone can use and run themselves. It's not like they're also not investing in inference research.
From https://www.theverge.com/command-line-newsletter/759897/sam-... , Sam Altman said: > “If we didn’t pay for training, we’d be a very profitable company.”
Or if they had to pay copyright costs. So much pirated data being repackaged and sold.
I don't believe the asymmetry between prefill and decode is that large. If it were, it would make no sense for most of the providers to have separate pricing for prefill with cache hits vs. without. Given the analysis is based on R1, Deepseek's actual in-production numbers seem highly relevant: https://github.com/deepseek-ai/open-infra-index/blob/main/20... (But yes, they claim 80% margins on the compute in that arti…
Yes. I was really surprised at this myself (author here). If you have some better numbers I'm all ears. Even on my lowly 9070XT I get 20x the tok/s input vs output, and I'm not doing batching or anything locally. I think the cache hit vs miss stuff makes sense at >100k tokens where you start getting compute bound.
Earlier quoted context omitted.
That doesn't seem compatible with what he stated more recently: > We're profitable on inference. If we didn't pay for training, we'd be a very profitable company. Source: https://www.axios.com/2025/08/15/sam-altman-gpt5-launch-chat... His possible incentives and the fact OpenAI isn't a public company simply make it hard for us to gauge which of these statements is closer to the truth.
This can be true if you assume that there exists a high number of $20 subscribers who don't use the product that much, but $200 subscribers squeeze every last bit and then some more. The balance could be still positive, but if you look at the power users alone, they might cost more than they pay.
On the other side, there's an insane booster of speculative decoding, that would give a semi-prefill rate for decoding, but the memory pressure is still a factor.
I would be happy to be corrected regarding both factors.
This whole article is built off using DeepSeek R1, which is a huge premise that I don't think is correct. DeepSeek is much more efficient and I don't think it's a valid way to estimate what OpenAI and Anthropic's costs are. https://www.wheresyoured.at/deep-impact/ Basically, DeepSeek is _very_ efficient at inference, and that was the whole reason why it shook the industry when it was released.
The reason it shook the market at least was because of the claim that its training cost was 5 million.
This whole article is built off using DeepSeek R1, which is a huge premise that I don't think is correct. DeepSeek is much more efficient and I don't think it's a valid way to estimate what OpenAI and Anthropic's costs are. https://www.wheresyoured.at/deep-impact/ Basically, DeepSeek is _very_ efficient at inference, and that was the whole reason why it shook the industry when it was released.
Then, today almost every lab uses methods like speculative decoding and caching which reduce the cost and speed up things significantly.
The input numbers are far off. The assumption is 37B of active parameters. Sonnet 4 is supposedly a 100B-200B param model. Opus is about 2T params. Both of them (even if we assume MoE) wont have exactly these number of output params. Then there is a cost to hosting and activating params at inference time. (the article kind of assumes it would be the same constant 37B params).