Earlier quoted context omitted.
How could that possibly work? Deepseek was undercutting every other provider by an order of magnitude on cached tokens. Do they just set a super low caching time and hope that drops effective cache rates low enough? Do all other providers somehow overcharge by that much? Are they just going to sell it as a loss leader?
> Do all other providers somehow overcharge by that much? This, I think. Cached inputs have an opportunity cost (keeping the KV cache until use) but a hit is basically free. “Basically” - if the cache is offloaded to system RAM or NVMe there’s some scheduling overhead. From a consumer viewpoint a more interesting metric than the raw costs is cached cost * hitrate + input cost * (1 - hitrate) from a personal standpoin…
I don't have a clue on what the real cost to inference providers comes out to, but it seems really weird that there would be such a big gap, in what should be a pretty competitive market.