71–80 of 91 posts
This is just making it more expensive for everyone else, right? How Claude handles its sessions is none of my business. I'm going to let them do the best they can to provide good service for everyone, and if they can't/won't, I'll switch to a provider that can. Using these massive models is already pretty danged extravagant, I'm not going to demand to be at the front of the queue at all times, too.
hey guys dario’s here!
This is just making it more expensive for everyone else, right? How Claude handles its sessions is none of my business. I'm going to let them do the best they can to provide good service for everyone, and if they can't/won't, I'll switch to a provider that can. Using these massive models is already pretty danged extravagant, I'm not going to demand to be at the front of the queue at all times, too.
Earlier quoted context omitted.
"Earn"? Cache duration is arbitrary. What it actually does (if used en masse) is decrease the amount of oversubscription their infra can handle..
Earn is synonymous with profit, and this hurts their profits. Introducing latency can benefit much more people and actually make usage more efficient. Reducing latency can sieze up a functioning system. Lately, I’ve been thinking about how this related to fractional banking. If you were to eliminate fractional banking introduced in the US by Hamilton, you would destroy a lot of current prosperity.
Earlier quoted context omitted.
You PAY for cache writes. Holding the prefill in VRAM for a certain period of time is part of what you're paying for.
Right, but also using the cache refreshes/extends it, which means you can pay for a write once, then have it extended for very cheap (1 token).
This is just making it more expensive for everyone else, right? How Claude handles its sessions is none of my business. I'm going to let them do the best they can to provide good service for everyone, and if they can't/won't, I'll switch to a provider that can. Using these massive models is already pretty danged extravagant, I'm not going to demand to be at the front of the queue at all times, too.
Cached input tokens cost 10% of uncached. So if you’re model runs for 45 minutes, generates 300k output tokens and asks you a question, it costs 10x more if you wait 5.01 minutes to answer.
Sure, you may be willing to pay 10x more (or get 10x less for your subscription). But the time limit is arbitrary and has nothing to do with other peoples’ workloads. So I think your point is a non sequitur.
Earlier quoted context omitted.
How do a few extra requests with the same prefix make it more expensive for everyone else?
The prefix cache is a resource shared by all users. This is basically a tragedy of the commons.
I don’t think your understanding works.
Earlier quoted context omitted.
I highly doubt that. Keeping your cache warm is a good thing, caching saves compute and electricity. Cached input is cheap for a reason, it is in everyone’s mutual interests to maximise cache hit rates.
If that was true then Anthropic would have built this functionality in themselves.
This is just making it more expensive for everyone else, right? How Claude handles its sessions is none of my business. I'm going to let them do the best they can to provide good service for everyone, and if they can't/won't, I'll switch to a provider that can. Using these massive models is already pretty danged extravagant, I'm not going to demand to be at the front of the queue at all times, too.
This has nothing to do with queueing. Cached input tokens cost 10% of uncached. So if you’re model runs for 45 minutes, generates 300k output tokens and asks you a question, it costs 10x more if you wait 5.01 minutes to answer. Sure, you may be willing to pay 10x more (or get 10x less for your subscription). But the time limit is arbitrary and has nothing to do with other peoples’ workloads. So I think your point is…
No it has to do with others workloads, now you keep their cache for longer so others will get less. And no its not arbitrary, they run out of memory, if more people do this they will have the dial it down further or run out of capacity.
This is just making it more expensive for everyone else, right? How Claude handles its sessions is none of my business. I'm going to let them do the best they can to provide good service for everyone, and if they can't/won't, I'll switch to a provider that can. Using these massive models is already pretty danged extravagant, I'm not going to demand to be at the front of the queue at all times, too.
This has nothing to do with queueing. Cached input tokens cost 10% of uncached. So if you’re model runs for 45 minutes, generates 300k output tokens and asks you a question, it costs 10x more if you wait 5.01 minutes to answer. Sure, you may be willing to pay 10x more (or get 10x less for your subscription). But the time limit is arbitrary and has nothing to do with other peoples’ workloads. So I think your point is…
2) Now imagine Anthropic or OpenAI now charge your per minute of reserved VRAM time. It would be more fair if they did. Would you still want to run a tool like this?
Earlier quoted context omitted.
Keeping a conversation with a very large model active requires hundreds of GB of memory. If my conversation can never be swapped out, like when I go to lunch or take an afternoon walk, that's several hundred GB not available for other users. If everybody does that, Anthropic needs even more infrastructure than the quarter trillion dollars in infra they're already using.
But you pay for cache writes, it's not like you're abusing anything, you're simply choosing to call a billed API for the explicitly intended purpose of writing/refreshing cache. If you keep this running for hours without doing anything, it will drain your limits and API. The use case of keeping the main thread cache warm while subagents work is very genuine and legitimate.