51–60 of 91 posts
Interested in how the critics of approaches like this defend an agentic session (with Fable, for example) that stops and runs a multi-hour ML training session. It's a script, so the actual LLM convo goes stale, but then when the results get returned to the main thread you get an expensive cache hit without doing anything. You would have avoided that cache hit if the LLM session was kept "alive" for those few hours. W…
Earlier quoted context omitted.
How do a few extra requests with the same prefix make it more expensive for everyone else?
The prefix cache is a resource shared by all users. This is basically a tragedy of the commons.
https://code.claude.com/docs/en/prompt-caching#on-a-claude-s...
(Thank you to EliasWatson for giving me this link just a few days ago, as I was previously confused too.)
How will this not lead to tragedy of the commons?
Glad this exists. It will force anthropic to fix their flawed cache mechanism.
While Anthropic is far behind on caching and efficiency (made more dramatically apparent by how much cheaper it is to use GPT 5.6 Sol at API rates than even Opus 4.8, much less Fable), a bunch of people forcing their way to the front of the queue at the expense of everyone else isn't going to solve that.
This is just making it more expensive for everyone else, right? How Claude handles its sessions is none of my business. I'm going to let them do the best they can to provide good service for everyone, and if they can't/won't, I'll switch to a provider that can. Using these massive models is already pretty danged extravagant, I'm not going to demand to be at the front of the queue at all times, too.
I thought the cache length was 1 hour, not 5 minutes?