Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Claude-thermos keeps your Claude session warm for you

github.com

Post reply on HN
Print view View original

21–30 of 91 posts

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#21

Post by s0ck_r4w »

cortesoft wrote: post #17
ATMLOTTOBEER wrote: post #12

Glad this exists. It will force anthropic to fix their flawed cache mechanism.

How should they fix it?

Make the cache TTL more adaptive. Have more tiers than just 5m and 1h. Long-running subagents become increasingly ubiquitous. There's no reason for Anthropic to not do a better job for this use case.

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#22

Post by davesque »

cosmotic wrote: post #20

How will this not lead to tragedy of the commons?

Exactly. I really wish people wouldn't use this. If this becomes popular, anthropic will just modify their cache policy to be much less fair. It's not like they have infinite cache.

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#23

Post by addaon »

cortesoft wrote: post #17
ATMLOTTOBEER wrote: post #12

Glad this exists. It will force anthropic to fix their flawed cache mechanism.

How should they fix it?

There are a lot of workflows where cache is likely to be consumed exactly once (e.g. anything chat-like where a single thread is extended; if a prefix cache hit is found, and a set of new longer prefixes is inserted in the cache, it is unlikely that the original shorter prefix will be consumed in later turns). I could see allowing each session key a small number (1?) of 24-hour cache entries, where inserting a new cache entry (perhaps the maximum-length prefix, perhaps an API-tagged specified-length prefix) consumes that slot and either boots the rest of the items from cache, or demotes them to 5 minutes or something. Basically: workflow awareness, not uniform handling.

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#24

Post by supern0va »

gogobio wrote: post #6
alukin wrote: post #2

Feels like will be shut down real quick

Yes, this wastes cycle. They dump your cache and deallocate the VMs so that others can use it. This will result in tokens wasted and downtime for others.

Can you expand on this? I can't imagine this is holding up an entire machine or materially impacting capacity. Isn't it common to have a tiered cache and to ultimately evict (or move to another tier) if there's an active inference request and no available capacity elsewhere?

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#25

Post by janderson215 »

cadamsdotcom wrote: post #13
agluszak wrote: post #5

Earlier quoted context omitted.

Because it makes Anthropic earn less money

"Earn"? Cache duration is arbitrary. What it actually does (if used en masse) is decrease the amount of oversubscription their infra can handle..

Earn is synonymous with profit, and this hurts their profits. Introducing latency can benefit much more people and actually make usage more efficient. Reducing latency can sieze up a functioning system.

Lately, I’ve been thinking about how this related to fractional banking. If you were to eliminate fractional banking introduced in the US by Hamilton, you would destroy a lot of current prosperity.

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#26

Post by Lalabadie »

supern0va wrote: post #24
gogobio wrote: post #6

Earlier quoted context omitted.

Yes, this wastes cycle. They dump your cache and deallocate the VMs so that others can use it. This will result in tokens wasted and downtime for others.

Can you expand on this? I can't imagine this is holding up an entire machine or materially impacting capacity. Isn't it common to have a tiered cache and to ultimately evict (or move to another tier) if there's an active inference request and no available capacity elsewhere?

Yes, however if a tool like this commoditizes a way to stay "higher" in the cache hierarchy to avoid getting dropped, then for the same total capacity, Anthropic have to become stricter about retention.

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#27

Post by supern0va »

Lalabadie wrote: post #26
supern0va wrote: post #24

Earlier quoted context omitted.

Can you expand on this? I can't imagine this is holding up an entire machine or materially impacting capacity. Isn't it common to have a tiered cache and to ultimately evict (or move to another tier) if there's an active inference request and no available capacity elsewhere?

Yes, however if a tool like this commoditizes a way to stay "higher" in the cache hierarchy to avoid getting dropped, then for the same total capacity, Anthropic have to become stricter about retention.

It's unfortunate that there isn't some better way to signal that there's a high chance that a turn is coming (ie, due to active sub-agents). If CC could send a ping saying "I don't need a turn, but please keep this cached at some tier because there will be a turn soon." that would probably help with cache efficiency. Instead, it seems like there's just the shotgun one-hour solution.

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#29

Post by SwellJoe »

This is just making it more expensive for everyone else, right?

How Claude handles its sessions is none of my business. I'm going to let them do the best they can to provide good service for everyone, and if they can't/won't, I'll switch to a provider that can.

Using these massive models is already pretty danged extravagant, I'm not going to demand to be at the front of the queue at all times, too.

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#30

Post by SwellJoe »

ATMLOTTOBEER wrote: post #12

Glad this exists. It will force anthropic to fix their flawed cache mechanism.

While Anthropic is far behind on caching and efficiency (made more dramatically apparent by how much cheaper it is to use GPT 5.6 Sol at API rates than even Opus 4.8, much less Fable), a bunch of people forcing their way to the front of the queue at the expense of everyone else isn't going to solve that.
Post reply on HN