Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Claude-thermos keeps your Claude session warm for you

github.com

Post reply on HN
Print view View original

81–90 of 91 posts

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#81

Post by searealist »

dannyw wrote: post #65
kakugawa wrote: post #49

Earlier quoted context omitted.

The prefix cache is a resource shared by all users. This is basically a tragedy of the commons.

Compute is also a resource shared by all users. When your prompt isn’t cached, your input needs to go through prefill; which is compute-bound. The cache is discounted for a reason. They WANT you to use it.

They could always just extend the cache timeout beyond 5 minutes themselves. They don't do that because it is an expensive resource and there is a trade-off bewteen saving computation and reserving VRAM. Running a tool like this will force them to do one of the following:

1) Start charging for VRAM reservations.

2) Charge _other_ customers more.

3) Eat the cost themselves.

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#83

Post by SwellJoe »

skeledrew wrote: post #57
SwellJoe wrote: post #30

Earlier quoted context omitted.

While Anthropic is far behind on caching and efficiency (made more dramatically apparent by how much cheaper it is to use GPT 5.6 Sol at API rates than even Opus 4.8, much less Fable), a bunch of people forcing their way to the front of the queue at the expense of everyone else isn't going to solve that.

They definitely aren't. Over 98% of my tokens over the last 6 month have been cache reads (did an analysis just a couple days ago), which has made my Pro plan highly valuable.

I certainly don't see much in the way of cached token rates when I use the API, whereas when I use DeepSeek via API, it's mostly at the cached token rate.

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#84

Post by dannyw »

searealist wrote: post #81
dannyw wrote: post #65

Earlier quoted context omitted.

Compute is also a resource shared by all users. When your prompt isn’t cached, your input needs to go through prefill; which is compute-bound. The cache is discounted for a reason. They WANT you to use it.

They could always just extend the cache timeout beyond 5 minutes themselves. They don't do that because it is an expensive resource and there is a trade-off bewteen saving computation and reserving VRAM. Running a tool like this will force them to do one of the following: 1) Start charging for VRAM reservations. 2) Charge _other_ customers more. 3) Eat the cost themselves.

They charge you for cache writes already. There's a price assigned to it. Even if you're on subscription, it counts against your usage (very easy to observe with a big cache miss).

Anthropic (and now OpenAI too for 5.6) prompt caching is not free.

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#85

Post by toasty228 »

brookst wrote: post #75
SwellJoe wrote: post #29

This is just making it more expensive for everyone else, right? How Claude handles its sessions is none of my business. I'm going to let them do the best they can to provide good service for everyone, and if they can't/won't, I'll switch to a provider that can. Using these massive models is already pretty danged extravagant, I'm not going to demand to be at the front of the queue at all times, too.

This has nothing to do with queueing. Cached input tokens cost 10% of uncached. So if you’re model runs for 45 minutes, generates 300k output tokens and asks you a question, it costs 10x more if you wait 5.01 minutes to answer. Sure, you may be willing to pay 10x more (or get 10x less for your subscription). But the time limit is arbitrary and has nothing to do with other peoples’ workloads. So I think your point is…

> has nothing to do with other peoples’ workloads.

Because ram/memory is free and not in demande at all these days?

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#87

Post by brookst »

Jensson wrote: post #78
brookst wrote: post #75

Earlier quoted context omitted.

This has nothing to do with queueing. Cached input tokens cost 10% of uncached. So if you’re model runs for 45 minutes, generates 300k output tokens and asks you a question, it costs 10x more if you wait 5.01 minutes to answer. Sure, you may be willing to pay 10x more (or get 10x less for your subscription). But the time limit is arbitrary and has nothing to do with other peoples’ workloads. So I think your point is…

> But the time limit is arbitrary and has nothing to do with other peoples’ workloads No it has to do with others workloads, now you keep their cache for longer so others will get less. And no its not arbitrary, they run out of memory, if more people do this they will have the dial it down further or run out of capacity.

Explain to me how “every gets 5 minutes, 24/7, regardless of load” is remotely compatible with your statement?

Are you imagining this a fixed MRU where duration scales with usage? Becasue that is not at all what Anthropic documents: https://platform.claude.com/docs/en/build-with-claude/prompt...

You would not get more than 5 minutes if you were the only user in the world. You would not get less at their peak hours.

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#88

Post by brookst »

searealist wrote: post #79
brookst wrote: post #75

Earlier quoted context omitted.

This has nothing to do with queueing. Cached input tokens cost 10% of uncached. So if you’re model runs for 45 minutes, generates 300k output tokens and asks you a question, it costs 10x more if you wait 5.01 minutes to answer. Sure, you may be willing to pay 10x more (or get 10x less for your subscription). But the time limit is arbitrary and has nothing to do with other peoples’ workloads. So I think your point is…

1) To be clear, it costs you 10x more for uncached input tokens _for the next call_, which are still 5x cheaper than output tokens. 2) Now imagine Anthropic or OpenAI now charge your per minute of reserved VRAM time. It would be more fair if they did. Would you still want to run a tool like this?

Totally agree that if the economics changed, the economics would change.

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#90

Post by searealist »

dannyw wrote: post #84
searealist wrote: post #81

Earlier quoted context omitted.

They could always just extend the cache timeout beyond 5 minutes themselves. They don't do that because it is an expensive resource and there is a trade-off bewteen saving computation and reserving VRAM. Running a tool like this will force them to do one of the following: 1) Start charging for VRAM reservations. 2) Charge _other_ customers more. 3) Eat the cost themselves.

They charge you for cache writes already. There's a price assigned to it. Even if you're on subscription, it counts against your usage (very easy to observe with a big cache miss). Anthropic (and now OpenAI too for 5.6) prompt caching is not free.

Holding your cache in VRAM for 5 minutes or 3 hours have very different costs to them.

"They already charge me to park my car, why can't I leave it there for a year for the same price as 1 week?"

Post reply on HN