Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Claude-thermos keeps your Claude session warm for you

github.com

Post reply on HN
Print view View original

61–70 of 91 posts

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#61

Post by dannyw »

SwellJoe wrote: post #29

This is just making it more expensive for everyone else, right? How Claude handles its sessions is none of my business. I'm going to let them do the best they can to provide good service for everyone, and if they can't/won't, I'll switch to a provider that can. Using these massive models is already pretty danged extravagant, I'm not going to demand to be at the front of the queue at all times, too.

You PAY for cache writes. Holding the prefill in VRAM for a certain period of time is part of what you're paying for.

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#62

Post by leemoore »

broodbucket wrote: post #19
razodactyl wrote: post #11

Don't we pay for cache input though?

You can send and receive 1 byte and refresh the cache. Keeping the cache alive isn't free, but it's close.

if you are refreshing a cache of 120k tokens, you have to input the same 120k tokens. you can't bounce the cache with 1 byte. A proper cache hit requires your entire context that is cached. if you change 1 byte in it, everything after that byte is a cache write and no longer a cache hit

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#63

Post by dannyw »

SwellJoe wrote: post #35
s0ck_r4w wrote: post #32

Earlier quoted context omitted.

How do a few extra requests with the same prefix make it more expensive for everyone else?

Keeping a conversation with a very large model active requires hundreds of GB of memory. If my conversation can never be swapped out, like when I go to lunch or take an afternoon walk, that's several hundred GB not available for other users. If everybody does that, Anthropic needs even more infrastructure than the quarter trillion dollars in infra they're already using.

But you pay for cache writes, it's not like you're abusing anything, you're simply choosing to call a billed API for the explicitly intended purpose of writing/refreshing cache.

If you keep this running for hours without doing anything, it will drain your limits and API. The use case of keeping the main thread cache warm while subagents work is very genuine and legitimate.

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#64

Post by broodbucket »

leemoore wrote: post #62
broodbucket wrote: post #19

Earlier quoted context omitted.

You can send and receive 1 byte and refresh the cache. Keeping the cache alive isn't free, but it's close.

if you are refreshing a cache of 120k tokens, you have to input the same 120k tokens. you can't bounce the cache with 1 byte. A proper cache hit requires your entire context that is cached. if you change 1 byte in it, everything after that byte is a cache write and no longer a cache hit

Yes, you're paying the cache read, not the cache write, which is much more expensive. You can do the math for how many 5min refreshes you can be afk for until it starts costing instead of saving

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#65

Post by dannyw »

kakugawa wrote: post #49
s0ck_r4w wrote: post #32

Earlier quoted context omitted.

How do a few extra requests with the same prefix make it more expensive for everyone else?

The prefix cache is a resource shared by all users. This is basically a tragedy of the commons.

Compute is also a resource shared by all users. When your prompt isn’t cached, your input needs to go through prefill; which is compute-bound.

The cache is discounted for a reason. They WANT you to use it.

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#66

Post by dannyw »

fearmerchant wrote: post #52
kakugawa wrote: post #49

Earlier quoted context omitted.

The prefix cache is a resource shared by all users. This is basically a tragedy of the commons.

If true, I suspect Anthropic will begin suspending accounts that use it.

I highly doubt that.

Keeping your cache warm is a good thing, caching saves compute and electricity.

Cached input is cheap for a reason, it is in everyone’s mutual interests to maximise cache hit rates.

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#67

Post by dannyw »

idonotknowwhy wrote: post #48
devnonymous wrote: post #41

Earlier quoted context omitted.

Yes https://github.com/izeigerman/claude-thermos/blob/main/src%2...

CC actually prompts Claude about this by default in the ~20k system prompt and instructs it to avoid 300s timeout and to be mindful of the 300s cache expiration.

CC also will also block sleeps longer than eg 300s; the harness handholds the model quite a bit.

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#68

Post by gruez »

dannyw wrote: post #61
SwellJoe wrote: post #29

This is just making it more expensive for everyone else, right? How Claude handles its sessions is none of my business. I'm going to let them do the best they can to provide good service for everyone, and if they can't/won't, I'll switch to a provider that can. Using these massive models is already pretty danged extravagant, I'm not going to demand to be at the front of the queue at all times, too.

You PAY for cache writes. Holding the prefill in VRAM for a certain period of time is part of what you're paying for.

Right, but also using the cache refreshes/extends it, which means you can pay for a write once, then have it extended for very cheap (1 token).

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#69

Post by OccamsMirror »

dannyw wrote: post #66
fearmerchant wrote: post #52

Earlier quoted context omitted.

If true, I suspect Anthropic will begin suspending accounts that use it.

I highly doubt that. Keeping your cache warm is a good thing, caching saves compute and electricity. Cached input is cheap for a reason, it is in everyone’s mutual interests to maximise cache hit rates.

If that was true then Anthropic would have built this functionality in themselves.

Re: Show HN: Claude-thermos keeps your Claude session warm for you

#70

Post by icase »

SwellJoe wrote: post #29

This is just making it more expensive for everyone else, right? How Claude handles its sessions is none of my business. I'm going to let them do the best they can to provide good service for everyone, and if they can't/won't, I'll switch to a provider that can. Using these massive models is already pretty danged extravagant, I'm not going to demand to be at the front of the queue at all times, too.

hey guys dario’s here!
Post reply on HN