Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fooling around with encrypted reasoning blobs

blog.cryptographyengineering.com

Post reply on HN
Print view View original

11–20 of 36 posts

Re: Fooling around with encrypted reasoning blobs

#11

Post by tn1 »

geocar wrote: post #5
glitchc wrote: post #4

Very interesting. The state management is the really insightful find here. I always wondered how these large AI companies managed access for millions of simultaneous users without having to allocate a dedicated LLM instance for each user. Pushing the complete state down to the user after every call makes perfect sense. The LLM itself stays memoryless and ready to respond to an arbitrary prompt. Very nice.

N.B. This is exactly how seaside, vba, and even arc[1] do server-side state generally : by encrypting the blob-representing-state and sending to the client to be sent back on future requests (where it will be decrypted and rehydrated). It's an old trick that everyone designing protocols should know, since there are lots of applications beyond AI companies. [1]: As in, pg's lisp: https://arclanguage.github.io/ref/srv.…

And don't forget the venerable .NET Forms with its kilobytes of __VIEWSTATE

Re: Fooling around with encrypted reasoning blobs

#12

Post by londons_explore »

glitchc wrote: post #4

Very interesting. The state management is the really insightful find here. I always wondered how these large AI companies managed access for millions of simultaneous users without having to allocate a dedicated LLM instance for each user. Pushing the complete state down to the user after every call makes perfect sense. The LLM itself stays memoryless and ready to respond to an arbitrary prompt. Very nice.

Except the providers also cache the parsing of the prompt (the KV cache), and that has substantial cost savings (easily an 80% saving on typical coding use cases).

That caching is done server side and not passed to the client. Which in turn means they still need state management on the server side, although it perhaps doesn't need the same level of global replication and availability.

Re: Fooling around with encrypted reasoning blobs

#13

Post by londons_explore »

b65e8bee43c2ed0 wrote: post #9
glitchc wrote: post #4

Very interesting. The state management is the really insightful find here. I always wondered how these large AI companies managed access for millions of simultaneous users without having to allocate a dedicated LLM instance for each user. Pushing the complete state down to the user after every call makes perfect sense. The LLM itself stays memoryless and ready to respond to an arbitrary prompt. Very nice.

the exchange rate between text and its representation in memory is brutal. here's a bit from a recent article: >An 82 GB footprint in DDR3 on a 2016 Xeon. About 25 GB of weights and 56 GB of KV cache at the full 262K context. The KV cache is larger than the model. 262k tokens is not much at all. with ~5 characters per token, that's only 1.3 MB of plaintext.

The providers must have a more efficient approach. Most cache every request for 12+ hours, and they certainly can't spare 100GB of ram per request for 12 hours.

Re: Fooling around with encrypted reasoning blobs

#15

Post by choppaface »

londons_explore wrote: post #13
b65e8bee43c2ed0 wrote: post #9

Earlier quoted context omitted.

the exchange rate between text and its representation in memory is brutal. here's a bit from a recent article: >An 82 GB footprint in DDR3 on a 2016 Xeon. About 25 GB of weights and 56 GB of KV cache at the full 262K context. The KV cache is larger than the model. 262k tokens is not much at all. with ~5 characters per token, that's only 1.3 MB of plaintext.

The providers must have a more efficient approach. Most cache every request for 12+ hours, and they certainly can't spare 100GB of ram per request for 12 hours.

or maybe they don’t actually cache (fully) but lie and just don’t charge the user right now. at least half the users, who are probably also using the most similar tokens / prompts, wouldn’t really know the difference in latency (or care)

Re: Fooling around with encrypted reasoning blobs

#16

Post by dist-epoch »

londons_explore wrote: post #13
b65e8bee43c2ed0 wrote: post #9

Earlier quoted context omitted.

the exchange rate between text and its representation in memory is brutal. here's a bit from a recent article: >An 82 GB footprint in DDR3 on a 2016 Xeon. About 25 GB of weights and 56 GB of KV cache at the full 262K context. The KV cache is larger than the model. 262k tokens is not much at all. with ~5 characters per token, that's only 1.3 MB of plaintext.

The providers must have a more efficient approach. Most cache every request for 12+ hours, and they certainly can't spare 100GB of ram per request for 12 hours.

This is one reason why price of SSDs also doubled, not just of RAM.

> LMCache extends the KV Cache from the NVIDIA GPU's fast HBM (Tier 1) to larger, more cost-effective tiers like CPU RAM and local SSDs.

https://cloud.google.com/blog/topics/developers-practitioner...

Re: Fooling around with encrypted reasoning blobs

#17

Post by londons_explore »

choppaface wrote: post #15
londons_explore wrote: post #13

Earlier quoted context omitted.

The providers must have a more efficient approach. Most cache every request for 12+ hours, and they certainly can't spare 100GB of ram per request for 12 hours.

or maybe they don’t actually cache (fully) but lie and just don’t charge the user right now. at least half the users, who are probably also using the most similar tokens / prompts, wouldn’t really know the difference in latency (or care)

If it actually cost that much RAM, they would almost certainly add extra things to the API to manage cache lifetime. Ie. A 'please cache this for X minutes' flag, or a setting for a single re-use cache (the most common use case)

Re: Fooling around with encrypted reasoning blobs

#18

Post by cyanydeez »

glitchc wrote: post #4

Very interesting. The state management is the really insightful find here. I always wondered how these large AI companies managed access for millions of simultaneous users without having to allocate a dedicated LLM instance for each user. Pushing the complete state down to the user after every call makes perfect sense. The LLM itself stays memoryless and ready to respond to an arbitrary prompt. Very nice.

they still have to cache the tokens. its not completely stateless.

in theory, every conversation is replayed from the beginning. in practice, its only going to be economical to heavily cache the stable portions of the text as tokens inside the GPU

one of the reasons the Cloud providers have such heavy prompts is because that can be cached for all users, but its essentially poisonong the state before you even start. alot of the variability appears related to changing the context rather than the model.

models are expensive and the bean counters know fine tuning and context changes are cheaper. id guess the IPOs are essentially the SOTA EOL.

Re: Fooling around with encrypted reasoning blobs

#19

Post by cyanydeez »

londons_explore wrote: post #17
choppaface wrote: post #15

Earlier quoted context omitted.

or maybe they don’t actually cache (fully) but lie and just don’t charge the user right now. at least half the users, who are probably also using the most similar tokens / prompts, wouldn’t really know the difference in latency (or care)

If it actually cost that much RAM, they would almost certainly add extra things to the API to manage cache lifetime. Ie. A 'please cache this for X minutes' flag, or a setting for a single re-use cache (the most common use case)

https://platform.claude.com/docs/en/build-with-claude/prompt...

suggests the can cache outside the gpu.

Re: Fooling around with encrypted reasoning blobs

#20

Post by cyanydeez »

londons_explore wrote: post #12
glitchc wrote: post #4

Very interesting. The state management is the really insightful find here. I always wondered how these large AI companies managed access for millions of simultaneous users without having to allocate a dedicated LLM instance for each user. Pushing the complete state down to the user after every call makes perfect sense. The LLM itself stays memoryless and ready to respond to an arbitrary prompt. Very nice.

Except the providers also cache the parsing of the prompt (the KV cache), and that has substantial cost savings (easily an 80% saving on typical coding use cases). That caching is done server side and not passed to the client. Which in turn means they still need state management on the server side, although it perhaps doesn't need the same level of global replication and availability.

from the march changes, it looked like they increased cache eviction rates on the VRAM at claude causing everyone to start burning tokens as they had to regen token state.
Post reply on HN