Can I Buy Your KV Cache?
arxiv.org
Can I Buy Your KV Cache?
1–10 of 30 posts
Re: Can I Buy Your KV Cache?
#2Lambda computing for prompts?
Re: Can I Buy Your KV Cache?
#3A truly global singleton
Re: Can I Buy Your KV Cache?
#4[dead]
Re: Can I Buy Your KV Cache?
#5The KV cache is order dependent and dependent on the context of tokens which exist before the KV cache.
There are some transformation approaches to re-use the kv cache across inferences, but none are in wide use due to accuracy concerns following the transformation.
Re: Can I Buy Your KV Cache?
#6The KV cache is order dependent and dependent on the context of tokens which exist before the KV cache. There are some transformation approaches to re-use the kv cache across inferences, but none are in wide use due to accuracy concerns following the transformation.
Just curious, do you have links to read more about transformations or other techniques for KV cache reuse?
Re: Can I Buy Your KV Cache?
#7The KV cache is order dependent and dependent on the context of tokens which exist before the KV cache. There are some transformation approaches to re-use the kv cache across inferences, but none are in wide use due to accuracy concerns following the transformation.
Just curious, do you have links to read more about transformations or other techniques for KV cache reuse?
All major model providers offer prefix caching, which is this.
Re: Can I Buy Your KV Cache?
#8The KV cache is order dependent and dependent on the context of tokens which exist before the KV cache. There are some transformation approaches to re-use the kv cache across inferences, but none are in wide use due to accuracy concerns following the transformation.
The paper has a section on "Reusing precomputed KV across queries" which talks about how other papers have tried to address this problem, but yeah, this paper adds nothing on its own besides a catchy title.
Re: Can I Buy Your KV Cache?
#9Does anyone have a good recommendation for explaining or as a primer on KV cache?
Re: Can I Buy Your KV Cache?
#10> Then the part that matters: where the KV lives
When your abstract was clearly generated by an LLM and not curated to at least make it sound human, it does not make me want to read your paper.