Have a look at my semantic caching project!
It's built to easily integrate in existing LLM workflows, you can use it as a proxy where the cache forwards missed requests without modification to a specified upstream, automatically updating it's cache with the response.
You can also use it as a cache-aside cache with a provided python library.
It works by computing embedding vectors of input queries, and matches them to seen query + response pairs using a vector store.
Everything is in-memory, so it should be blazing fast :)
Show HN: Semcache – I built a semantic cache in Rust
github.com