Here's the repository incase anyone wants to have a look at the code. leave a star if you find it interesting :P https://github.com/pythongiant/KVBoost
Show HN: KVBoost – chunk-level KV cache reuse for HuggingFace, 5–48x faster TTFT
21–23 of 23 posts
Re: Show HN: KVBoost – chunk-level KV cache reuse for HuggingFace, 5–48x faster TTFT
#22Drop in replacement for what exactly? Can I use it with llama.cpp and Vulkan?
Or vLLM and ROCm?
Re: Show HN: KVBoost – chunk-level KV cache reuse for HuggingFace, 5–48x faster TTFT
#23Drop in replacement for what exactly? Can I use it with llama.cpp and Vulkan? Or vLLM and ROCm?
KVBoost is a drop-in replacement for AutoModelForCausalLM. Same API surface (KVBoost.from_pretrained(...), engine.generate(...)), but with cross-request KV reuse, FlashAttention-2, AWQ layer streaming, and speculative decoding bolted on.