Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: KVBoost – chunk-level KV cache reuse for HuggingFace, 5–48x faster TTFT

pythongiant.github.io

Post reply on HN
Print view View original

11–20 of 23 posts

Re: Show HN: KVBoost – chunk-level KV cache reuse for HuggingFace, 5–48x faster TTFT

#11

Post by sigmoid10 »

stpedgwdgfhgdd wrote: post #10

I just dont get why people choose Python and not e.g. Go for high performance problems.

Python is a very convenient skeleton for gluing together high performance modules that were written in C or cuda. Writing boilerplate code in those to adapt them to your project is much more inconvenient.

Re: Show HN: KVBoost – chunk-level KV cache reuse for HuggingFace, 5–48x faster TTFT

#13

Post by Yoric »

stpedgwdgfhgdd wrote: post #10

I just dont get why people choose Python and not e.g. Go for high performance problems.

Go is pretty good at performance, but pretty bad at expressing domain-specific logics. Python is the opposite, but once you have isolated the parts that need to be optimized, it's quite easy to rewrite them in a native language (in particular, the Rust-Python bindings are really good, although in this project, it's C++).

Re: Show HN: KVBoost – chunk-level KV cache reuse for HuggingFace, 5–48x faster TTFT

#14

Post by larme »

stpedgwdgfhgdd wrote: post #10

I just dont get why people choose Python and not e.g. Go for high performance problems.

Go is not high performance enough. Like what others said, you implement the high performance part in C++ and use python to glue them.

Re: Show HN: KVBoost – chunk-level KV cache reuse for HuggingFace, 5–48x faster TTFT

#15

Post by pythongiant »

stpedgwdgfhgdd wrote: post #10

I just dont get why people choose Python and not e.g. Go for high performance problems.

my initial choice was to use Rust for this actually (Probably should've too :P) but i went with python for an initial mvp/skeleton for a future rewrite

Re: Show HN: KVBoost – chunk-level KV cache reuse for HuggingFace, 5–48x faster TTFT

#17

Post by pythongiant »

arjie wrote: post #7
pythongiant wrote: post #3

KVBoost is a chunk-level KV cache reuse library for HuggingFace models (pip install kvboost). It supports two recompute strategies (selective boundary and CacheBlend), int8/int4 KV quantization for 2–4x RAM reduction, disk-backed cold storage, and 11 architectures including Llama, Qwen, Gemma, Mistral, and Phi. On Qwen2.5-3B we measured 47.9x TTFT speedup on an 8-turn conversation, 21x on code context reuse, 100–743x…

I don't get it. The output of the CacheBlend paper is in LMCache. Did you compare against vLLM with LMCache? This is confusing.

[flagged]

Re: Show HN: KVBoost – chunk-level KV cache reuse for HuggingFace, 5–48x faster TTFT

#18

Post by pythongiant »

x0ruman wrote: post #8

The functionality is impressive, but the website needs some work

Thanks! this is a weekend project that i am working on in the side just to learn more about ml engineering and custom cuda kernels. didnt think much about the website

Re: Show HN: KVBoost – chunk-level KV cache reuse for HuggingFace, 5–48x faster TTFT

#19

Post by pythongiant »

pythongiant wrote: post #15
stpedgwdgfhgdd wrote: post #10

I just dont get why people choose Python and not e.g. Go for high performance problems.

my initial choice was to use Rust for this actually (Probably should've too :P) but i went with python for an initial mvp/skeleton for a future rewrite

[flagged]
Post reply on HN