Viewing profile — pythongiant

Tenosra is all about making AI more efficient and resourceful you can read more abour it at: https://www.tenosra.com/proxima

hey everyone, i decided to make a vLLM plugin that implements the Star-KV paper. the results are quiet promising with a decode kernel thats faster than FA2 in higher batch sizes. w…

Hey guys, i'm especially interested in feedback on the kernel design and integration onto mlx lm. Its on pypi as well as a simple pip install mlx-turboquant :P

Hi HN, I built mlx-turboquant, an implementation of Google's TurboQuant KV-cache compression algorithm for Apple's MLX framework. The repository includes quality benchmarks, memory…

KVBoost is a drop-in replacement for AutoModelForCausalLM. Same API surface (KVBoost.from_pretrained(...), engine.generate(...)), but with cross-request KV reuse, FlashAttention-2,…

Here's the repository incase anyone wants to have a look at the code. leave a star if you find it interesting :P https://github.com/pythongiant/KVBoost

[flagged]

[flagged]

Thanks! this is a weekend project that i am working on in the side just to learn more about ml engineering and custom cuda kernels. didnt think much about the website

[flagged]

Makes sense, fixing that. thanks!

my initial choice was to use Rust for this actually (Probably should've too :P) but i went with python for an initial mvp/skeleton for a future rewrite

KVBoost is a chunk-level KV cache reuse library for HuggingFace models (pip install kvboost). It supports two recompute strategies (selective boundary and CacheBlend), int8/int4 KV…

[dead]

just published a beginner-friendly, hands-on guide to GPU programming with CUDA aimed at folks who are comfortable with basic programming but new to parallel computing. Instead of …

If you’ve ever been curious about how GPT actually works under the hood, I built a small project you might find interesting. I implemented a GPT-style transformer from scratch in a…