KVarN: Native vLLM backend for KV-cache quantization by Huawei
1–10 of 21 posts
Re: KVarN: Native vLLM backend for KV-cache quantization by Huawei
#2Why this is not a PR for vLLM ?
Re: KVarN: Native vLLM backend for KV-cache quantization by Huawei
#3Better performance than TQ and better quality than FP16?
Am I reading this right??
Re: KVarN: Native vLLM backend for KV-cache quantization by Huawei
#4Why this is not a PR for vLLM ?
It's the output of a research paper; the authors are not trying to build up vLLM, and they probably have no incentive to do so. You can submit a PR, though! It's easier now while the divergence is low, so don't wait. Since there are six authors, I bet you could get help with the inevitable review chores if you just take the step of creating the PR.
edit: It might not be clear that it is based on vLLM 0.22, which is the current version: https://github.com/huawei-csl/KVarN/commit/d6290e99098d7426d.... All you have to do is create a diff off it; it's fairly straightforward.
Re: KVarN: Native vLLM backend for KV-cache quantization by Huawei
#5Why this is not a PR for vLLM ?
It's the output of a research paper; the authors are not trying to build up vLLM, and they probably have no incentive to do so. You can submit a PR, though! It's easier now while the divergence is low, so don't wait. Since there are six authors, I bet you could get help with the inevitable review chores if you just take the step of creating the PR. edit: It might not be clear that it is based on vLLM 0.22, which is t…
And with the help of AI, pointing at AI at this paper and saying "making a vLLM PR from this paper" tends to work surprisingly well, even if you need to nudge it a little bit along the way.
Re: KVarN: Native vLLM backend for KV-cache quantization by Huawei
#6Better performance than TQ and better quality than FP16? Am I reading this right??
[dead]
Re: KVarN: Native vLLM backend for KV-cache quantization by Huawei
#7Better performance than TQ and better quality than FP16? Am I reading this right??
Faster than Fp16, not better quality i guess
Re: KVarN: Native vLLM backend for KV-cache quantization by Huawei
#8Better performance than TQ and better quality than FP16? Am I reading this right??
It's not better quality: 59.3% vs 59.4% fp16 on AIME 25
Re: KVarN: Native vLLM backend for KV-cache quantization by Huawei
#9[dead]
Re: KVarN: Native vLLM backend for KV-cache quantization by Huawei
#10Why this is not a PR for vLLM ?
it should be easy to do btw