KVarN: Native vLLM backend for KV-cache quantization by Huawei • Hacker News Board

Re: KVarN: Native vLLM backend for KV-cache quantization by Huawei

#11

Post by woadwarrior01 » Thu, Jun 04, 2026, 8:09 PM UTC

v3ss0n wrote: post #2

Why this is not a PR for vLLM ?

Last I heard, vLLM was backed by a company that has raised $150m in seed funding. I'm sure they've got the resources to port it.

Re: KVarN: Native vLLM backend for KV-cache quantization by Huawei

#12

Post by electroglyph » Thu, Jun 04, 2026, 9:33 PM UTC

throwa356262 wrote: post #3

Better performance than TQ and better quality than FP16? Am I reading this right??

any divergence (even if the benchmark is better) from full precision is error

Re: KVarN: Native vLLM backend for KV-cache quantization by Huawei

#13

Post by 0xjeffro » Thu, Jun 04, 2026, 9:58 PM UTC

yao yao ling xian

Re: KVarN: Native vLLM backend for KV-cache quantization by Huawei

#14

Post by electronsoup » Thu, Jun 04, 2026, 11:05 PM UTC

v3ss0n wrote: post #2

Why this is not a PR for vLLM ?

Why this is not a PR for llama.cpp

Re: KVarN: Native vLLM backend for KV-cache quantization by Huawei

#15

Post by sheepscreek » Fri, Jun 05, 2026, 12:42 AM UTC

qeternity wrote: post #8

throwa356262 wrote: post #3

Better performance than TQ and better quality than FP16? Am I reading this right??

It's not better quality: 59.3% vs 59.4% fp16 on AIME 25

0.1% is within margin of error. Depending on the performance boost, it might be worthwhile taking a minuscule quality hit.

Re: KVarN: Native vLLM backend for KV-cache quantization by Huawei

#16

Post by 7e » Fri, Jun 05, 2026, 4:32 AM UTC

electroglyph wrote: post #12

throwa356262 wrote: post #3

Better performance than TQ and better quality than FP16? Am I reading this right??

any divergence (even if the benchmark is better) from full precision is error

Just pretend that it is the next step update when training. You didn’t train your model to step=inf, I hope?

Re: KVarN: Native vLLM backend for KV-cache quantization by Huawei

#17

Post by mikeayles » Fri, Jun 05, 2026, 8:41 AM UTC

[dead]

Re: KVarN: Native vLLM backend for KV-cache quantization by Huawei

#18

Post by sspoisk » Fri, Jun 05, 2026, 11:35 AM UTC

[flagged]

Re: KVarN: Native vLLM backend for KV-cache quantization by Huawei

#19

Post by lukasc-ch » Fri, Jun 05, 2026, 3:16 PM UTC

... and it's on llama.cpp that to this guy! https://www.reddit.com/r/LocalLLaMA/comments/1txlhxu/i_imple...

Re: KVarN: Native vLLM backend for KV-cache quantization by Huawei

#20

Post by lukasc-ch » Fri, Jun 05, 2026, 3:18 PM UTC

lukasc-ch wrote: post #19

... and it's on llama.cpp that to this guy! https://www.reddit.com/r/LocalLLaMA/comments/1txlhxu/i_imple...

This is awesome! Let's give them some stars: - https://github.com/huawei-csl/KVarN (original repo, vLLM implementation) - https://github.com/Anbeeld/beellama.cpp (llama.cpp implementation + awesome evals)