Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Smaller, faster, safer: running Kimi and GLM at scale

blog.cloudflare.com

Post reply on HN
Print view View original

51–60 of 74 posts

Re: Smaller, faster, safer: running Kimi and GLM at scale

#51

Post by Tepix »

anonova wrote: post #15
scrlk wrote: post #4

Nice to see a provider being transparent about KV cache quantisation. I've been suspecting that some providers do this silently whilst heavily promoting their unquantised weights, even though KV quantisation can degrade quality more than weight quantisation. However, I wish their testing were more detailed. Firstly, some model families are more sensitive to KV quantisation than others (only Kimi K2.6 was tested). Sec…

vLLM's study also concluded that "FP8 can deliver meaningful latency and capacity gains with small or negligible accuracy loss". Their benchmarks include LiveCodeBench 6. https://vllm-project.github.io/2026/04/22/fp8-kvcache.html

vLLM tested with Llama-3.3-70B-Instruct, Qwen3-30B-A3B-Instruct-2507, Qwen3-30B-A3B-Thinking-2507, and Qwen3.5-27B.

I'm wondering if it needs to be tested with every other model or not.

Re: Smaller, faster, safer: running Kimi and GLM at scale

#52

Post by Tepix »

bigbaguette wrote: post #47
vietvu wrote: post #46

Thanks for the transpiration but this is too shallow when talking about LLM serving.

Could you suggest good resources about LLM serving? Blogs, articles...

The latest Latent.Space episode https://www.latent.space/p/inference-eng

Re: Smaller, faster, safer: running Kimi and GLM at scale

#54

Post by [deleted] »

buildbot wrote: post #40
simonw wrote: post #26

Earlier quoted context omitted.

I've seen this called MLOps.

MLOps though typically wouldn't been going quantization? It requires some careful testing of accuracy and performance even today. Applied ML Research Engineer or something maybe, not that I have ever seen that title. Maybe just catchall ML Engineer...

[deleted]

Re: Smaller, faster, safer: running Kimi and GLM at scale

#55

Post by Tepix »

joshuamcginnis wrote: post #25

> If squeezing the best open models onto GPUs and serving them to millions of developers sounds like your kind of problem, come work with us. What is the typical job title and/or skillset for this?

Latent space is calling the whole subject "inference engineering" in their episode https://www.latent.space/p/inference-eng

Re: Smaller, faster, safer: running Kimi and GLM at scale

#57

Post by stingraycharles »

amluto wrote: post #16
scrlk wrote: post #4

Nice to see a provider being transparent about KV cache quantisation. I've been suspecting that some providers do this silently whilst heavily promoting their unquantised weights, even though KV quantisation can degrade quality more than weight quantisation. However, I wish their testing were more detailed. Firstly, some model families are more sensitive to KV quantisation than others (only Kimi K2.6 was tested). Sec…

They made an extremely strong claim: > None of this would matter if it changed the model's answers If they want to assert that the answers don’t change, then perhaps they should calculate the statistical distance between the token probability outputs or something to that effect. I doubt the results would indicate that the answers don’t change by any reasonable interpretation. Maybe the results are still good enough.

Isn’t that already in detail by the research of these quantization techniques?

Re: Smaller, faster, safer: running Kimi and GLM at scale

#58

Post by stingraycharles »

HDBaseT wrote: post #42

I think Cloudflare not providing ZDR on their inference is the biggest public indicator that Cloudlare glows. We let all traffic get MITM'd, now we're letting our AI conversation get tracked. Cloudflare reeks like a US Honeypot.

This sounds like FUD unless backed by evidence.

Cloudflare’s inference absolutely does support ZDR, as long as you use unified billing (ie not using BYOK).

Re: Smaller, faster, safer: running Kimi and GLM at scale

#60

Post by scrlk »

Tepix wrote: post #51
anonova wrote: post #15

Earlier quoted context omitted.

vLLM's study also concluded that "FP8 can deliver meaningful latency and capacity gains with small or negligible accuracy loss". Their benchmarks include LiveCodeBench 6. https://vllm-project.github.io/2026/04/22/fp8-kvcache.html

vLLM tested with Llama-3.3-70B-Instruct, Qwen3-30B-A3B-Instruct-2507, Qwen3-30B-A3B-Thinking-2507, and Qwen3.5-27B. I'm wondering if it needs to be tested with every other model or not.

IMO, yes. For example, Qwen 3.x is insensitive to weight and KV cache quantisation, whereas Gemma 4 is more sensitive: https://localbench.substack.com/p/kv-cache-quantization-benc...
Post reply on HN