> View pricing in the Cloudflare dashboard ↗
Why… I wanted to see if it’s worth it to use cloudflare’s endpoint but I can’t even see the pricing
Smaller, faster, safer: running Kimi and GLM at scale
11–20 of 74 posts
Re: Smaller, faster, safer: running Kimi and GLM at scale
#12I was interested in reading this until my slop detector went off at the paragraph starting with “It's worth being precise about where the benefit comes from, because it isn't raw speed.” I love AI, but I really hate reading it.
I have had to stop commenting this because it would end up on 50% of the posts here. I really wish we could flag prose as ai-generated on here and just filter it out.
Don’t stop commenting about it, if there is something we (the readers) can do is ensure it is seen as uncool to post slop content
Re: Smaller, faster, safer: running Kimi and GLM at scale
#13I was interested in reading this until my slop detector went off at the paragraph starting with “It's worth being precise about where the benefit comes from, because it isn't raw speed.” I love AI, but I really hate reading it.
Cloudflare blogs are not meant to be human-read, AFAIK. They're raw material meant to be fed into an agent to be filtered down. I rarely read the contents because they are usually word-expanded to a greater degree than an article from The Atlantic .
That’s disappointing, in the past cloudflare had some of the best engineering blog articles
Re: Smaller, faster, safer: running Kimi and GLM at scale
#14Earlier quoted context omitted.
I have had to stop commenting this because it would end up on 50% of the posts here. I really wish we could flag prose as ai-generated on here and just filter it out.
LinkedIn (of all places!) announced a button for flagging this recently: https://www.linkedin.com/posts/hsrinivasan1_ai-slop-is-a-top... How well it would work on this site, I'm not sure.
If there is an action on AI slop on LI, it will end up with almost no posts at all
Re: Smaller, faster, safer: running Kimi and GLM at scale
#15Nice to see a provider being transparent about KV cache quantisation. I've been suspecting that some providers do this silently whilst heavily promoting their unquantised weights, even though KV quantisation can degrade quality more than weight quantisation. However, I wish their testing were more detailed. Firstly, some model families are more sensitive to KV quantisation than others (only Kimi K2.6 was tested). Sec…
vLLM's study also concluded that "FP8 can deliver meaningful latency and capacity gains with small or negligible accuracy loss". Their benchmarks include LiveCodeBench 6.
Re: Smaller, faster, safer: running Kimi and GLM at scale
#16Nice to see a provider being transparent about KV cache quantisation. I've been suspecting that some providers do this silently whilst heavily promoting their unquantised weights, even though KV quantisation can degrade quality more than weight quantisation. However, I wish their testing were more detailed. Firstly, some model families are more sensitive to KV quantisation than others (only Kimi K2.6 was tested). Sec…
They made an extremely strong claim:
> None of this would matter if it changed the model's answers
If they want to assert that the answers don’t change, then perhaps they should calculate the statistical distance between the token probability outputs or something to that effect. I doubt the results would indicate that the answers don’t change by any reasonable interpretation.
Maybe the results are still good enough.
Re: Smaller, faster, safer: running Kimi and GLM at scale
#17Earlier quoted context omitted.
Cloudflare blogs are not meant to be human-read, AFAIK. They're raw material meant to be fed into an agent to be filtered down. I rarely read the contents because they are usually word-expanded to a greater degree than an article from The Atlantic .
That’s disappointing, in the past cloudflare had some of the best engineering blog articles
I don't disagree, but at some point in the last year they ended up severely word-expanded. So in a revealed sense, they are no longer meant for human consumption except for those who don't significantly value their own time. There is very little information in the post that an agent can't pull for you:
* they use quantized models
* they quantize KV cache
* they have a cache tagging mechanism to prevent cache misuse (neat)
The agent can extract numbers without filler prose as well.
Re: Smaller, faster, safer: running Kimi and GLM at scale
#18I was interested in reading this until my slop detector went off at the paragraph starting with “It's worth being precise about where the benefit comes from, because it isn't raw speed.” I love AI, but I really hate reading it.
I'm getting somewhat conflicted. There is a Twitter account I follow that has great takes and lots of deeply personal posts... BUT he obviously uses an LLM in his writing pipeline somewhere. Too many AIisms scattered about to ignore.
I'm not thrilled with it, but he is obviously using it to improve his writing overall- to communicate some great ideas that are personal and germane. I've decided that being too inflexible serves no one. If it is true slop, I'll not revisit the writer in the future- if they are using AI to polish writing that at its core is a unique voice, I'll accept it and learn to live with it...
Re: Smaller, faster, safer: running Kimi and GLM at scale
#19I was interested in reading this until my slop detector went off at the paragraph starting with “It's worth being precise about where the benefit comes from, because it isn't raw speed.” I love AI, but I really hate reading it.
Came here to comment this.
Re: Smaller, faster, safer: running Kimi and GLM at scale
#20So they quantize models, only tell about it in the blog post (instead of a warning on the model page), and even in the blog post pretend there's no difference by benchmarking on small context tasks many of which are saturated. Coding agents will probably be severely negatively affected by KV quantization.
I'd say serving quantized models without saying so on the "store" page is fraud.