Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When “idle” isn't idle: how a Linux kernel optimization became a QUIC bug

blog.cloudflare.com

Post reply on HN
Print view View original

21–30 of 39 posts

Re: When “idle” isn't idle: how a Linux kernel optimization became a QUIC bug

#21

Post by otterley »

vasilvv wrote: post #15
neuralkoi wrote: post #3

I can see why they rewrote QUIC in Rust and for use in userspace, though going the in-house approach would warrant keeping an eye on the relevant kernel commits like a hawk to avoid missing bug fixes like these. These in-house implementations tend to have less eyeballs than the kernel. I found it interesting that Cloudflare is not yet using BBR as the default in quiche. CUBIC's recovery in this day and age, and espec…

> though going the in-house approach would warrant keeping an eye on the relevant kernel commits like a hawk to avoid missing bug fixes like these. These in-house implementations tend to have less eyeballs than the kernel. This is somewhat funny to read because this specific issue in CUBIC (sudden CWND jump upon existing quiescence) was originally discovered in Google's QUIC library and then later reported to the tea…

Would formal validation of these algorithms (e.g. with TLA+) help avoid such bugs?

Re: When “idle” isn't idle: how a Linux kernel optimization became a QUIC bug

#22

Post by kedihacker »

otterley wrote: post #21
vasilvv wrote: post #15

Earlier quoted context omitted.

> though going the in-house approach would warrant keeping an eye on the relevant kernel commits like a hawk to avoid missing bug fixes like these. These in-house implementations tend to have less eyeballs than the kernel. This is somewhat funny to read because this specific issue in CUBIC (sudden CWND jump upon existing quiescence) was originally discovered in Google's QUIC library and then later reported to the tea…

Would formal validation of these algorithms (e.g. with TLA+) help avoid such bugs?

I think a audited algorithm where each type is strictly defined like int32 added to that really help with what exactly should be inputted to it so it remains correct.

Re: When “idle” isn't idle: how a Linux kernel optimization became a QUIC bug

#24

Post by benmmurphy »

masklinn wrote: post #7
neuralkoi wrote: post #3

I can see why they rewrote QUIC in Rust and for use in userspace, though going the in-house approach would warrant keeping an eye on the relevant kernel commits like a hawk to avoid missing bug fixes like these. These in-house implementations tend to have less eyeballs than the kernel. I found it interesting that Cloudflare is not yet using BBR as the default in quiche. CUBIC's recovery in this day and age, and espec…

> I can see why they rewrote QUIC in Rust and for use in userspace As far as I know, while they might have either way, they did not (" re write QUICK [...] for use in userspace"): the linux kernel implementation only landed late 2025. Quiche was started ca 2018 (that's when Cloudflare started beta-deploying QUIC, the first public alpha of quiche was january 2019). I don't know that there even was an in-kernel impleme…

I think the original commenter confused taking the CUBIC implementation from the kernel and rewriting that in Rust for use in their QUIC implementation or they just jumbled their wording. It does make sense to use an existing battle tested implementation of a congestion algorithm because there are potential many real world failure modes that you might not anticipate if you try and write an implementation from scratch.

Re: When “idle” isn't idle: how a Linux kernel optimization became a QUIC bug

#25

Post by OptionOfT »

I tried to read the article, and had to go back and forth between reading about terms and etc, because I'm not _that_ familiar in the space. But previously I could understand Cloudflare's blog posts.

This one just feels... off. The buildup just doesn't feel right.

The fact that there is an Em Dash (sorry...) in the code tells me that it's at least AI assisted, which explains the vibe the article emanates.

And once I finally made it to the end I read the following:

> If you're interested in congestion control, transport protocols, or contributing to open-source networking code, check out the quiche repository. We're always looking for talented engineers who love digging into problems like these, please explore our open positions.

You don't add that to your blog-posts 5 days after laying off 20% of the company, regardless of whether they're sales people or engineers. If you want to add it, delay the post by 2 weeks.

Equally, there is only 1 role open in Engineering, and it's an intern role, posted yesterday:

https://www.cloudflare.com/careers/ (filter by Engineering).

Did they lay off their PR team as well?

Re: When “idle” isn't idle: how a Linux kernel optimization became a QUIC bug

#26

Post by rslashuser »

embedding-shape wrote: post #10
blahgeek wrote: post #2

The more precise title should be: How we copied code from Linux kernel without fully understand it and missed its follow-up fixes, now it bites us

Also, not a single takeaway about how to prevent that very preventable issue in the first place, as you allude to. I wonder what happened with the very hardcore engineering that used to happen at Cloudflare and was published? Almost every blog post today seems to expose some weirdness at Cloudflare, rather than highlighting excellence in engineering, what changes? Been slowly changing over the years, did they change…

The test is the hardcore engineering tell here. The test is dialed in on the key area, and when the graph wasn't coming out the right shape, they kept at it. Plus one from me!

Re: When “idle” isn't idle: how a Linux kernel optimization became a QUIC bug

#27

Post by neuralkoi »

benmmurphy wrote: post #24
masklinn wrote: post #7

Earlier quoted context omitted.

> I can see why they rewrote QUIC in Rust and for use in userspace As far as I know, while they might have either way, they did not (" re write QUICK [...] for use in userspace"): the linux kernel implementation only landed late 2025. Quiche was started ca 2018 (that's when Cloudflare started beta-deploying QUIC, the first public alpha of quiche was january 2019). I don't know that there even was an in-kernel impleme…

I think the original commenter confused taking the CUBIC implementation from the kernel and rewriting that in Rust for use in their QUIC implementation or they just jumbled their wording. It does make sense to use an existing battle tested implementation of a congestion algorithm because there are potential many real world failure modes that you might not anticipate if you try and write an implementation from scratch…

Yes, I meant CUBIC implementation! But I'm glad I made the mistake-I learned some interesting things from the responses above.

Re: When “idle” isn't idle: how a Linux kernel optimization became a QUIC bug

#28

Post by bonzini »

twoodfin wrote: post #19
bonzini wrote: post #5

Earlier quoted context omitted.

Yes, and it becomes unbearable after a while.

I don’t get it. Unlike a lot of the technical article slop that is posted here, this obviously had a lot of human thought and effort put into the prompt. The LLM pass (unsurprisingly) made it worse. For example: The results were conclusive: 100% pass rate, showing Reno recovered cleanly after the loss phase, and revealing that this is a CUBIC-related bug. Look, I’m reading a description of a Linux kernel network cong…

Yeah, you aren't selling anything. "Reno has a 100% pass rate for recovering cleanly after the loss phase, so the bug is almost certainly related to CUBIC" is a perfectly fine technical text.

Re: When “idle” isn't idle: how a Linux kernel optimization became a QUIC bug

#29

Post by neuralkoi »

echoangle wrote: post #8

Looking at the last plot, it seems like the backoff is roughly 1/5 of the total bandwith and it happens every 50 ms or so. Wouldn't it make sense to reduce the backoff and the growth speed if a backoff occurs repeatedly in rapid succession? We want to maximize the area under the curve (transmitted packages), right?

As per the article, CCAs aim to maximize data transfer by inferring the "available bandwidth" of the network. CUBIC relies primarily on packet loss as a congestion signal. For recovery, CUBIC's window size is a cubic function of time since the last congestion event.

After the initial packet loss triggered purposefully the first two seconds in this experiment, the only thing which could cause loss is the network queue (i.e. a simple tail drop, fq-codel, etc) which cannot process packets faster than they can arrive. At this point the link is saturated. The loss becomes a signal for CUBIC to reduce its window. This causes the oscillations you pointed out.

Unlike CUBIC, BBR [0] uses a model-based approach that estimates the available bandwidth and leaves some headroom kind of like you suggest to achieve higher throughput, and doesn't react as aggressively to loss as CUBIC.

[0] https://datatracker.ietf.org/meeting/104/materials/slides-10...

Re: When “idle” isn't idle: how a Linux kernel optimization became a QUIC bug

#30

Post by xxs »

OptionOfT wrote: post #25

I tried to read the article, and had to go back and forth between reading about terms and etc, because I'm not _that_ familiar in the space. But previously I could understand Cloudflare's blog posts. This one just feels... off. The buildup just doesn't feel right. The fact that there is an Em Dash (sorry...) in the code tells me that it's at least AI assisted, which explains the vibe the article emanates. And once I…

> it's at least AI assisted, which explains the vibe the article emanates

You're being too kind -- "The [b]epoch[b] is the reference timestamp CUBIC". Weird style to have random bold words. Its a blog post for the sake of it - no real takeaway. Well, there is the takeaway section that's a special summary of the article, instead.

Post reply on HN