Cloudflare incident on August 21, 2025

You'd be surprised how low the capacity of a lot of internet links is. 10Gbps is common on smaller networks - let me rephrase that, a small to medium ISP might only have 10Gbps to each of most of their peering partners. Normally, traffic is distributed, going to different places, coming from different places, and each link is partially utilized. But unusual patterns can fill up one specific link.

10Gbps is old technology now and any real ISP can probably afford 40 or 100 - for hundreds of dollars per link. But they're going to deploy that on their most utilized links first, and only if their peering partner can also afford it and exchanges enough traffic to justify it. So the smallest connections are typically going to be 10. (Lower than 10 is too small to justify a point-to-point peering at all).

If you have 10Gbps fiber at home, you could congest one of these links all by yourself.

Now this is Cloudflare talking to aws-east-1, so they should have shitloads of capacity there, probably at least 8x100 or more. But considering that AWS is the kind of environment where you can spin up 800 servers for a few hours to perform a massively parallel task, it's not surprising that someone did eventually create 800Gbps of traffic to the same place, or however much they have. Actually it's surprising it doesn't happen more often. Perhaps that's because AWS charges an arm and a leg for data transfer - 800Gbps is $5-$9 per second.

Re: Cloudflare incident on August 21, 2025

#8

Post by yaboi3 » Fri, Aug 22, 2025, 3:53 PM UTC

Anyone want to tell Cloudflare that BGP advertisements at AWS are automated and their congested network directly cause BGP withdrawals as the automated system detected congestion and decreased traffic to remediate it?

Re: Cloudflare incident on August 21, 2025

#9

Post by erulabs » Sun, Aug 24, 2025, 6:34 PM UTC

It’s gonna turn out it was one guy on one machine calling “pnpm install” on a fast machine with a 100gbps uplink.

Re: Cloudflare incident on August 21, 2025

#10

Post by grumple » Sun, Aug 24, 2025, 6:43 PM UTC

yaboi3 wrote: post #8

Anyone want to tell Cloudflare that BGP advertisements at AWS are automated and their congested network directly cause BGP withdrawals as the automated system detected congestion and decreased traffic to remediate it?

It wouldn't surprise me if the BGP routes in the DCI PNI were manually configured, since this is probably one of the most direct and important connections. I would be surprised if Cloudflare didn't have firsthand knowledge of what happened with AWS during this incident.

I think the withdrawal approach by AWS would normally work, as this action should desaturate the connections. Just really unfortunate that this caused routing through a link that was at half capacity.