Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare incident on August 21, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

11–20 of 46 posts

Re: Cloudflare incident on August 21, 2025

#11

Post by aianus »

immibis wrote: post #7
iqfareez wrote: post #2

Wild that one tenant’s cache-hit traffic could tip over Cloudflare’s interconnect capacity

You'd be surprised how low the capacity of a lot of internet links is. 10Gbps is common on smaller networks - let me rephrase that, a small to medium ISP might only have 10Gbps to each of most of their peering partners. Normally, traffic is distributed, going to different places, coming from different places, and each link is partially utilized. But unusual patterns can fill up one specific link. 10Gbps is old techno…

Downloading cached data from Cloudflare to AWS is free to the person doing the downloading if they use Internet gateway

Re: Cloudflare incident on August 21, 2025

#12

Post by __turbobrew__ »

> This system will allot network resources on a per-customer basis, creating a budget that, once exceeded, will prevent a customer's traffic from degrading the service for anyone else on the platform

How would this work practically? If a single client is overflowing the edge router queues you are kindof screwed already? Even if you dropped all packets from that client you would need to still process the packets to figure out what client they belong to before dropping the packets?

I guess you could somehow do some shuffle sharding where a single client belongs to a few IP prefixes and when that client misbehaves you withdraw those prefixes using BGP to essentially black hole the network routes for that client. If the shuffle sharding is done right only the problem client will have issues as other clients on the same prefixes will be sharded to other prefixes.

Re: Cloudflare incident on August 21, 2025

#13

Post by jeffbee »

__turbobrew__ wrote: post #12

> This system will allot network resources on a per-customer basis, creating a budget that, once exceeded, will prevent a customer's traffic from degrading the service for anyone else on the platform How would this work practically? If a single client is overflowing the edge router queues you are kindof screwed already? Even if you dropped all packets from that client you would need to still process the packets to fi…

Perhaps they drop the client's flows on the host side.

Re: Cloudflare incident on August 21, 2025

#14

Post by __turbobrew__ »

jeffbee wrote: post #13
__turbobrew__ wrote: post #12

> This system will allot network resources on a per-customer basis, creating a budget that, once exceeded, will prevent a customer's traffic from degrading the service for anyone else on the platform How would this work practically? If a single client is overflowing the edge router queues you are kindof screwed already? Even if you dropped all packets from that client you would need to still process the packets to fi…

Perhaps they drop the client's flows on the host side.

I don’t understand? The issue is that a client/customer outside of cloudflares control DOSed one of their network links. Cloudflare has no control on the client side to implement rate limiting?

Re: Cloudflare incident on August 21, 2025

#15

Post by themafia »

iqfareez wrote: post #2

Wild that one tenant’s cache-hit traffic could tip over Cloudflare’s interconnect capacity

That's what started the incident.

It was prolonged by the fact that Cloudflare didn't react correctly to withdrawn BGP routes to a major peer, that the secondary routes had reduced capacity due to unaddressed problems, and basic nuisance rate limiting had to be done manually.

It seems like they just build huge peering pipes and basically just hope for the best. They've maybe gotten so used to this working that they'll let degraded "secondary" links persist for much longer than they should. It's the typical "Swiss Cheese" style of failure.

Re: Cloudflare incident on August 21, 2025

#17

Post by everfrustrated »

__turbobrew__ wrote: post #12

> This system will allot network resources on a per-customer basis, creating a budget that, once exceeded, will prevent a customer's traffic from degrading the service for anyone else on the platform How would this work practically? If a single client is overflowing the edge router queues you are kindof screwed already? Even if you dropped all packets from that client you would need to still process the packets to fi…

I think you're overthinking this. Just having a per (cloudflare) customer rate limit would go a long long way.

Re: Cloudflare incident on August 21, 2025

#18

Post by fusl »

__turbobrew__ wrote: post #14
jeffbee wrote: post #13

Earlier quoted context omitted.

Perhaps they drop the client's flows on the host side.

I don’t understand? The issue is that a client/customer outside of cloudflares control DOSed one of their network links. Cloudflare has no control on the client side to implement rate limiting?

I think you misunderstand the flow of traffic here. The data flow, initiated by requests coming from AWS us-east-1, was Cloudflare towards AWS, not the other way around. Cloudflare can easily control where and how their egress traffic gets to the destination (as long as there are multiple paths towards the target) as well as rate limit that traffic to sane levels.

Re: Cloudflare incident on August 21, 2025

#19

Post by __turbobrew__ »

fusl wrote: post #18
__turbobrew__ wrote: post #14

Earlier quoted context omitted.

I don’t understand? The issue is that a client/customer outside of cloudflares control DOSed one of their network links. Cloudflare has no control on the client side to implement rate limiting?

I think you misunderstand the flow of traffic here. The data flow, initiated by requests coming from AWS us-east-1, was Cloudflare towards AWS, not the other way around. Cloudflare can easily control where and how their egress traffic gets to the destination (as long as there are multiple paths towards the target) as well as rate limit that traffic to sane levels.

Ah I see now. Yes in that case they could just reply with 429 codes or just not reply at all.

Re: Cloudflare incident on August 21, 2025

#20

Post by chatmasta »

cluckindan wrote: post #16
erulabs wrote: post #9

It’s gonna turn out it was one guy on one machine calling “pnpm install” on a fast machine with a 100gbps uplink.

Can we stop with the 2015 jokes already?

I’ve actually had an npm install that failed on my ISP but succeeded with Cloudflare VPN and the OP comment was more or less the explanation.
Post reply on HN