It’s gonna turn out it was one guy on one machine calling “pnpm install” on a fast machine with a 100gbps uplink.
Can we stop with the 2015 jokes already?
In 2015 it would have been "npm install"
(Thanks Rauch.)
Cloudflare incident on August 21, 2025
31–40 of 46 posts
Re: Cloudflare incident on August 21, 2025
#32> This system will allot network resources on a per-customer basis, creating a budget that, once exceeded, will prevent a customer's traffic from degrading the service for anyone else on the platform How would this work practically? If a single client is overflowing the edge router queues you are kindof screwed already? Even if you dropped all packets from that client you would need to still process the packets to fi…
It's load shedding, but it's weighted towards people abusing their quota usually over some rolling weighted average. The benefit is that they are dropped immediately at the edge rather than holding sockets open or using compute/resources. It usually takes 30s-1m to kick in.
Re: Cloudflare incident on August 21, 2025
#33Only real long term mitigation is to move to another aws region; us-east-1 seems to suffer from all kinds of scaling challenges.
There's nothing to suggest the link between Cloudflare and any other AWS region has more capacity or that there aren't more disruptive Cloudflare customers using those regions.
yeah but us-east-1 is cursed
Re: Cloudflare incident on August 21, 2025
#34I wonder which customer triggered this…
Also curious if it was legit, misconfigured, or attack traffic.
Re: Cloudflare incident on August 21, 2025
#35I'm having trouble understanding the second diagram in the article. I can make sense of a directed graph, but this one has thin horizontal lines with arrows leaving them in both directions. These lines look like dividers, not nodes, so I'm not sure how to interpret it.
I think the intention is to show the divide between Amazon's and Cloudflare's responsibility, over the piece of fibre linking their network devices together. It would have been clearer to continue the lines and just put a dotted divider between them I feel.
Re: Cloudflare incident on August 21, 2025
#36I wonder which customer triggered this…
Braze is my guess. They let customers do a lot of stuff with pushing and pulling data per user and I would guess every customer is in a sandbox. They were also impacted by the incident.
Re: Cloudflare incident on August 21, 2025
#37Wild that one tenant’s cache-hit traffic could tip over Cloudflare’s interconnect capacity
You'd be surprised how low the capacity of a lot of internet links is. 10Gbps is common on smaller networks - let me rephrase that, a small to medium ISP might only have 10Gbps to each of most of their peering partners. Normally, traffic is distributed, going to different places, coming from different places, and each link is partially utilized. But unusual patterns can fill up one specific link. 10Gbps is old techno…
Hot take. 40 Gbps is not a real rate; it's just four 10 Gbps in a trenchcoat stacked on top of one another!
Re: Cloudflare incident on August 21, 2025
#38Come on, some CF insider please create a throwaway account and tell us which client was :)
Re: Cloudflare incident on August 21, 2025
#39Earlier quoted context omitted.
You'd be surprised how low the capacity of a lot of internet links is. 10Gbps is common on smaller networks - let me rephrase that, a small to medium ISP might only have 10Gbps to each of most of their peering partners. Normally, traffic is distributed, going to different places, coming from different places, and each link is partially utilized. But unusual patterns can fill up one specific link. 10Gbps is old techno…
Hot take. 40 Gbps is not a real rate; it's just four 10 Gbps in a trenchcoat stacked on top of one another!
Thats true for several other speeds too. 100GE first generation was 10x10GbE, second generation was 4x25GbE. 200GE first version was 25GbE based and so on.
Re: Cloudflare incident on August 21, 2025
#40> This system will allot network resources on a per-customer basis, creating a budget that, once exceeded, will prevent a customer's traffic from degrading the service for anyone else on the platform How would this work practically? If a single client is overflowing the edge router queues you are kindof screwed already? Even if you dropped all packets from that client you would need to still process the packets to fi…
In this specific case, it wasn't requests from the client that caused overload. It was the responses to those requests. So Cloudflare can avoid sending responses, and prevent the problem.
You're right that this doesn't solve all cases, but it would have prevented this case.