There are some missing y-axis labels that would be interesting to see
Cloudflare incident on August 21, 2025
41–46 of 46 posts
Re: Cloudflare incident on August 21, 2025
#42> This system will allot network resources on a per-customer basis, creating a budget that, once exceeded, will prevent a customer's traffic from degrading the service for anyone else on the platform How would this work practically? If a single client is overflowing the edge router queues you are kindof screwed already? Even if you dropped all packets from that client you would need to still process the packets to fi…
> Even if you dropped all packets from that client you would need to still process the packets to figure out what client they belong to before dropping the packets?
In modern Linux you can write BPF-XDP programs to drop traffic at the lowest level in the driver before any computation is spent on them at all. Nearly the first thing the driver does after getting new packets in the rx ring buffer is run your program on them.
Re: Cloudflare incident on August 21, 2025
#43> This system will allot network resources on a per-customer basis, creating a budget that, once exceeded, will prevent a customer's traffic from degrading the service for anyone else on the platform How would this work practically? If a single client is overflowing the edge router queues you are kindof screwed already? Even if you dropped all packets from that client you would need to still process the packets to fi…
> Even if you dropped all packets from that client you would need to still process the packets to figure out what client they belong to before dropping the packets? In modern Linux you can write BPF-XDP programs to drop traffic at the lowest level in the driver before any computation is spent on them at all. Nearly the first thing the driver does after getting new packets in the rx ring buffer is run your program on…
Say you have a BPF-XDP program which processes the packet to figure out what client the packet is coming from and selectively drops those packets. Is that really going to be faster than just forwarding the packet from the edge router to the next hop? I find it hard to believe that running such a program would actually alleviate full queues when all the edge router is doing is just forwarding to the next hop?
Re: Cloudflare incident on August 21, 2025
#44I wonder which customer triggered this…
Braze is my guess. They let customers do a lot of stuff with pushing and pulling data per user and I would guess every customer is in a sandbox. They were also impacted by the incident.
It sounds like anyone relying on Cloudflare and AWS us-east-1 were impacted. Not sure it's quite the smoking gun you're implying
Re: Cloudflare incident on August 21, 2025
#45Earlier quoted context omitted.
Braze is my guess. They let customers do a lot of stuff with pushing and pulling data per user and I would guess every customer is in a sandbox. They were also impacted by the incident.
It sounds like anyone relying on Cloudflare and AWS us-east-1 were impacted. Not sure it's quite the smoking gun you're implying
It was the company that I found that was impacted and could have caused it. They also don't play nice with others. It's even in the documentation. We looked into buying their services. They have also caused load problems in other clouds. But it was just a guess. Could be anything.
Re: Cloudflare incident on August 21, 2025
#46Earlier quoted context omitted.
> Even if you dropped all packets from that client you would need to still process the packets to figure out what client they belong to before dropping the packets? In modern Linux you can write BPF-XDP programs to drop traffic at the lowest level in the driver before any computation is spent on them at all. Nearly the first thing the driver does after getting new packets in the rx ring buffer is run your program on…
Say you have a BPF-XDP program which processes the packet to figure out what client the packet is coming from and selectively drops those packets. Is that really going to be faster than just forwarding the packet from the edge router to the next hop? I find it hard to believe that running such a program would actually alleviate full queues when all the edge router is doing is just forwarding to the next hop?
Where is the queueing happening? Maybe I misunderstood.
I assumed you meant the hosts are queueing in the kernel because their userspace consumers can't keep up. In that case, XDP can help, because it can drop things out of the rx ring buffers before the network stack and later userspace spend cpu cycles processing them.
If you meant the router is queueing because it's receiving more traffic than the sum of its downstream link bandwidth, like a raw static spam flood DDoS, I don't think the hosts can't do anything about that.