What sort of regular expression pitfalls can cause this sort of CPU utilization? I know they're possible but I am curious about specific examples of something similar to what caused Cloudflare's issue here.
Cloudflare outage caused by bad software deploy
31–40 of 137 posts
Re: Cloudflare outage caused by bad software deploy
#32What sort of regular expression pitfalls can cause this sort of CPU utilization? I know they're possible but I am curious about specific examples of something similar to what caused Cloudflare's issue here.
StackOverflow had a similar case a while back: https://stackstatus.net/post/147710624694/outage-postmortem-...
Re: Cloudflare outage caused by bad software deploy
#33Earlier quoted context omitted.
As an engineer, I get pissed whenever I see 100% uptime, or eleven-nines, nine-nines, or other impossible targets. Like, how am I supposed to design a system with numbers like that?
Deploy once, never update, and deploy a missile defense to prevent backhoes from digging up fiber?
Re: Cloudflare outage caused by bad software deploy
#34Kinda wonder at this point what findings exist on their Availability SOC 2, assuming they've gotten one. The repeated outages plus the constant malicious advertising by scammy ad providers through cloudflare are slowly turning me off to the service as a potential enterprise customer. Unfortunate too since plenty of superlatively qualified people build great things there (hat tip to Nick Sullivan), but it seems like t…
As an engineer, I get pissed whenever I see 100% uptime, or eleven-nines, nine-nines, or other impossible targets. Like, how am I supposed to design a system with numbers like that?
Re: Cloudflare outage caused by bad software deploy
#35Earlier quoted context omitted.
As an engineer, I get pissed whenever I see 100% uptime, or eleven-nines, nine-nines, or other impossible targets. Like, how am I supposed to design a system with numbers like that?
Deploy once, never update, and deploy a missile defense to prevent backhoes from digging up fiber?
Re: Cloudflare outage caused by bad software deploy
#36At 1402 UTC we understood what was happening and decided to issue a ‘global kill’ on the WAF Managed Rulesets, which instantly dropped CPU back to normal and restored traffic. That occurred at 1409 UTC. So for about 50 minutes, those who relied on the WAF were open to attack?
Isn't open to DDoS better than can't be reached?
Re: Cloudflare outage caused by bad software deploy
#37Kinda wonder at this point what findings exist on their Availability SOC 2, assuming they've gotten one. The repeated outages plus the constant malicious advertising by scammy ad providers through cloudflare are slowly turning me off to the service as a potential enterprise customer. Unfortunate too since plenty of superlatively qualified people build great things there (hat tip to Nick Sullivan), but it seems like t…
As an engineer, I get pissed whenever I see 100% uptime, or eleven-nines, nine-nines, or other impossible targets. Like, how am I supposed to design a system with numbers like that?
Agreements to uphold past performance are much better.
Re: Cloudflare outage caused by bad software deploy
#381. Why are WAF rules not progressively deployed since there's already a system to do so?
2. Maybe there should also be a testing environment that receives a mirror of production traffic before deployments reach real users?
(I understand the WAF change was not set to take action, but a separate environment would be less likely to affect production)
Re: Cloudflare outage caused by bad software deploy
#39Earlier quoted context omitted.
As an engineer, I get pissed whenever I see 100% uptime, or eleven-nines, nine-nines, or other impossible targets. Like, how am I supposed to design a system with numbers like that?
I agree, advertised SLAs are garbage. Agreements to uphold past performance are much better.
Re: Cloudflare outage caused by bad software deploy
#40I run a service placing bids the last few seconds on eBay. Every time this happens I lose measurable business (we place thousands of bids per day). While it doesn’t affect scheduled bids, they can’t place bids and are likely to move to a competitor. These recent outages have been costly. Does anyone know a more reliable provider?
Haha, people pay you to bid-snipe on eBay for them? It just baffles me that manual/third-party bid-sniping is still a thing. eBay has had automatic bidding for more than twenty years. You'll pay the same whether you put in the winning bid a week in advance or 5 seconds. But people see that "you lost this auction" notice and they're irrationally convinced that it would have gone differently if they'd bid at the last m…