Live data from Hacker News

Cloudflare outage caused by bad software deploy

blog.cloudflare.com

31–40 of 137 posts

Re: Cloudflare outage caused by bad software deploy

#31
post #27

What sort of regular expression pitfalls can cause this sort of CPU utilization? I know they're possible but I am curious about specific examples of something similar to what caused Cloudflare's issue here.

Some regex languages allow backtracking, and backtracking is usually the thing that causes regexes to blow up in resource cost: https://www.regular-expressions.info/catastrophic.html

Re: Cloudflare outage caused by bad software deploy

#32
post #27

What sort of regular expression pitfalls can cause this sort of CPU utilization? I know they're possible but I am curious about specific examples of something similar to what caused Cloudflare's issue here.

DoS with regex is a thing: https://www.owasp.org/index.php/Regular_expression_Denial_of...

StackOverflow had a similar case a while back: https://stackstatus.net/post/147710624694/outage-postmortem-...

Re: Cloudflare outage caused by bad software deploy

#33
post #18

Earlier quoted context omitted.

As an engineer, I get pissed whenever I see 100% uptime, or eleven-nines, nine-nines, or other impossible targets. Like, how am I supposed to design a system with numbers like that?

Deploy once, never update, and deploy a missile defense to prevent backhoes from digging up fiber?

Ah yes, missile defenses, like the MIM-104 Patriot: https://en.wikipedia.org/wiki/MIM-104_Patriot#Failure_at_Dha...

Re: Cloudflare outage caused by bad software deploy

#34
post #6

Kinda wonder at this point what findings exist on their Availability SOC 2, assuming they've gotten one. The repeated outages plus the constant malicious advertising by scammy ad providers through cloudflare are slowly turning me off to the service as a potential enterprise customer. Unfortunate too since plenty of superlatively qualified people build great things there (hat tip to Nick Sullivan), but it seems like t…

As an engineer, I get pissed whenever I see 100% uptime, or eleven-nines, nine-nines, or other impossible targets. Like, how am I supposed to design a system with numbers like that?

You don't. Like others have said SLA's are a sales tool, not an objective for the engineering team to achieve.

Re: Cloudflare outage caused by bad software deploy

#35
post #18

Earlier quoted context omitted.

As an engineer, I get pissed whenever I see 100% uptime, or eleven-nines, nine-nines, or other impossible targets. Like, how am I supposed to design a system with numbers like that?

Deploy once, never update, and deploy a missile defense to prevent backhoes from digging up fiber?

You honestly think a missile defense system will work. Backhoes are much more creative than that. You will need defense in depth, roaming patrols, as well as air and satellite based monitoring assets.

Re: Cloudflare outage caused by bad software deploy

#36
post #28
post #17

At 1402 UTC we understood what was happening and decided to issue a ‘global kill’ on the WAF Managed Rulesets, which instantly dropped CPU back to normal and restored traffic. That occurred at 1409 UTC. So for about 50 minutes, those who relied on the WAF were open to attack?

Isn't open to DDoS better than can't be reached?

WAF wouldn’t just prevent DDoS I assume. I’m pretty sure there are WAF rulesets that attempt to block attacks such as XSS or even remote code execution vulnerabilities.

Re: Cloudflare outage caused by bad software deploy

#37
post #6

Kinda wonder at this point what findings exist on their Availability SOC 2, assuming they've gotten one. The repeated outages plus the constant malicious advertising by scammy ad providers through cloudflare are slowly turning me off to the service as a potential enterprise customer. Unfortunate too since plenty of superlatively qualified people build great things there (hat tip to Nick Sullivan), but it seems like t…

As an engineer, I get pissed whenever I see 100% uptime, or eleven-nines, nine-nines, or other impossible targets. Like, how am I supposed to design a system with numbers like that?

I agree, advertised SLAs are garbage.

Agreements to uphold past performance are much better.

Re: Cloudflare outage caused by bad software deploy

#38
For the size and importance of Cloudflare some insights to a couple questions would be nice:

1. Why are WAF rules not progressively deployed since there's already a system to do so?

2. Maybe there should also be a testing environment that receives a mirror of production traffic before deployments reach real users?

(I understand the WAF change was not set to take action, but a separate environment would be less likely to affect production)

Re: Cloudflare outage caused by bad software deploy

#39
post #37

Earlier quoted context omitted.

As an engineer, I get pissed whenever I see 100% uptime, or eleven-nines, nine-nines, or other impossible targets. Like, how am I supposed to design a system with numbers like that?

I agree, advertised SLAs are garbage. Agreements to uphold past performance are much better.

SLAs are all about getting compensation when they are broken. It isn't about actual uptime.

Re: Cloudflare outage caused by bad software deploy

#40
post #2

I run a service placing bids the last few seconds on eBay. Every time this happens I lose measurable business (we place thousands of bids per day). While it doesn’t affect scheduled bids, they can’t place bids and are likely to move to a competitor. These recent outages have been costly. Does anyone know a more reliable provider?

Haha, people pay you to bid-snipe on eBay for them? It just baffles me that manual/third-party bid-sniping is still a thing. eBay has had automatic bidding for more than twenty years. You'll pay the same whether you put in the winning bid a week in advance or 5 seconds. But people see that "you lost this auction" notice and they're irrationally convinced that it would have gone differently if they'd bid at the last m…

Sniping is the only rational way to big on eBay. You bid a single time, with the absolute maximum you would go for, and put it in the last 10 seconds. It prevents both you and your opponents from raising the price irrationally. Automatic bidding just encourages prices to go up, based on emotion. Sniping in the last few seconds removes the emotional component.
Post reply on HN