Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage caused by bad software deploy

blog.cloudflare.com

Post reply on HN
Print view View original

71–80 of 137 posts

Re: Cloudflare outage caused by bad software deploy

#71

Post by threezero »

txcwpalpha wrote: post #67

Peculiar that eastdakota (Cloudflare's CEO) doesn't seem to be tweeting at the Cloudflare team responsible for this, telling them they should be ashamed and are guilty of malpractice. When it was Verizon that took down the internet he felt it was appropriate to do that to the Verizon teams, after all. edit: right after posting this comment, he did tweet the following: https://twitter.com/eastdakota/status/11461968360…

Cloudflare was responsive and reasonable. Verizon was unreachable and deflected responsibility when they finally made a statement. And public shaming does often motivate companies to be more responsive to their customers.

Re: Cloudflare outage caused by bad software deploy

#72

Post by txcwpalpha »

threezero wrote: post #71
txcwpalpha wrote: post #67

Peculiar that eastdakota (Cloudflare's CEO) doesn't seem to be tweeting at the Cloudflare team responsible for this, telling them they should be ashamed and are guilty of malpractice. When it was Verizon that took down the internet he felt it was appropriate to do that to the Verizon teams, after all. edit: right after posting this comment, he did tweet the following: https://twitter.com/eastdakota/status/11461968360…

Cloudflare was responsive and reasonable. Verizon was unreachable and deflected responsibility when they finally made a statement. And public shaming does often motivate companies to be more responsive to their customers.

AFAIK Cloudflare isn't in any way a "customer" of Verizon. Verizon doesn't owe Cloudflare any kind of response or devotion of resources. Verizon owes it's actual customers a resolution to their problem, which they gave.

I'm not saying Verizon is perfect nor absolved of fault, but Cloudflare was/is not owed any kind of explanation or assistance by VZ, and it's absurd of CF to still be whining about that fact (as they are doing in some other tweets today). If CF wants some kind of SLA with VZ, they should engage them in a business relationship, not try to publicly shame them.

Re: Cloudflare outage caused by bad software deploy

#73

Post by _wmd »

aleem wrote: post #55

If a single regex can take down the Internet for a half hour, that's definitely not good -- for a class of errors that can be easily prevented, tested, etc. The timing is unfortunate too, after calling out Verizon for lack of due process and negligence. I'm sure they have an undo or rollback for deployments but probably worth investing into further. They also need to resolve the catch-22 where people could not login…

You'd think after leaking private data for literally months less than 3 years ago (and only noticing because Google had to point it out to them) that they'd, y'know, have at least some kind of QA environment fed with sample traffic by now. Really hard to believe they're still getting caught testing in prod

Re: Cloudflare outage caused by bad software deploy

#75

Post by tyingq »

sequoia wrote: post #27

What sort of regular expression pitfalls can cause this sort of CPU utilization? I know they're possible but I am curious about specific examples of something similar to what caused Cloudflare's issue here.

There's also other similar things in this space, like hash table collisions that can eat CPU.

Re: Cloudflare outage caused by bad software deploy

#77

Post by txcwpalpha »

quickthrower2 wrote: post #62

Probably for the kind of work they are doing avoid regex? Or at least the very complicated modern regex (simple autonoma that you can compile in advance might be ok)

If you're trying to do pattern matching, is there actually a widely used alternative to regex? The more I can avoid using regex for mission-critical things, the happier I will be, but I'm really not aware of anything better for this type of application.

Re: Cloudflare outage caused by bad software deploy

#78

Post by rainyMammoth »

_wmd wrote: post #73
aleem wrote: post #55

If a single regex can take down the Internet for a half hour, that's definitely not good -- for a class of errors that can be easily prevented, tested, etc. The timing is unfortunate too, after calling out Verizon for lack of due process and negligence. I'm sure they have an undo or rollback for deployments but probably worth investing into further. They also need to resolve the catch-22 where people could not login…

You'd think after leaking private data for literally months less than 3 years ago (and only noticing because Google had to point it out to them) that they'd, y'know, have at least some kind of QA environment fed with sample traffic by now. Really hard to believe they're still getting caught testing in prod

For working in that field, the arrogance of CloudFlare is still unbelievable to me.

After their huge Cloudbleed issue with the addition of this one, they continue to call out everyone through their blog posts. And everyone seems fine with it because they are a hype company.

Re: Cloudflare outage caused by bad software deploy

#79

Post by almost_usual »

grey-area wrote: post #50

I really want to know the regexp and corresponding input(s) which killed the internet now :) Was it just aaaaaaaaaaaah? https://swtch.com/~rsc/regexp/regexp1.html

I'm assuming it's something pretty embarrassing if it's not in the post mortem.
Post reply on HN