Can anybody suggest a Systems Engineer-centric forum/site? (Not Windows 'help I can't print' level, more DataCenter grade.)
HN does have some great content/replies that touch on these topics, but I'd like something more.
Cloudflare outage caused by bad software deploy
51–60 of 137 posts
Re: Cloudflare outage caused by bad software deploy
#52Can anybody suggest a Systems Engineer-centric forum/site? (Not Windows 'help I can't print' level, more DataCenter grade.) HN does have some great content/replies that touch on these topics, but I'd like something more.
Perhaps these QA sites are interesting?
https://superuser.com/ https://serverfault.com/
But yes, the content like this on HN is fascinating and I would also like more.
Re: Cloudflare outage caused by bad software deploy
#53I'm always beyond impressed with how responsive and transparent CF is with incidence and post mortem communication. Given who the CEO and COO are, I suppose this shouldn't be surprising, never the less as a customer it builds a great deal of trust. Kudos.
Yes, they do really well on this - open, transparent, posting information quickly as soon as they were fairly sure what the problem was. I always really enjoy their writing, both incident reports and writeups of new features. The only thing I think they could have managed better was their status page, which claimed they were up (every service was green) when they were not.
I think they were blindsided that this was even possible. So they hadn't thought to add a panel to the status page for this one.
I bet they will now.
Re: Cloudflare outage caused by bad software deploy
#54Earlier quoted context omitted.
Yes, they do really well on this - open, transparent, posting information quickly as soon as they were fairly sure what the problem was. I always really enjoy their writing, both incident reports and writeups of new features. The only thing I think they could have managed better was their status page, which claimed they were up (every service was green) when they were not.
I think they were blindsided that this was even possible. So they hadn't thought to add a panel to the status page for this one. I bet they will now.
I think their status page doesn't update service status automatically in response to downtime, which it really should.
Re: Cloudflare outage caused by bad software deploy
#55If a single regex can take down the Internet for a half hour, that's definitely not good -- for a class of errors that can be easily prevented, tested, etc.
The timing is unfortunate too, after calling out Verizon for lack of due process and negligence.
I'm sure they have an undo or rollback for deployments but probably worth investing into further.
They also need to resolve the catch-22 where people could not login and disable CloudFlare proxy ("orange cloud") since cloudflare.com itself was down.
Re: Cloudflare outage caused by bad software deploy
#56Kinda wonder at this point what findings exist on their Availability SOC 2, assuming they've gotten one. The repeated outages plus the constant malicious advertising by scammy ad providers through cloudflare are slowly turning me off to the service as a potential enterprise customer. Unfortunate too since plenty of superlatively qualified people build great things there (hat tip to Nick Sullivan), but it seems like t…
You can read our SOC3 (public facing SOC2) if you're curious about your availability question: https://www.cloudflare.com/compliance/ There's a lot of good info in there
Ah, hi Evan.
There's a lot of good info here, but there are many more questions raised in my mind based on what I'm reading in the SOC3 than perhaps what you might've expected. I can ideally run through them if I catch you again at DEF CON this year. I'm also willing to sign your standard MNDA to review your SOC 2, but we can take that thread offline.
Re: Cloudflare outage caused by bad software deploy
#57Earlier quoted context omitted.
Deploy once, never update, and deploy a missile defense to prevent backhoes from digging up fiber?
You honestly think a missile defense system will work. Backhoes are much more creative than that. You will need defense in depth, roaming patrols, as well as air and satellite based monitoring assets.
And then the fibre will get cut by a building crew working on a guard tower.
Re: Cloudflare outage caused by bad software deploy
#58It is interesting NGINX returned 502 nearly instantly under very heavy CPU load. I would have expected requests to just hang or timeout.
I would imagine it's tiered. The Nginx servers at the front returning the 502 probably aren't the boxes running the code
Re: Cloudflare outage caused by bad software deploy
#59Earlier quoted context omitted.
Isn't open to DDoS better than can't be reached?
WAF wouldn’t just prevent DDoS I assume. I’m pretty sure there are WAF rulesets that attempt to block attacks such as XSS or even remote code execution vulnerabilities.
I can confirm that there are WAF rules that block things like basic SQL injection. A client uses Akamai and if it detects certain strings in a request, like "", it'll block the request before it ever gets to the application. The bad part is that some developers get complacent in their development and rely on the WAF to do their security for them.
Re: Cloudflare outage caused by bad software deploy
#60Earlier quoted context omitted.
As an engineer, I get pissed whenever I see 100% uptime, or eleven-nines, nine-nines, or other impossible targets. Like, how am I supposed to design a system with numbers like that?
SLAs aren't for engineers, they are for financial people to make agreements on payments for downtime.
Good SLAs are also for engineers.