Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on June 21, 2022

blog.cloudflare.com

Post reply on HN
Print view View original

81–90 of 234 posts

Re: Cloudflare outage on June 21, 2022

#81

Post by bombcar »

cowsandmilk wrote: post #77
kevin_nisbet wrote: post #41

Earlier quoted context omitted.

I agree, I think the transparency builds trust and I encourage it where I can. The counter thought I had when reading this case though, is it almost feels too fast. What I mean by that is I hope there isn't an incentive to wrap up the internal investigation quickly and write the blog and send it, and go we're done. Doing incident response (both outage and security), the tactical fixes for a specific problem are usual…

I have to agree. The environment that leads to a fast blog post may also lead to this quote from the post: > This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically. They are running as fast as they can and this extended the incident. There is a “slow is smooth, smooth is fast” lesson in here. I’d rather have a team that tak…

It can be easy to arm-chair it afterwards, but unless things can be done in parallel (and systems should be designed so this can be done, things like "we're not sure what's wrong, we're bringing up a new cluster on the last known good version even as we try to repair this one") you have to make a choice, and sometimes it won't be optimal.

Re: Cloudflare outage on June 21, 2022

#83

Post by johnklos »

samwillis wrote: post #43
johnklos wrote: post #30

Earlier quoted context omitted.

I think you've already drunk the Flavor Aid. What do you have when you have all DNS going through them, via DoH, and all web requests going through them, if not recentralization? Sure, they want us to think they give us the freedom to host our web sites anywhere because they're "protected" by them, but that "protection" means we've agreed to recentralize. It's pretty dismissive to describe something as a meme just be…

I understand that for their WAF, DDOS and threat detection products they need to have a very large amount of traffic going through them. They have been very aggressive with their free service to achieve that, to the benefit of all their customers (including the free ones). Some could see that as a push to at centralisation, I don't. What I don't understand, or believe, is that they want to be the sole (as in centrali…

They want to be a monopoly. They want everyone to depend on them. They may not want recentralization in general, but they definitely want as much of the Internet to depend on them as possible.

Re: Cloudflare outage on June 21, 2022

#84

Post by rrss »

hunter2_ wrote: post #58
viraptor wrote: post #19

Earlier quoted context omitted.

I feel like others lose opportunities by not doing the same. By publishing early and publishing the details they: keep the company in the news with positive stuff (free ad), get an internal documentation of the incident (ignoring the customer oriented "we're sorry" part), effectively get a free recruitment post (you're reading this because you're in tech and we do cool stuff, wink), release some internal architecture…

> I wonder how much those posts are calculated and how much organic/culture related. Don't companies have a fiduciary duty to calculate things; the reason for doing something actually cannot just be that it's a nice thing to do? Not down to the word, but at least the general decision to be this way?

No.

https://scholarship.law.cornell.edu/cgi/viewcontent.cgi?http...

Re: Cloudflare outage on June 21, 2022

#85

Post by dangrossman »

thejosh wrote: post #5

07:42: The last of the reverts has been completed. This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically. Ouch

I think I experienced first-hand the moment those network engineers were reverting their own reverts, breaking the web again. For example, DoorDash.com had come back online, then went back to serving only HTTP 500 errors from Cloudflare, then came back online again. I raised it in the HN discussion and @jgrahamc responded minutes later.

https://news.ycombinator.com/item?id=31821290

Re: Cloudflare outage on June 21, 2022

#87

Post by ThaDood »

thesuitonym wrote: post #82

Where does one even start with learning BGP? It always seemed super interesting to me, but not really something that could be dealt with on a small scale, lab type basis. Or am I wrong there?

DN42 https://dn42.eu/Home> gets mentioned a lot. Its basically a big dynamic VPN that you can do BGP stuff with. Pretty cool but I could never get my node working properly.

Re: Cloudflare outage on June 21, 2022

#88

Post by jamal-kumar »

thesuitonym wrote: post #82

Where does one even start with learning BGP? It always seemed super interesting to me, but not really something that could be dealt with on a small scale, lab type basis. Or am I wrong there?

Nah Cisco has labs you can download and learn for their networking certifications, which are kinda the standard.

Networking talent is kind of hard to find and if you learn that your chances of employment get pretty high.

Re: Cloudflare outage on June 21, 2022

#89

Post by throwaway_uke »

i'm gonna go with the less popular view here that overly detailed post mortems do little in the grand scheme of things other than satisfy tech p0rn for a tiny, highly technical audience. does wonders for hiring indeed.

sure, transparency is better than "something went wrong, we take this very seriously, sorry." (although the non technical crowd couldn't care less)

only people who dont do anything make no mistakes, but doing such highly impactful changes so quickly (inside one day!) for where 50% of traffic happens seems a huge red flag to me, no matter the procedure and safety valves.

Re: Cloudflare outage on June 21, 2022

#90

Post by kylegalbraith »

As others have said, this is a clear and concise write up of the incident. That is underlined even more when you take into account how quickly they published this. I have seen some companies take weeks or even months to publish an analysis that is half as good as this.

Not trying to take the light away from the outage, the outage was bad. But the relative quickness to recovery is pretty impressive, in my opinion. Sounds like they could have recovered even quicker if not for a bit of toe stepping that happened.

Post reply on HN