Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on June 21, 2022

blog.cloudflare.com

Post reply on HN
Print view View original

61–70 of 234 posts

Re: Cloudflare outage on June 21, 2022

#61

Post by ec109685 »

Belphemur wrote: post #13

It's interesting that in 2022 we still have network issues caused by wrong order of rules. Everybody at one time experiences the dreaded REJECT not being at the end of the rule stack but just too early. Kudos to CF for such a good explanation of what caused the issue.

I wonder what tool the engineers used to view that diff. With a side by side one, it’s a bit more obvious when lines are reordered.

Even better if the tool was syntax aware so it could highlight the different types of rules in unique colors.

Re: Cloudflare outage on June 21, 2022

#64

Post by simiones »

kurtextrem wrote: post #40

Yet another BGP caused outage. At some point we should collect all of them: - Cloudflare 2022 (this one) - Facebook 2021: https://news.ycombinator.com/item?id=28752131 - this one probably had the single biggest impact, since engineers got locked out of their systems, which made the fixing part look like a sci-fi movie - (Indirectly caused by BGP: Cloudflare 2020: https://blog.cloudflare.com/cloudflare-outage-on-july-…

Google in Japan 2017: https://www.internetsociety.org/blog/2017/08/google-leaked-p...

Re: Cloudflare outage on June 21, 2022

#66

Post by ttul »

Every outage represents an opportunity to demonstrate resilience and ingenuity. Outages are guaranteed to happen. Might as well make the most of it to reveal something cool about their infrastructure.

Re: Cloudflare outage on June 21, 2022

#67

Post by ElectronShak »

jgrahamc wrote: post #51
ElectronShak wrote: post #49

What's it like to be an engineer designing and working on these systems? Must be sooo fulfiling! #Goals; Y'all are my heores!!

https://www.cloudflare.com/careers/

Thanks, unfortunately I live in Africa, no roles yet for my location. I'll wait as I use the products :)

Re: Cloudflare outage on June 21, 2022

#68

Post by ssms27 »

kurtextrem wrote: post #40

Yet another BGP caused outage. At some point we should collect all of them: - Cloudflare 2022 (this one) - Facebook 2021: https://news.ycombinator.com/item?id=28752131 - this one probably had the single biggest impact, since engineers got locked out of their systems, which made the fixing part look like a sci-fi movie - (Indirectly caused by BGP: Cloudflare 2020: https://blog.cloudflare.com/cloudflare-outage-on-july-…

The internet runs on BGP, I would think that most internet issues would be a result of BGP then.

Re: Cloudflare outage on June 21, 2022

#69

Post by bluehatbrit »

mattferderer wrote: post #35
samwillis wrote: post #2

In a world where it can take weeks for other companies to publish a postmortem after an outage (if they ever do), I never ceases to amaze me how quickly CF manage to get something like this out. I think it's a testament to their Ops/Incident response teams and internal processes, it builds confidence in their ability to respond quickly when something does go wrong. Incredible work!

To further add to your point, the CTO is the one who shared it here & the CEO is incredibly active on forums & social media everywhere with customers. Communication has always been one of their strengths.

I do wonder what would happen should happen if either of them left the company, I feel like there's a lot of trust on HN (and other places) that's heavily attached to them as individuals and their track record of good communication.

Re: Cloudflare outage on June 21, 2022

#70

Post by simiones »

gcau wrote: post #9

Am I the only who really doesn't think this is a big deal? They had an outage, they fixed it very quickly. Life goes on. Talking about the outage as if it's reason for us to all ditch CF, then buy/ run our own hardware (which will be totally better), so hyperbolic.

It is kind of a big deal to discover just how much of the Internet and the WWW is now dependent on CloudFlare.

For their part, they handled this very well, and are to be commended (quick fix, quick explanation of failure).

But you also can't help but see that they have a dangerous amount of control over such important systems.

Post reply on HN