Live data from Hacker News

Cloudflare outage on June 21, 2022

blog.cloudflare.com

101–110 of 234 posts

Re: Cloudflare outage on June 21, 2022

#101

Where does one even start with learning BGP? It always seemed super interesting to me, but not really something that could be dealt with on a small scale, lab type basis. Or am I wrong there?

https://github.com/Exa-Networks/exabgp

They've got some Docker examples in the README.

Re: Cloudflare outage on June 21, 2022

#102

Where does one even start with learning BGP? It always seemed super interesting to me, but not really something that could be dealt with on a small scale, lab type basis. Or am I wrong there?

You can learn BGP with mininet: https://mininet.org/

You can simulate arbitrarily large networks and internetworks with this, provided you have the hardware to run a large enough number of virtual appliances, but they are pretty lightweight.

Re: Cloudflare outage on June 21, 2022

#103

Yet another BGP caused outage. At some point we should collect all of them: - Cloudflare 2022 (this one) - Facebook 2021: https://news.ycombinator.com/item?id=28752131 - this one probably had the single biggest impact, since engineers got locked out of their systems, which made the fixing part look like a sci-fi movie - (Indirectly caused by BGP: Cloudflare 2020: https://blog.cloudflare.com/cloudflare-outage-on-july-…

> since engineers got locked out of their systems

Sounds like the same happened here:

"Due to this withdrawal, Cloudflare engineers experienced added difficulty in reaching the affected locations to revert the problematic change. We have backup procedures for handling such an event and used them to take control of the affected locations."

But Cloudflare had sufficient backup connectivity to fix it. I'm curious how Cloudflare does that today-- the solution long ago was always a modem on an auxiliary port.

Re: Cloudflare outage on June 21, 2022

#104
CF is the only company I have ever seen that can have an outage and get pages of praise for it. I don't have any (current) use for CloudFlare's products but I would love to see the culture that makes them praiseworthy spread to other companies.

Re: Cloudflare outage on June 21, 2022

#106
post #104

CF is the only company I have ever seen that can have an outage and get pages of praise for it. I don't have any (current) use for CloudFlare's products but I would love to see the culture that makes them praiseworthy spread to other companies.

I think a lot of companies don't realize the whole "Acknowledging our problems in public" thing CF got going for it is a positive. Lots of companies don't want to publish public post-mortems as they think it'll make them look weak rather than showing that they care about transparency in the face of failures/downtimes.

Re: Cloudflare outage on June 21, 2022

#107
post #104

CF is the only company I have ever seen that can have an outage and get pages of praise for it. I don't have any (current) use for CloudFlare's products but I would love to see the culture that makes them praiseworthy spread to other companies.

[deleted]

Re: Cloudflare outage on June 21, 2022

#108
post #60

Earlier quoted context omitted.

I feel like others lose opportunities by not doing the same. By publishing early and publishing the details they: keep the company in the news with positive stuff (free ad), get an internal documentation of the incident (ignoring the customer oriented "we're sorry" part), effectively get a free recruitment post (you're reading this because you're in tech and we do cool stuff, wink), release some internal architecture…

I agree that this is a free ad/recruitment. However, it’s easy to see how more conservative businesses see this as a risk. They are highlighting their deficiencies, letting their big important clients know that human error can bring their network down. Additionally, these post-mittens work for Cloudflare because they have a great reputation and good uptime. If this were happening daily or weekly, it would be a warnin…

OTOH, I think most actual engineers would know that everywhere has deficiencies and can be brought down by human error, and I'd personally rather use a product where the people running it admit this rather than just claim that their genius engineers made it 100% foolproof and nothing could ever possibly go wrong

Re: Cloudflare outage on June 21, 2022

#109

One of our sites uses Cloudflare and serves 400k pageviews per month and generates around $650/day in ad and affiliate revenue. If the site is not up the business is not making any money. Looking at the hourly chart in Google Analytics (compared to the previous day) there isn't even a blip during this outage. So for all the advantages we get from Cloudflare (caching, WAF, security [our WP admin is secured with Cloudf…

Could you possibly, kindly, mention which tools you use to track/buy/calculate conversions/revenue? Many thanks (Or DM the puppet email in my profile)

Not OP, but my team really, really enjoys using a combination of Segment.io for event tracking and piping that data into Amplitude for data viz, funnel metrics, A/B tests, conversion, etc.

Re: Cloudflare outage on June 21, 2022

#110
post #104

CF is the only company I have ever seen that can have an outage and get pages of praise for it. I don't have any (current) use for CloudFlare's products but I would love to see the culture that makes them praiseworthy spread to other companies.

Nerds in the executive office (CEO & CTO, etc). People just like us.
Post reply on HN