Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on June 21, 2022

blog.cloudflare.com

Post reply on HN
Print view View original

91–100 of 234 posts

Re: Cloudflare outage on June 21, 2022

#91

Post by algo_trader »

weird-eye-issue wrote: post #75

One of our sites uses Cloudflare and serves 400k pageviews per month and generates around $650/day in ad and affiliate revenue. If the site is not up the business is not making any money. Looking at the hourly chart in Google Analytics (compared to the previous day) there isn't even a blip during this outage. So for all the advantages we get from Cloudflare (caching, WAF, security [our WP admin is secured with Cloudf…

Could you possibly, kindly, mention which tools you use to track/buy/calculate conversions/revenue?

Many thanks

(Or DM the puppet email in my profile)

Re: Cloudflare outage on June 21, 2022

#92

Post by jgrahamc »

cowsandmilk wrote: post #77
kevin_nisbet wrote: post #41

Earlier quoted context omitted.

I agree, I think the transparency builds trust and I encourage it where I can. The counter thought I had when reading this case though, is it almost feels too fast. What I mean by that is I hope there isn't an incentive to wrap up the internal investigation quickly and write the blog and send it, and go we're done. Doing incident response (both outage and security), the tactical fixes for a specific problem are usual…

I have to agree. The environment that leads to a fast blog post may also lead to this quote from the post: > This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically. They are running as fast as they can and this extended the incident. There is a “slow is smooth, smooth is fast” lesson in here. I’d rather have a team that tak…

There's normal operating procedure and sign offs and automation etc. etc. and then there's "we've lost contact with these data centers and normal procedures don't work we need to break glass and use the secondary channels". In that situation you are in an emergency without normal visibility.

Re: Cloudflare outage on June 21, 2022

#93

Post by unityByFreedom »

bluehatbrit wrote: post #69
mattferderer wrote: post #35

Earlier quoted context omitted.

To further add to your point, the CTO is the one who shared it here & the CEO is incredibly active on forums & social media everywhere with customers. Communication has always been one of their strengths.

I do wonder what would happen should happen if either of them left the company, I feel like there's a lot of trust on HN (and other places) that's heavily attached to them as individuals and their track record of good communication.

Good communicators generally foster that environment. And their customers appreciate it, so there is an external expectation now too. Everything ends some day, but I think this will be regarded as a valuable attribute for awhile.

Re: Cloudflare outage on June 21, 2022

#94

Post by jgrahamc »

unityByFreedom wrote: post #93
bluehatbrit wrote: post #69

Earlier quoted context omitted.

I do wonder what would happen should happen if either of them left the company, I feel like there's a lot of trust on HN (and other places) that's heavily attached to them as individuals and their track record of good communication.

Good communicators generally foster that environment. And their customers appreciate it, so there is an external expectation now too. Everything ends some day, but I think this will be regarded as a valuable attribute for awhile.

This is deeply, deeply embedded in Cloudflare culture.

Re: Cloudflare outage on June 21, 2022

#95

Post by unityByFreedom »

bombcar wrote: post #80
bluehatbrit wrote: post #69

Earlier quoted context omitted.

I do wonder what would happen should happen if either of them left the company, I feel like there's a lot of trust on HN (and other places) that's heavily attached to them as individuals and their track record of good communication.

It could be good or bad; I suspect they've thought about it and have worked on succession (I hope!) and have like-minded people in the wings. But once it happens things will change and, to be honest, likely for the worse. edit> fix typo

> secession

Succession?

Re: Cloudflare outage on June 21, 2022

#96

Post by unityByFreedom »

jgrahamc wrote: post #94
unityByFreedom wrote: post #93

Earlier quoted context omitted.

Good communicators generally foster that environment. And their customers appreciate it, so there is an external expectation now too. Everything ends some day, but I think this will be regarded as a valuable attribute for awhile.

This is deeply, deeply embedded in Cloudflare culture.

Devil's advocate, you could get taken over or end up with a different board. I wouldn't like to see it but someone's got to compete with you or we'll have to send in the FTC! :)

Re: Cloudflare outage on June 21, 2022

#97

Post by tyingq »

samwillis wrote: post #2

In a world where it can take weeks for other companies to publish a postmortem after an outage (if they ever do), I never ceases to amaze me how quickly CF manage to get something like this out. I think it's a testament to their Ops/Incident response teams and internal processes, it builds confidence in their ability to respond quickly when something does go wrong. Incredible work!

>take weeks for other companies to publish a postmortem

And with nowhere near the detail level of what was presented here. Typically lots of sweeping generalizations that don't tell you much about what happened, or give you any confidence they really know what happened or have the right fix in place.

Re: Cloudflare outage on June 21, 2022

#99

Post by april_22 »

kylegalbraith wrote: post #90

As others have said, this is a clear and concise write up of the incident. That is underlined even more when you take into account how quickly they published this. I have seen some companies take weeks or even months to publish an analysis that is half as good as this. Not trying to take the light away from the outage, the outage was bad. But the relative quickness to recovery is pretty impressive, in my opinion. Sou…

I think it's even better that they explained the backgorund of the outage in a really easy to understand way, so that not only experts can get a hang of what was happening.
Post reply on HN