Cloudflare outage on June 21, 2022 • Hacker News Board

Re: Cloudflare outage on June 21, 2022

#21

Post by gcau » Tue, Jun 21, 2022, 1:11 PM UTC

nielsole wrote: post #17

gcau wrote: post #9

Am I the only who really doesn't think this is a big deal? They had an outage, they fixed it very quickly. Life goes on. Talking about the outage as if it's reason for us to all ditch CF, then buy/ run our own hardware (which will be totally better), so hyperbolic.

> Talking about the outage as if it's reason for us to all ditch CF at time of writing no comment has done that except you.

I'm referring to other posts and discussions outside this website. I don't expect as much criticism in this post.

Re: Cloudflare outage on June 21, 2022

#22

Post by zinekeller » Tue, Jun 21, 2022, 1:13 PM UTC

ransom1538 wrote: post #11

Still seeing failed network calls. https://i.imgur.com/xHqvOzj.png

Yeah, that's not Cloudflare at all (it's unlikely that CF still uses nginx/1.14).

Re: Cloudflare outage on June 21, 2022

#23

Post by Jamie9912 » Tue, Jun 21, 2022, 1:14 PM UTC

ransom1538 wrote: post #11

Still seeing failed network calls. https://i.imgur.com/xHqvOzj.png

Is that actually coming from Cloudflare? iirc Cloudflare reports it self as Cloudflare not nginx in the 5xx error pages

Re: Cloudflare outage on June 21, 2022

#24

Post by jgrahamc » Tue, Jun 21, 2022, 1:14 PM UTC

nerdbaggy wrote: post #15

Really interesting that 19 cities handle 50% of the requests.

Actually, I think the flip side is even more interesting. If you want to give good, low latency service to 50% of the world you need a lot of data centers.

Re: Cloudflare outage on June 21, 2022

#25

Post by Tsiklon » Tue, Jun 21, 2022, 1:16 PM UTC

This is a great concise explanation. Thank you for providing it so quickly

If you forgive my prying, was this an implementation issue with the maintenance plan (operator or tooling error), a fundamental issue with the soundness of the plan as it stood, or an unexpected outcome from how the validated and prepared changes interacted with the system?

I imagine that an outage of this scope wasn’t foreseen in the development of the maintenance & rollback plan of the work.

Re: Cloudflare outage on June 21, 2022

#26

Post by samwillis » Tue, Jun 21, 2022, 1:17 PM UTC

Jamie9912 wrote: post #23

ransom1538 wrote: post #11

Still seeing failed network calls. https://i.imgur.com/xHqvOzj.png

Is that actually coming from Cloudflare? iirc Cloudflare reports it self as Cloudflare not nginx in the 5xx error pages

The outage this morning manifested itself as a Nginx error page, somewhat unusually for CF.

Re: Cloudflare outage on June 21, 2022

#27

Post by terom » Tue, Jun 21, 2022, 1:17 PM UTC

TODO: use commit-confirm for automated rollbacks

Sounds like a good idea!

Re: Cloudflare outage on June 21, 2022

#28

Post by trollied » Tue, Jun 21, 2022, 1:17 PM UTC

Sounds like Cloudflare need a small low-traffic MCP that they can deploy to first.

Re: Cloudflare outage on June 21, 2022

#29

Post by viraptor » Tue, Jun 21, 2022, 1:19 PM UTC

samwillis wrote: post #7

johnklos wrote: post #3

...and yet they still push so hard for recentralization of the web...

CloudFlare are a hosting provider and CDN, they aren't "push[ing] ... hard for recentralization of the web". If it was AWS, Akamai, Google Cloud, or any of the other massive providers this comment wouldn't be made. I don't really understand the association between centralisation and CloudFlare, other than it being a Meme.

It's often mentioned about AWS, especially when us-east-1 fails. The others are not big enough to affect basically "the internet" when they go down, so don't get pointed out as centralisation issues as much.

And yeah, cf is trying to get as much traffic to go through them as possible and add edge services for more opportunities - that's literally their business. Also now r2 with object storage. They're already too big, harmful (as in actually putting people in danger) and untouchable in some ways.

Re: Cloudflare outage on June 21, 2022

#30

Post by johnklos » Tue, Jun 21, 2022, 1:21 PM UTC

samwillis wrote: post #7

johnklos wrote: post #3

...and yet they still push so hard for recentralization of the web...

CloudFlare are a hosting provider and CDN, they aren't "push[ing] ... hard for recentralization of the web". If it was AWS, Akamai, Google Cloud, or any of the other massive providers this comment wouldn't be made. I don't really understand the association between centralisation and CloudFlare, other than it being a Meme.

I think you've already drunk the Flavor Aid.

What do you have when you have all DNS going through them, via DoH, and all web requests going through them, if not recentralization?

Sure, they want us to think they give us the freedom to host our web sites anywhere because they're "protected" by them, but that "protection" means we've agreed to recentralize.

It's pretty dismissive to describe something as a meme just because you don't understand it, and either you're pretending to not understand it, or you truly don't.

Look at it this way: If a single company goes down for an hour, and that company going down for an hour causes half the web traffic on the Internet to fail for that hour, what is that if not recentralization?