Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare was down

cloudflarestatus.com

Post reply on HN
Print view View original

451–460 of 499 posts

Re: Cloudflare was down

#451

Post by dijit »

macNchz wrote: post #272
kiobu wrote: post #188

Earlier quoted context omitted.

I imagine most people would never expect something like this to happen, so having a fallback option when Cloudflare has a huge interruption of service like this is just unthinkable.

All the major cloud infrastructure providers have had outages of varying severity at one point or another...it's something you'd want to take into account for, say, a system that remote controls life-critical devices, but likely isn't worth the engineering time and added complexity for a productivity or social app with a small userbase. Working on many of the latter over the years I've generally said "well if {major…

>"well if {major cloud provider} is down, the internet is going to be all messed up for a bit anyway, so we'll accept the risk of being down when they're down, and reassess whether that keeps making sense as we grow."

This is a very common pattern and falls into the 'nobody got fired for buying cisco/microsoft/intel' trap.

I have two issues with it;

1) It entrenches the largest provider.

You would not extend the same leniency's of outages to the third best cloud provider, this means that people will just keep pushing the monopoly forward. Even if the uptime or service is actually better on another provider.

2) You create a tight coupling of monocultures;

Simply put: You slowly erode the internet. Your site becomes an application in a distributed mainframe operated by a tiny minority of tech companies universally based in the US.

Why is this a problem? I could give moral answers here but I thing pragmatic ones are more convincing..

Giving ownership of the internet to the few gives them the ability to set the rules.

If you're on Amazon's AWS, what's to say they don't inspect your e-commerce systems and incorporate your business logic into their amazon.com shopping experience. They do this to their marketplace and create competing products already[0].

If you're doing really well, why not just drop a few packets here and there? I mean, they wont.. you're paying, right?

Hell, if you do super well they can just change the rules and make it so your services get expensive in the exact way you use them, or even legislate you off the platform entirely.

Probably wont happen, but it's a lot of trust you have to admit, and people shit on Apple for having that kind of power, and Apple is not even competing in the same market as most people on this site.

If you're on google cloud (which, I'm a fan of btw) and you feed an ML model, well, you paid for it but why shouldn't they also have a copy.. after all, it's green to do so! Their bigdata platform? Google loves data! Feed the beast.

[0]: https://fortune.com/2016/04/20/amazon-copies-merchants/

Re: Cloudflare was down

#452

Post by Kalium »

mmahemoff wrote: post #383
adsjhdashkj wrote: post #332

Earlier quoted context omitted.

I feel like for a lot of sites CF & CDNs are the only way to survive Reddit/HN/etc - do you disagree? I definitely agree in concept with you, but then i think back to how frequently script kiddies took down sites ~10 years ago, or w/e. I feel like what has changes is the massive CDNs in front of so many sites. So while i do want a better solution, i'm not sure what it looks like. Thoughts?

Some kind of decentralized CDN in theory.

CDNs are decentralized by nature.

Re: Cloudflare was down

#453

Post by hmmazoids »

karlmcguire wrote: post #14

"All systems operational" What's the point of a status page if it doesn't reflect the real status... It's either the status page goes down with everything else or the status page is wrong. Great. EDIT: Looks like it's accurate now, 20 minutes later.

Ahh I remember when AWS went down (think it was 2 years ago now?) or at least a data center in us-east? Majority of the internet went down and status page went down as well. Man good times.

Re: Cloudflare was down

#454

Post by rconti »

jchw wrote: post #48

Something’s wonky, because it’s not just Cloudflare. One of my personal sites is down that uses nothing but a VPS, and I noticed my Unifi AP disconnect from its controller a little bit ago. Fiber cut? Routing issues?

Huh. My Ubiquiti was reporting WAN link down during this outage. I'm using ATT fiber. I'm wondering if "link down" doesn't mean what I think it means. Now that I check, it says "WAN iface [eth2] transition to state [inactive]". I'm wondering if that means link down or if it's doing service checking.

I actually have a WAN2 configured but not plugged in and it was set to "Load Balancing: Failover Only" ... I wonder if all of my 'connection issues' were software assuming my network link is down and switching interfaces to an unavailable one.

Re: Cloudflare was down

#455

Post by Godel_unicode »

zackees wrote: post #437
brian-armstrong wrote: post #401

Earlier quoted context omitted.

This was a BGP/routing issue and has already been documented. Please don't spread misinformation and hysteria, especially on technical issues like this https://twitter.com/eastdakota/status/1284253034596331520

CEO's are notorious liars. I just had a discussion with John McAfee who said that all good CEO's are feeding misinformation to the press as standard business practice. You can see it here: https://www.youtube.com/watch?v=w_QWrzjKC9I&lc=UgyXzVtjVLbEs... The willingness that you believe a CEO who has every incentive to lie to the press, and the preposterous narrative that this was a simple "router issue" - as if half t…

It's not failure, it's misconfiguration. Totally different issue to be resilient to fat fingers than fires.

Re: Cloudflare was down

#456

Post by rconti »

rconti wrote: post #454
jchw wrote: post #48

Something’s wonky, because it’s not just Cloudflare. One of my personal sites is down that uses nothing but a VPS, and I noticed my Unifi AP disconnect from its controller a little bit ago. Fiber cut? Routing issues?

Huh. My Ubiquiti was reporting WAN link down during this outage. I'm using ATT fiber. I'm wondering if "link down" doesn't mean what I think it means. Now that I check, it says "WAN iface [eth2] transition to state [inactive]". I'm wondering if that means link down or if it's doing service checking. I actually have a WAN2 configured but not plugged in and it was set to "Load Balancing: Failover Only" ... I wonder if…

to reply to myself, if you have a second interface configured for failover, it actually tests against ping.ubnt.com. I bet every single time my ATT fiber has "gone out" for a minute or two at a time, it's been bogus.

  root@USG-PRO-4:~# show load-balance watchdog
  Group wan_failover
  eth2
  status: Running
  pings: 2
  fails: 0
  run fails: 0/3
  route drops: 0
  ping gateway: ping.ubnt.com - REACHABLE

  eth3
  status: Waiting on recovery (0/3)
  failover-only mode
  pings: 1
  fails: 1
  run fails: 3/3
  route drops: 1
  ping gateway: ping.ubnt.com - DOWN
  last route drop   : Fri Jul 17 17:32:58 2020

Re: Cloudflare was down

#457

Post by awinder »

ricopags wrote: post #282
awinder wrote: post #162

NextDNS got taken out by this, id been really happy with it up until now. And unfortunately “dns service went down” has a wide enough blast radius at home now that it’s a real pain.

How did you verify that? I determined the issue was with Cloudflare's DNS by toggling on NextDNS, which worked and continues to.

Most of my devices went belly up and was trying to figure out what it might be (I run NextDNS on my router), switched off to cell and noticed discord was down too so started thinking about NextDNS. I toggled dns to google and noticed it immediately work.

Re: Cloudflare was down

#458

Post by ci5er »

deadbunny wrote: post #436
ci5er wrote: post #406

Earlier quoted context omitted.

With DO, these days, they don't run me out of bandwidth, but my instance falls over (depending on what I am doing - which ain't much), but with AWS, they auto-scale and I get a $5000 bill at the end of the month. I prefer the former.

AWS only auto scales if you configure it to...

Yeah, but I'm stupid and didn't understand all the switches.

Re: Cloudflare was down

#459

Post by saagarjha »

peterwwillis wrote: post #447
rozab wrote: post #136

We can't keep going on like this. The vulnerability of centralised internet infrastructure is a huge problem for everyone. Somebody, somewhere, really ought to sort it all out

Or just stop using the internet. The majority of tech problems stem from people using tech. Don't rely on it alone and you don't have problems.

Yells 'peterwwillis on the internet ;)

Re: Cloudflare was down

#460

Post by thathndude »

ashleyn wrote: post #107

It really defies the original vision of the internet to have so many services depend on a single company. Almost every news site I was reading dropped off at once. I thought for a second that I lost internet in my own house.

My CRM was nonfunctional. That’s some critical infrastructure for me. And then I’m wondering, is it me or is it my CRM. Turns out it’s door #3 - cloudflare
Post reply on HN