Cloudflare was down • Hacker News Board

Re: Cloudflare was down

#291

Post by hn_throwaway_99 » Fri, Jul 17, 2020, 9:53 PM UTC

ashleyn wrote: post #107

It really defies the original vision of the internet to have so many services depend on a single company. Almost every news site I was reading dropped off at once. I thought for a second that I lost internet in my own house.

Agreed, but I think people really underestimated the forces at work that would cause so much consolidation into a couple internet giants.

The original idea was that with the barrier to entry being so low, anyone and everyone could set up their own websites, mail servers, etc.

But with it being so easy to compare and contrast service (i.e. the market being so open), it means that the competitive forces naturally consolidate to a winner-take-all model. If when starting out Cloudflare was just 5% better than the competition, it could have easily taken the vast majority of the mindshare on the internet. Couple that with the fact that there are huge advantages with scale to a business like Cloudflare's, and it's not hard to see how so much of the internet has become dependent on it.

Re: Cloudflare was down

#292

Post by r1ch » Fri, Jul 17, 2020, 9:53 PM UTC

Interestingly this seemed to only affect resolver service. I use Cloudflare pretty extensively on all my sites, but only in DNS mode (no CDN / proxy). The hosts continued to resolve fine during the outage (following root DNS resolution chain, no recursive resolver involved). I imagine their CDN internally uses their resolver service which explains the outages, and some unrelated 3rd parties who don't use CF on their domain at all still created a hard dependency on CF by using their recursive DNS server.

Re: Cloudflare was down

#293

Post by [deleted] » Fri, Jul 17, 2020, 9:54 PM UTC

rozab wrote: post #136

We can't keep going on like this. The vulnerability of centralised internet infrastructure is a huge problem for everyone. Somebody, somewhere, really ought to sort it all out

[deleted]

Re: Cloudflare was down

#294

Post by clairegraham » Fri, Jul 17, 2020, 9:54 PM UTC

wcchandler wrote: post #258

clairegraham wrote: post #209

We were down (downforeveryoneorjustme.com) completely, but back up now (as of a few minutes ago). Our domain wasn't even resolving; we use Cloudflare for frontend and DNS. We had a surge of people checking if Discord was down on our site, then I noticed everything went down shortly after. Discord is still the top check right now. I can't ever remember hitting these kind of traffic numbers before.

I enjoy your service. Have you ever thought about expanding your offerings? I would love to see a recreation of "Internet Pulse"

Thanks! Yep, we have a lot of things on the todo. We want to add more user-focused / location-based outage information since our site is still too reliant on simple HTTP checks to report downtime. This is especially a problem with a Discord outage, for example, where the frontend website is not down, but there might be problems with the API, apps, or other components.

And I'd like to be able to have our site communicate outages like this Cloudflare one, where more than one site might be affected by a larger provider. Automating that is difficult.

This is still a side project, though, so I mostly work on it when I get the urge :)

Re: Cloudflare was down

#295

Post by gruez » Fri, Jul 17, 2020, 9:54 PM UTC

AdamGibbins wrote: post #262

blisseyGo wrote: post #252

Earlier quoted context omitted.

Out of curiosity, done HN use any CDN or other way of DDOS protection? dang?

It's hosted on AWS, you don't need DDoS protection, just a big wallet.

Only if you use their "infinitely scaling" services. eg. s3. If the attacker is hammering you with expensive queries and your database is on 1 ec2 server, you're still going to go down.

Re: Cloudflare was down

#296

Post by jrockway » Fri, Jul 17, 2020, 9:54 PM UTC

rgbrenner wrote: post #254

jrockway wrote: post #109

I don't use Cloudflare, but I do notice Cloudflare services being down. Right now, I can't get to my own website (hosted on DigitalOcean, not through Cloudflare), but Oh Dear claims it's up. So I suspect that the problem is closer to me than it is to DigitalOcean (or Cloudflare).

DO uses cloudflare for their DNS... both for digitalocean.com and for their DNS service.

Good to know! That makes perfect sense based on what I saw during the outage. I had no idea.

Re: Cloudflare was down

#297

Post by xen2xen1 » Fri, Jul 17, 2020, 9:54 PM UTC

rickyc091 wrote: post #146

ashleyn wrote: post #107

It really defies the original vision of the internet to have so many services depend on a single company. Almost every news site I was reading dropped off at once. I thought for a second that I lost internet in my own house.

Same here. Rebooted the router and modem thinking it was me, but my phone was still on wifi then realized it was probably my cloudflare DNS.

Yup, reinforces the thought that you never have both DNS servers with the same service.

Re: Cloudflare was down

#298

Post by randomstring » Fri, Jul 17, 2020, 9:54 PM UTC

What luck, I chose today to install a new piece of network gear. I thought I had managed to totally FUBAR my network. DNS was failing, "ping 1.1" (my current goto test "Am I connected to the internet?" as it requires the fewest keystrokes and hits the Cloudflare DNS 1.0.0.1) failed and I just assumed it was my fault. Backed out my changes, and discover in fact, the internet was down.

Re: Cloudflare was down

#299

Post by WillPostForFood » Fri, Jul 17, 2020, 9:54 PM UTC

anderskaseorg wrote: post #245

tremon wrote: post #129

Earlier quoted context omitted.

I get a DNS servfail when resolving that DNS record, and many others: Server: 8.8.8.8 Address: 8.8.8.8#53 ** server can't find status.discord.com: SERVFAIL So it's either not just cloudflare, or all those sites use cloudflare to host their DNS.

The latter. $ dig +short discord.com ns sima.ns.cloudflare.com. gabe.ns.cloudflare.com.

8.8.8.8 is Google and is down in addition to Cloudflare. 8.8.4.4 was still up.

Re: Cloudflare was down

#300

Post by Fabricio20 » Fri, Jul 17, 2020, 9:55 PM UTC

Rockjodd wrote: post #81

For people who reports this is down, which country are you in? Because all the reported sites works flawless from Norway (Europe) :-)

Brazil here, basically everything down.

I noticed a lot of packet loss to 1.1.1.1, not an outright "outage", maybe they were rolling a deployment?

Edit: Looks like a deployment to me (looking at the logs I could see cascading traces, so it took down one DC and the other started responding - increased latency - and then down, etc..), gonna be an interesting post-mortem!