Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare was down

cloudflarestatus.com

Post reply on HN
Print view View original

441–450 of 499 posts

Re: Cloudflare was down

#441

Post by _asummers »

imron wrote: post #422
leon-z wrote: post #273

Earlier quoted context omitted.

Kudos to the people at Discord. Just a few minutes after I got disconnected they already tweeted about the issue. Some minutes later and they have a message in their desktop app confirming it's an issue with Cloudflare. All while Cloudflare's statuspage says there are 'minor outages'.

As a percentage of total traffic, a 'minor' outage for Cloudflare probably equates to a significant outage for a non-trivial amount of the internet. It will also be especially noticeable to end-users, because sites using Cloudflare are typically high-traffic sites, and so a 'minor' issue that affects only a handful of sites is still going to be noticed by millions of people.

"Your nines are not my nines" [0]

[0] https://rachelbythebay.com/w/2019/07/15/giant/

Re: Cloudflare was down

#442

Post by eastdakota »

zackees wrote: post #437
brian-armstrong wrote: post #401

Earlier quoted context omitted.

This was a BGP/routing issue and has already been documented. Please don't spread misinformation and hysteria, especially on technical issues like this https://twitter.com/eastdakota/status/1284253034596331520

CEO's are notorious liars. I just had a discussion with John McAfee who said that all good CEO's are feeding misinformation to the press as standard business practice. You can see it here: https://www.youtube.com/watch?v=w_QWrzjKC9I&lc=UgyXzVtjVLbEs... The willingness that you believe a CEO who has every incentive to lie to the press, and the preposterous narrative that this was a simple "router issue" - as if half t…

It was a misconfiguration we applied to a router in Atlanta during routine maintenance. That caused bad routes on our private backbone. As a result, traffic from any locations connected to the backbone got routed to Atlanta. It resulted in about 50% of traffic to our network to not resolve for about 20 minutes. Locations not connected to our backbone were not impacted. It was a human error. It was not an attack. It was not a failure or bug in the router. We're adding mitigations to our backbone network as we speak to ensure that a mistake like this can't have broad impacts in the future. We'll have a blog post with a full explanation up in the next hour or so — being written now by our CTO.

Re: Cloudflare was down

#444

Post by wcchandler »

clairegraham wrote: post #294
wcchandler wrote: post #258

Earlier quoted context omitted.

I enjoy your service. Have you ever thought about expanding your offerings? I would love to see a recreation of "Internet Pulse"

Thanks! Yep, we have a lot of things on the todo. We want to add more user-focused / location-based outage information since our site is still too reliant on simple HTTP checks to report downtime. This is especially a problem with a Discord outage, for example, where the frontend website is not down, but there might be problems with the API, apps, or other components. And I'd like to be able to have our site communic…

That would be pretty interesting. Being able to drill down into individual pieces of a stack would be very informative, especially from a parent source. I bet a lot of services would even have that information readily available in their documentation for APIs.

How comfortable are you with open source? If you were willing to release your stack on Gitlab/Github, it might be worth your while.

Re: Cloudflare was down

#445

Post by tuananh »

ksec wrote: post #331
snazz wrote: post #265

Earlier quoted context omitted.

Don't think so. They used to use Cloudflare but stopped. To my knowledge, it's a single server without a database (using the filesystem as a database).

So HN is serving 5.5M page view daily (excluding API access ) on a single server without CDN and without a database? Holy crap I am thinking either there is some magic or everything we are doing in the modern web are wrong. Edit: The number is from Dang [1] > These days around 5.5M page views daily and something like 5M unique readers a month, depending on how you try to count them. [1] https://news.ycombinator.com/i…

that number is not very big

i used to host a wordpress site that has 5M pageviews a month on a $10 (and later $20) digitalocean instance.

that's wordpress and a shared vps. I imagine it could be a lot higher if I have dedicated server and use self-written software.

Re: Cloudflare was down

#446

Post by zetalemur »

mjlawson wrote: post #137
Jasper_ wrote: post #38

Earlier quoted context omitted.

The point of the status page is so you can point to it for your five nines SLA and go "look? we were only down for one hour". As soon as the money relies on the metric, the metric will reflect the money.

Goodhart's Law[1] in action. [1] https://en.wikipedia.org/wiki/Goodhart%27s_law

That's a good one, did not know that. Interesting read.

Re: Cloudflare was down

#447

Post by peterwwillis »

rozab wrote: post #136

We can't keep going on like this. The vulnerability of centralised internet infrastructure is a huge problem for everyone. Somebody, somewhere, really ought to sort it all out

Or just stop using the internet. The majority of tech problems stem from people using tech. Don't rely on it alone and you don't have problems.

Re: Cloudflare was down

#448

Post by fsociety »

zackees wrote: post #437
brian-armstrong wrote: post #401

Earlier quoted context omitted.

This was a BGP/routing issue and has already been documented. Please don't spread misinformation and hysteria, especially on technical issues like this https://twitter.com/eastdakota/status/1284253034596331520

CEO's are notorious liars. I just had a discussion with John McAfee who said that all good CEO's are feeding misinformation to the press as standard business practice. You can see it here: https://www.youtube.com/watch?v=w_QWrzjKC9I&lc=UgyXzVtjVLbEs... The willingness that you believe a CEO who has every incentive to lie to the press, and the preposterous narrative that this was a simple "router issue" - as if half t…

You should really read up on BGP. It really is that flimsy.

Re: Cloudflare was down

#449

Post by [deleted] »

macNchz wrote: post #180
rococode wrote: post #9

Discord is entirely down right now, both the website and the app itself. Amusingly, a lot of the sites that normally track outages are also down, which made me think it was my internet at first. Downdetector, monitortheinternet, etc. Lots of other big sites that are down: Patreon, npmjs, DigitalOcean, Coinbase, Zendesk, Medium, GitLab (502), Fiverr, Upwork, Udemy Edit: 15 min later, looks like things are starting to…

My iPhone actually popped up a message saying that my wifi didn't appear to have internet, which was strange and obviously false as I was actively using the internet on it and the laptop next to it, but now it makes sense that it must have been pinging something backed by cloudflare!

[deleted]

Re: Cloudflare was down

#450

Post by Kalium »

lima wrote: post #417
fivre wrote: post #353

Earlier quoted context omitted.

10-20 minute router misconfigurations and subsequent fixes are sometimes a fact of life. big network infrastructure is complicated, and sometimes the best laid route tables of mice and men do go abloop and die. Outages happen no matter what the infrastructure is. There's no solution, they're just something you need to recognize and handle, which Cloudflare seemingly did relatively quickly here.

Yes, but other providers are not a single point of failure for a significant percentage of the internet. Level 3 or Telia going offline is perfectly survivable for any customer who has multiple upstreams.

Remember the big Dyn outage? Or when AWS-US-East was severely disrupted by a hurricane?

It may perhaps be an exaggeration to say that there are not other providers that are similarly critical for a significant percentage of the internet.

Post reply on HN