Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare Network Performance Issues

cloudflarestatus.com

Post reply on HN
Print view View original

231–240 of 329 posts

Re: Cloudflare Network Performance Issues

#231

Post by profmonocle »

Once cloudflare.com came back I decided to check out their business SLA, and it's not very encouraging:

> For any and each Outage Period during a monthly billing period the Company will provide as a Service Credit an amount calculated as follows: Service Credit = (Outage Period minutes * Affected Customer Ratio) ÷ Scheduled Availability minutes

- https://www.cloudflare.com/business-sla/

So assuming an outage affects 100% of your users (this one seems like it did, but that's not clear), they only refund the time the service was offline? According to pingdom this outage lasted ~25 minutes, so that's 25/(31 * 24 * 60) = .056% of our bill, roughly 11 cents.

It sounds like you just don't pay for the time the service didn't work, which isn't much of a guarantee, that's just expected (of course you shouldn't pay for services not provided). Most SLAs for critical services have something like under 99.99% uptime you get 10% of your bill back, under 99.5% you get 20% back, under 99% you get 50% back. (*Numbers completely made up to demonstrate the concept.)

Am I misreading this? Morning coffee hasn't kicked in yet so maybe I am.

Re: Cloudflare Network Performance Issues

#232

Post by matt_oriordan »

buildbuildbuild wrote: post #152

Cloudflare: Your status page showed "all systems operational" for over 20 minutes while your primary domain was returning a 502 error. Please change this to update automatically, many other engineering teams depend on you. https://i.imgur.com/qHBM2JW.png

Sadly this reminds me of AWS outages too where the same applies. How is it that hundreds of developers know there's an issue before AWS do, or Cloudflare in this instance. See my blog post on similar AWS uptime reporting issues at https://www.ably.io/blog/honest-status-reporting-aws-service.

At Ably, our status site had an incident update about Cloudflare issues being worked on (by routing away from CF) before Cloudflare did: https://status.ably.io/incidents/647

We have machine generated incidents created automatically when error rates increase beyond a certain point stating "Our automated systems have detected a fault, we've been alerted and looking at it". See https://status.ably.io/incidents/569 for example. I think much larger companies like Cloudflare and Amazon could certainly invest a bit in similar systems to make it easier for their customers to know where the problem likely lies.

Re: Cloudflare Network Performance Issues

#233

Post by swiley »

LeonM wrote: post #101
danShumway wrote: post #50

Don't want to go off topic, but if I want to prevent my website going down because of stuff like this in the future, will having back up DNS entries solve the problem? I know DNS will fall back if it can't reach a service, but would a 502 trigger that?

This is a good question. I'm not sure if the browser receives all A/AAAA records from the syscall, or just one. I guess that if the browser has the whole list, and the error is in the 500 range, it could retry a different IP but I'm not sure if browsers do this.

DNS isn’t handled by the kernel, it’s handled by the network library runtime and that does return a list of addresses (I think it actually has you iterate through them in C anyway.)

Re: Cloudflare Network Performance Issues

#234

Post by allanmacgregor »

sauldcosta wrote: post #174

Some people on Twitter are reporting that it's due to a DDoS. http://www.digitalattackmap.com/#anim=1&color=0&country=ALL&... seems to indicate Iran is the source?

This seems FUD the dates are wrong and according to that report Iran is the one attacked

Re: Cloudflare Network Performance Issues

#235

Post by auscompgeek »

buildbuildbuild wrote: post #227
btown wrote: post #192

Earlier quoted context omitted.

If cloudflarestatus.com isn't serving text content as Cache-Control: no-cache, that's itself an egregious bug.

Their status page is hosted on Cloudfront, perhaps adding an unintended caching layer.

Is it? From my end it looks like a plain ec2 instance from statuspage.io.

Re: Cloudflare Network Performance Issues

#236

Post by eeeeeeeeeeeee »

pmlnr wrote: post #72

Good. I'm tired of people not learning that trusting a single gateway with 50% of the internet is bad. Yes, I know, free DDOS protection. There has to be another way of doing this, some mesh based DDOS protection or so.

I agree that the centralization of the internet is troubling, but Cloudflare is solving a systemic problem that nobody else is tackling. The DDoS problem was not being solved for anyone except for enterprise customers until Cloudflare came along and to this day there is very little competition in this space. Your post makes it sound like making a "mesh based DDoS" system is somehow trivial. Who is going to pay for this? How does it work? How do you ensure latency is not atrocious? Why hasn't someone made this already? Cloudflare at least has a financial model that can be sustained, and it doesn't include harvesting all of our personal data.

Without CF, many websites would not stay on-line during an attack. And they would cease to exist because many of those places would never be able to afford DDoS protection. I know so many sites, including ones I run, that I would not be able to keep on the public internet without CF DDoS protection. There really is no real competition in this space.

I think we need to consider the fact that while this outage does take a lot of sites off-line at once, it is temporary, and it is still extremely rare. And the alternative is potentially that many websites would cease to exist at all, period, without something like Cloudflare existing.

Re: Cloudflare Network Performance Issues

#237

Post by frenchman99 »

NKCSS wrote: post #136
frenchman99 wrote: post #106

Any workarounds or solutions ? I'm an on-call engineer with lots of questions coming in. I'm not sure what I can do apart from moving the domain off Cloudflare, bug DNS propagation would take a few hours and by then Cloudflare might be up again.

Outages can always happen, when they do with companies like this, at least you'll know that some of the best people out there are working on the issue and that it will be resolved asap. CloudFlare has proven in the past to be a very capable party, I don't think panicking now and try to move everything away is a smart move. Also, a few people have been saying that even if you want to, the site to do so is not reachabl…

Panicking was not the plan. Asking for advice was.

Re: Cloudflare Network Performance Issues

#238

Post by rc_kas »

madethemcry wrote: post #118

The internet is designed to withstand a nuclear attack. It's so sad to see half of the internet going down because of a single system failure.

It's very convenient for me to be able to tell me clients their site is down because half the internet is down, and there is nothing I can do about it.

A smaller DNS service would not make headlines.

Re: Cloudflare Network Performance Issues

#239

Post by duijf »

btown wrote: post #69
tmlee wrote: post #3

So it seems... Even https://cloudflare.com/ itself is down

More importantly, their admin dashboard is down. It's impossible to bypass their "orange cloud" proxies and send traffic directly to our hosting. That they can't flip a switch and have their nameservers send dash.cloudflare.com to a separate piece of redundant infrastructure is mind-boggling.

We were able to flip this switch on our services through their API, as we have our Cloudflare config in Terraform.

The API wasn't working perfectly, but with some retries we were able to change the config for our domains.

Re: Cloudflare Network Performance Issues

#240

Post by acdha »

EamonnMR wrote: post #187
voidwtf wrote: post #124

Earlier quoted context omitted.

Care to elaborate how they were "trolling" Verizon?

They came off that way in this blog post: https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-...

That didn't seem like trolling – just a public call for Verizon to follow internet best practices. Given that most large ISPs treat failures as a PR exercise, that's probably necessary.
Post reply on HN