Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Today's Outage Post Mortem

blog.cloudflare.com

Post reply on HN
Print view View original

91–100 of 159 posts

Re: Today's Outage Post Mortem

#91

Post by eastdakota »

[deleted] wrote: post #87
eastdakota wrote: post #72

Earlier quoted context omitted.

Buck stops with us. We choose the hardware and software that runs on our network. We test and work around thousands of bugs in it. It was up to us to check range limits before applying them. While we'll never be perfect, one of the things I am most proud of with the CloudFlare team is how quickly we do learn from mistakes.

[deleted]

To be clear, while I wrote out the effective command that was pushed to the router, it was a script that generated it and the fault lies with the script not having adequate error handling.

Re: Today's Outage Post Mortem

#93

Post by [deleted] »

eastdakota wrote: post #91
[deleted] wrote: post #87

Earlier quoted context omitted.

[deleted]

To be clear, while I wrote out the effective command that was pushed to the router, it was a script that generated it and the fault lies with the script not having adequate error handling.

[deleted]

Re: Today's Outage Post Mortem

#94

Post by larrys »

TazeTSchnitzel wrote: post #67
larrys wrote: post #42

Earlier quoted context omitted.

"The small team" Didn't see this when I posted my comment. So there is a team monitoring not a single person?

Uptime is a fundamental part of their business, of course it's a team.

What do you mean "of course it's a team" because "uptime is a fundamental part of their business"?

That's an assumption on your part. Many negative events happen because of assumptions people make and things they take for granted. I ask a valid question. Do you know for a fact how many people monitor their network or what systems they have in place? Or whether they even simulate events like this to see if other members of the team are even reachable? Have you ever seen the constant testing that goes on at large organizations (such as the military) to make sure battle systems are ready to deploy when necessary?

Note also that cloudflare charges $3000 per month for enterprise plans (in addition to free) and that their own website was taken offline.

Re: Today's Outage Post Mortem

#95

Post by Dylan16807 »

[deleted] wrote: post #89
jgrahamc wrote: post #24

Earlier quoted context omitted.

Everyone who was needed (management, network operations, and the technical support team) were woken up as soon as this happened. The small team who monitor things during the night were quickly calling people. Some folks physically went to the office to help answers phones; others drove to one of our data centers.

[deleted]

Well I was in the process of upvoting you to a less faded gray and suggesting you be less rude when you're making a valid point, but "edit: downvotes? truth hurts." tips the line to where you just go ahead and keep on keeping on and get yourself banned have a nice day see you never.

Re: Today's Outage Post Mortem

#96

Post by larrys »

tlrobinson wrote: post #79
powertower wrote: post #28

> CloudFlare currently runs 23 data centers worldwide. Shouldn't that always say - CloudFlare currently runs in 23 data centers worldwide? Or is that just how one would phrase that if you rent multiple racks or a cage in a datacenter? ...because I've seen that a bunch of times before from just about everyone. Just curious.

The distinction is a bit arbitrary. As a customer you should care that their service is geographically distributed, not whether they own the buildings where the servers are kept.

"As a customer you should care that their service is geographically distributed"

Don't agree. If you own the data center you have more control over it. We had a case where the UPS systems in a data center had bad batteries and equipment went down because the batteries failed to kick in. Since we don't own the data center we have no realistic way to make inspections and make sure the right thing happens or that the batteries (or the generators) are cycled and maintained. We just have to trust. [1]

Now this may or may not matter with the way they have their redundancy setup. But owning a data center does give you more control over more things.

"not whether they own the buildings where the servers are kept."

Owning the data center and owning the buildings are two different things. Owning the building is owning real estate. Owning the data center is owning the security setup, backup systems etc. Two different things.

[1] So as not to contradict myself with other things I have said I should not say "trust" because you can always put some things in place to verify the right thing is happening (inspections, logs etc.) if you want. But if you own the place it's easier. If I own my home I can decide when to replace the HVAC so it doesn't fail in the middle of the summer. If I rent that's up to the landlord.

Re: Today's Outage Post Mortem

#97

Post by [deleted] »

[deleted] wrote: post #89
jgrahamc wrote: post #24

Earlier quoted context omitted.

Everyone who was needed (management, network operations, and the technical support team) were woken up as soon as this happened. The small team who monitor things during the night were quickly calling people. Some folks physically went to the office to help answers phones; others drove to one of our data centers.

[deleted]

[deleted]

Re: Today's Outage Post Mortem

#98

Post by acdha »

larrys wrote: post #96
tlrobinson wrote: post #79

Earlier quoted context omitted.

The distinction is a bit arbitrary. As a customer you should care that their service is geographically distributed, not whether they own the buildings where the servers are kept.

"As a customer you should care that their service is geographically distributed" Don't agree. If you own the data center you have more control over it. We had a case where the UPS systems in a data center had bad batteries and equipment went down because the batteries failed to kick in. Since we don't own the data center we have no realistic way to make inspections and make sure the right thing happens or that the ba…

The downside is that this obligates you to do a wider variety of tasks. I'd be surprised if CloudFlare has enough profit margin to afford many redundant electrical engineers, physical network & power techs, etc. in 23 widely dispersed locations. It's obvious that the extra expense wouldn't have helped in this case but it certainly would consume a big chunk of money and management time.

Part of being a business is that you have to make tradeoffs in the real world rather than game-theoretical perfect moves. In many cases this means carefully writing contracts because you can't afford the certain expense and distraction of doing it in-house in the hope that the results might be slightly better.

Re: Today's Outage Post Mortem

#99

Post by rdl »

rincebrain wrote: post #75
ams6110 wrote: post #63

Earlier quoted context omitted.

By the same token you (the customer) should not have all your DNS eggs in one basket.

CloudFlare's CDN bits require you to give them DNS delegation of your stuff, last I looked.

There is a way around it with some of the premium accounts, but it kind of seems like a hack.

Re: Today's Outage Post Mortem

#100

Post by acdha »

random42 wrote: post #66

OT: I want to pitch cloudflare for our CDN needs. Can someone estimate the scale of cloudflare wrt. akamai (current provider), in terms of operations, consumers etc.?

Akamai is much larger – 100+K edge nodes, many collocated in ISP's local facilities – and has much, much better global coverage, particularly if you care about Asia or South America.

Downside: priced accordingly.

Post reply on HN