Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Network Instability in NYC2 on July 29, 2014

digitalocean.com

Post reply on HN
Print view View original

11–20 of 22 posts

Re: Network Instability in NYC2 on July 29, 2014

#11

Post by imbriaco »

bogomipz wrote: post #10

I find it alarming that they don't have a competent network engineer of their own on staff. Note the following: "We are working very closely with our networking partner to understand the nature of the failure, assess the chances of a repeat event, and to begin planning architectural changes for the future." "Our initial focus was on verifying the configuration so we initiated a line-by-line configuration review by en…

We have very competent network engineers, but it never hurts to have a second opinion. Verifying that there are no subtle interactions that we've missed is simply prudent.

At our scale, partnering with our vendors is a necessity. We often operate in areas that, while they are technically within specification, are toward the higher end of the range.

Re: Network Instability in NYC2 on July 29, 2014

#12

Post by jwatte »

imbriaco wrote: post #11
bogomipz wrote: post #10

I find it alarming that they don't have a competent network engineer of their own on staff. Note the following: "We are working very closely with our networking partner to understand the nature of the failure, assess the chances of a repeat event, and to begin planning architectural changes for the future." "Our initial focus was on verifying the configuration so we initiated a line-by-line configuration review by en…

We have very competent network engineers, but it never hurts to have a second opinion. Verifying that there are no subtle interactions that we've missed is simply prudent. At our scale, partnering with our vendors is a necessity. We often operate in areas that, while they are technically within specification, are toward the higher end of the range.

Which vendor? I have experience with two brands in similar condos, one good, one not so much...

Re: Network Instability in NYC2 on July 29, 2014

#13

Post by snewman »

A good postmortem... and a clear example of a Type 1 Outage. There are pretty much only two kinds of outage in a decently-run system:

1. Two or three things go wrong at once. Some of the problems are spontaneous, and some were always broken but it took the other problem(s) to uncover it. (Here, the SSD failure was presumably spontaneous, but the "not completely successful" routing engine failover sounds like a case of rarely-used-hence-poorly-tested.)

2. A systemic failure hits all redundant components at once (DDOS, fat-fingered global configuration change, calendar bug, etc.)

Re: Network Instability in NYC2 on July 29, 2014

#14

Post by akg_67 »

The network problems seem to have started earlier. I received a penny for my NYC2 droplet just before this major incident. Luckily, I decided to shutdown and move the droplet to SFO before this incident. Over the last couple of years dealing with cloud providers, I have learned to dust off the contingency plan as soon there is whiff of issues at a location.

Though the recap is good, there are red flags that show gaps in DO processes and incident management.

Upgrade of software before the original problem was diagnosed and resolved. This is a big no-no, never introduce a new variable in an existing problem even if the service provider or lab testing shows the chances of failure are minimal. I have worked long enough with technology infrastructure to experience situations where service provider insisted on upgrading software during unrelated incident and made situation worse.

A better approach would have been to fail the network to good switch first and when the bad switch was fixed, upgrade the software on bad switch first then failover to the upgraded switch and upgrade software on good switch. This time you guys got lucky but sooner or later your luck will run out.

Re: Network Instability in NYC2 on July 29, 2014

#15

Post by akg_67 »

bogomipz wrote: post #10

I find it alarming that they don't have a competent network engineer of their own on staff. Note the following: "We are working very closely with our networking partner to understand the nature of the failure, assess the chances of a repeat event, and to begin planning architectural changes for the future." "Our initial focus was on verifying the configuration so we initiated a line-by-line configuration review by en…

Requesting Failure Analysis (FA) and Root Cause Analysis (RCA) from equipment provider is pretty standard after major incidents. It helps customer understand the issue better and put process in place to handle similar incidents. Similarly, vendors do it willingly because it allows them to learn more about issue and either address issue with product enhancement or put out alerts to their service org/ other customers with potential troubleshooting and resolution guidelines.

Re: Network Instability in NYC2 on July 29, 2014

#16

Post by pnmahoney »

Donzo wrote: post #9

While the outage was upsetting, the response from DO was reassuring. Not only did they jump right on the problem, they maintained communication during the affected period. Then, they refunded me $160 (a month of service). Every host has an occasional problem. It's how they handle the problem that is important to me. This is a night and day contrast compared to the service I received with other hosts with which I've d…

was this other people's experience? e.g. getting a refund?

Re: Network Instability in NYC2 on July 29, 2014

#17

Post by imbriaco »

akg_67 wrote: post #14

The network problems seem to have started earlier. I received a penny for my NYC2 droplet just before this major incident. Luckily, I decided to shutdown and move the droplet to SFO before this incident. Over the last couple of years dealing with cloud providers, I have learned to dust off the contingency plan as soon there is whiff of issues at a location. Though the recap is good, there are red flags that show gaps…

Generally speaking I agree with you. However, in this specific case there were a couple of reasons we chose the path we did:

1. We had experienced bugs with the currently running release which we were fairly sure would manifest when we removed the damaged core from the network. These were primarily around MAC learning.

2. We had performed testing ourselves in a non-production environment and were already planning to take a network maintenance to update these switches in the next few days.

Given those factors, we judged that introducing another variable in the new version was less risky than proceeding with the defects that we knew about in the existing version.

Re: Network Instability in NYC2 on July 29, 2014

#18

Post by imbriaco »

jwatte wrote: post #12
imbriaco wrote: post #11

Earlier quoted context omitted.

We have very competent network engineers, but it never hurts to have a second opinion. Verifying that there are no subtle interactions that we've missed is simply prudent. At our scale, partnering with our vendors is a necessity. We often operate in areas that, while they are technically within specification, are toward the higher end of the range.

Which vendor? I have experience with two brands in similar condos, one good, one not so much...

I prefer not to publicly name the vendor. This is about problems on our network, and it's ultimately our responsibility.

Re: Network Instability in NYC2 on July 29, 2014

#19

Post by insaneirish »

imbriaco wrote: post #11
bogomipz wrote: post #10

I find it alarming that they don't have a competent network engineer of their own on staff. Note the following: "We are working very closely with our networking partner to understand the nature of the failure, assess the chances of a repeat event, and to begin planning architectural changes for the future." "Our initial focus was on verifying the configuration so we initiated a line-by-line configuration review by en…

We have very competent network engineers, but it never hurts to have a second opinion. Verifying that there are no subtle interactions that we've missed is simply prudent. At our scale, partnering with our vendors is a necessity. We often operate in areas that, while they are technically within specification, are toward the higher end of the range.

I urge you to seriously reconsider your design. VPC/MLAG/etc. are attractive because of the ease of making any VLAN available anywhere without the traditional spanning tree nightmare of doing so. However, you're trading one nightmare for another. Your core or aggregation switches are now stateful, and you've just experienced the results of stateful network components not failing cleanly.

I'm not saying an overlay (e.g. VMware NSX née Nicira) is the solution either. I'd suggest rethinking why you need any VLAN anywhere, which is likely the reason for VPC/MLAG. If I'm wrong and you don't put the same VLAN in multiple cabinets, then rip VPC/MLAG out immediately and switch to a pure L3 design. It is so much simpler and consistent with industry best practices.

Re: Network Instability in NYC2 on July 29, 2014

#20

Post by nieve »

pnmahoney wrote: post #16
Donzo wrote: post #9

While the outage was upsetting, the response from DO was reassuring. Not only did they jump right on the problem, they maintained communication during the affected period. Then, they refunded me $160 (a month of service). Every host has an occasional problem. It's how they handle the problem that is important to me. This is a night and day contrast compared to the service I received with other hosts with which I've d…

was this other people's experience? e.g. getting a refund?

I only keep three machines with them, but two are in NYC2. For the July 24th outage they issued a combined 4¢ SLA credit for claimed two hours downtime. For this much worse problem? Nada. I suspect that DO is prioritizing refunds for more lucrative customers and if you're not spending enough you're not going to get anything. I may need to look into moving some stuff over to Ramnode if they've got a NYC location now (Wall Street clients).
Post reply on HN