Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Today's Outage Post Mortem

blog.cloudflare.com

Post reply on HN
Print view View original

61–70 of 159 posts

Re: Today's Outage Post Mortem

#61

Post by GigabyteCoin »

jcr wrote: post #21

To the couldflare folks; It's refreshing to see you take responsibility, but I think you've been a bit too hard on yourselves by taking all the blame. First of all, what you hit was a unknown bug in JunOS, and Juniper is to blame for their part. Using some form of staging to slow roll-out of rule changes might have saved you from a full meltdown, but when you're getting attacked, every second counts. Slow versus fast…

Is junpier to blame for the bug in their OS? Or is cloudflare to blame for not testing JunOS enough before relying on that OS?

Re: Today's Outage Post Mortem

#62

Post by contingencies »

Developing good software comes down to consistently carrying out fundamental practices (regardless of the technology) - Paul M. Duvall

In this case: Development. Versioned change. Test or staging environment. Tests pass. Production.

Re: Today's Outage Post Mortem

#63

Post by ams6110 »

rdl wrote: post #12
eastdakota wrote: post #11

Earlier quoted context omitted.

While we'll discuss it more at length and after a bit of sleep, my hunch is this will be closer to our approach.

The thing which annoyed me the most was losing all DNS. You really need to have the DNS servers in separate infrastructure (ASN, netblock, while anycasted) so there is never a case where both of your DNS are out for a customer domain. The "CNAME" product looks pretty kludgey.

By the same token you (the customer) should not have all your DNS eggs in one basket.

Re: Today's Outage Post Mortem

#64

Post by mprovost »

justincormack wrote: post #60
rdl wrote: post #48

Earlier quoted context omitted.

I've never seen remote power cyclers on big routers in major facilities which have on-site remote hands, even when servers all get both IPMI/LOM board cyclers and physical external cyclers. At most, the routers get a serial port connected to a serial port console server or directly to a modem, and/or an admin ethernet network. I've seen smaller routers, CSU/DSU, etc. type devices in branch offices on cyclers, though.…

You would need another network (not just a vlan) to run this as well, if you are going to try to reach it when nothing else is working.

We just hook up a DSL modem to the OOB network or plug it straight into the OOB interface on a core router. You used to do this with actual modems but it's cheap enough to do it with DSL these days, then you're not dependent on any of your own network to access the device in case of failure.

Re: Today's Outage Post Mortem

#65

Post by mprovost »

naww wrote: post #57
noselasd wrote: post #52

> attack packets were between 99,971 and 99,985 bytes long. This should raise a red flag, as it must be impossible. Ethernet NICs would just bail out on packets longer than what you've set the MTU to, and ethernet frames would just come from the next hop in most cases. And IP packets have a max length field of 16 bit.

http://en.wikipedia.org/wiki/Jumbogram > An optional feature of IPv6, the jumbo payload option, allows the exchange of packets with payloads of up to one byte less than 4 GiB

Yes but they were still seeing packets bigger than the MTU of Ethernet (or Sonet or whatever other layer 1/2 tech they're connected to the rest of the net with). It doesn't matter what higher level protocols can handle.

Re: Today's Outage Post Mortem

#67

Post by TazeTSchnitzel »

larrys wrote: post #42
jgrahamc wrote: post #24

Earlier quoted context omitted.

Everyone who was needed (management, network operations, and the technical support team) were woken up as soon as this happened. The small team who monitor things during the night were quickly calling people. Some folks physically went to the office to help answers phones; others drove to one of our data centers.

"The small team" Didn't see this when I posted my comment. So there is a team monitoring not a single person?

Uptime is a fundamental part of their business, of course it's a team.

Re: Today's Outage Post Mortem

#68

Post by ChuckMcM »

powertower wrote: post #28

> CloudFlare currently runs 23 data centers worldwide. Shouldn't that always say - CloudFlare currently runs in 23 data centers worldwide? Or is that just how one would phrase that if you rent multiple racks or a cage in a datacenter? ...because I've seen that a bunch of times before from just about everyone. Just curious.

A bit nit picky yes? Even Google mentions places as "data centers" which really are co-location facilities hosting a bit of gear. It has come to be a way to identify unique numbers of building rather than ownership in most papers or articles that I've read.

To be clear I share your desire for precise speech, perhaps the could have said 'Cloudflare currently uses 23 data centers worldwide' but given that the person writing this is doing it on a Sunday, probably after a long night and having missed all the things they normally would have been doing on Sunday, I'm willing to cut them some slack.

Re: Today's Outage Post Mortem

#69

Post by eastdakota »

alexchamberlain wrote: post #34
druiid wrote: post #26

As always I'm glad to see Cloudflare post such detailed outage reports. They are one of the few providers I know of that is willing to go into such depth and that is one of the things I appreciate about them. That said, the outage that occurred was one that was indeed fully preventable. We don't exactly have as many locations as they do, but for internal resources at least, not pushing configuration changes to all de…

Although I agree, it would be rather hard to fight an attack if you didn't roll out fairly quickly...

You've identified the rock and the hard place.

Re: Today's Outage Post Mortem

#70

Post by toast0 »

nathannecro wrote: post #13
eastdakota wrote: post #4

Earlier quoted context omitted.

Don't sell yourself short: our ops team has been on our internal chat talking about how to do something exactly like this for the last hour or so. It's difficult at our scale to truly simulate traffic, but we should be able to roll rules out to just subsets of our network. That's already how we handle router OS upgrades. If a small handful of data centers had crashed, likely no one would have noticed because we've de…

Similar to yRetsyM, my domain knowledge doesn't extend to this side, but if your network is undergoing a DDOS or some other form of attack, taking the time to test rules in a pre-production/test server seems to be quite dangerous. What does the ops team think about using multiple hardware vendors?

Doing the wrong fix is at least as dangerous as not handling the DDoS (as shown in this case). Based on their general network architecture, I would think a prudent thing would be a quick sanity test on a pre-production system if available, then deploy to the various colos in groups at intervals that seem appropriate given the nature of the change. If pre-production isn't available, then having the first group be one colo limits the production impact.

If they did some colos as vendor J and some colos as vendor C, I think it would be manageable, but I don't really know how much of the cross colo traffic is actually their routers talking to their routers. Homogeneity in networks makes things easier to manage, until a platform fault breaks everything at the same time. In this case, at least it was related to a change they had made and happened quickly, so it was easy to determine the cause; other platform faults may not be as easy to determine, but if only your vendor X colos fell over, at least you'd have your vendor C colos up and something to look for.

Post reply on HN