Today's Outage Post Mortem • Hacker News Board

Re: Today's Outage Post Mortem

#101

Post by cbsmith » Sun, Mar 03, 2013, 11:02 PM UTC

mprovost wrote: post #65

naww wrote: post #57

Earlier quoted context omitted.

http://en.wikipedia.org/wiki/Jumbogram > An optional feature of IPv6, the jumbo payload option, allows the exchange of packets with payloads of up to one byte less than 4 GiB

Yes but they were still seeing packets bigger than the MTU of Ethernet (or Sonet or whatever other layer 1/2 tech they're connected to the rest of the net with). It doesn't matter what higher level protocols can handle.

You are joking right? The packet size at the higher layer is what they were matching against. The size of the layer 2 packets is irrelevant.

Re: Today's Outage Post Mortem

#102

Post by scoot » Sun, Mar 03, 2013, 11:06 PM UTC

graycat wrote: post #90

Yes, case number 384,449,194 of systems management causing a system problem. Also case number 439,224 of what looked like a localized problem quickly causing a huge system, e.g., all 23 data centers around the world, to crash. They have my sympathy: So, they typed in a 'rule'. At one time I was working in 'artificial intelligence' (AI), actually 'expert systems', based on using 'rules' to implement real time manageme…

'in the words of the HAL 9000, "human error".'

Except that it wasn't human error, at least not in the sense that the decision to enter the rule, or the rule itself was in error. The human error was with the bug in the Juniper firmware that caused this rule to crash this router, and arguably with the CloudFlare process that allows rules to be propagated to all routers concurrently, rather than segmenting the network and testing for success before further deployment.

Re: Today's Outage Post Mortem

#103

Post by [deleted] » Sun, Mar 03, 2013, 11:06 PM UTC

graycat wrote: post #90

Yes, case number 384,449,194 of systems management causing a system problem. Also case number 439,224 of what looked like a localized problem quickly causing a huge system, e.g., all 23 data centers around the world, to crash. They have my sympathy: So, they typed in a 'rule'. At one time I was working in 'artificial intelligence' (AI), actually 'expert systems', based on using 'rules' to implement real time manageme…

[deleted]

Re: Today's Outage Post Mortem

#104

Post by packetslave » Sun, Mar 03, 2013, 11:10 PM UTC

rdl wrote: post #48

rainsford wrote: post #33

That was a pretty interesting writeup and I always like it when companies are totally (and quickly) upfront about negative events. One thing that occurred to me though is that performing a hard reboot of the routers required calling people to physically access the devices and took some time to perform (as you would expect). Although I wouldn't expect it to be needed very often, I'm sort of surprised CloudFlare doesn'…

I've never seen remote power cyclers on big routers in major facilities which have on-site remote hands, even when servers all get both IPMI/LOM board cyclers and physical external cyclers. At most, the routers get a serial port connected to a serial port console server or directly to a modem, and/or an admin ethernet network. I've seen smaller routers, CSU/DSU, etc. type devices in branch offices on cyclers, though.…

A lot of big routers run (or have the option to run) on DC power, too, which makes it harder. Much of the remote power control gear out there is AC

Re: Today's Outage Post Mortem

#105

Post by rdl » Sun, Mar 03, 2013, 11:10 PM UTC

macros wrote: post #71

mprovost wrote: post #64

Earlier quoted context omitted.

We just hook up a DSL modem to the OOB network or plug it straight into the OOB interface on a core router. You used to do this with actual modems but it's cheap enough to do it with DSL these days, then you're not dependent on any of your own network to access the device in case of failure.

We've been doing this with mikrotik boxes with either wifi or usb gsm modems depending on the what is available in the location.

Yeah, I've seen a lot of great options for OOB access: 1) At carrier hotels, wifi (heh) 2) Cellular modems (ideal for branch offices; a lot of datacenters have bad cell coverage inside the racks/cabinets/floor though) 3) Cross-connect (in places with free/cheap cross connects) to someone you don't use for transit. Can be mutual 4) Some facilities give you an OOB network, although this often has issues (if you buy transit from them, it's possible your outage is due to something going wrong with them, and it might take out your OOB access)

I'm looking at the Verizon Private-IP thing (an outsourced private network over Verizon's cell infrastructure) for OOB management of lots of CPE; the cost per device per month is low, and then you pay for bandwidth across all of them. Makes initial provisioning easier, plus ongoing monitoring/maintenance.

Re: Today's Outage Post Mortem

#106

Post by packetslave » Sun, Mar 03, 2013, 11:13 PM UTC

windexh8er wrote: post #20

rdl wrote: post #15

Earlier quoted context omitted.

That works when the interfaces are totally standard, but edge/core routers are not like that. Cisco supports one set of protocols for talking to other Cisco products; another set for talking to everything else. The "everything else" protocols suck in a lot of ways (they're ok inter-site, but not really so great intra-site). Same with Juniper. (there aren't really other viable options besides those two) You could buil…

Re: "there aren't really any viable options..." Total misconception. BGP, OSPF, ISIS, LISP, etc. are all non proprietary. Sure, the root cause of this particular problem is that CF is using something specific to Juniper, however router interoperability is not predicated on components like that. This example was a tool CF operationalized, and likely had little to do with their routing with the exception of it being a…

You'd be amazed how often "standard" network protocols behave subtly different between vendors. You have to exhaustively test interoperability for every single feature and config option if you want assurance that it isn't going to break in some bizarre way.

Re: Today's Outage Post Mortem

#107

Post by packetslave » Sun, Mar 03, 2013, 11:16 PM UTC

dubcanada wrote: post #78

GigabyteCoin wrote: post #61

Earlier quoted context omitted.

Is junpier to blame for the bug in their OS? Or is cloudflare to blame for not testing JunOS enough before relying on that OS?

One would assume paying a company for an OS should be tested via the developers. Juniper SHOULD have tested that route scheme since they sell mission critical architecture. However you are most likely right, CloudFlare should have test ed it before rolling out.

Well-known Google SRE motto: "hope is not a strategy".

Re: Today's Outage Post Mortem

#108

Post by scoot » Sun, Mar 03, 2013, 11:17 PM UTC

windexh8er wrote: post #20

rdl wrote: post #15

Earlier quoted context omitted.

That works when the interfaces are totally standard, but edge/core routers are not like that. Cisco supports one set of protocols for talking to other Cisco products; another set for talking to everything else. The "everything else" protocols suck in a lot of ways (they're ok inter-site, but not really so great intra-site). Same with Juniper. (there aren't really other viable options besides those two) You could buil…

Re: "there aren't really any viable options..." Total misconception. BGP, OSPF, ISIS, LISP, etc. are all non proprietary. Sure, the root cause of this particular problem is that CF is using something specific to Juniper, however router interoperability is not predicated on components like that. This example was a tool CF operationalized, and likely had little to do with their routing with the exception of it being a…

"But, the reality of the situation is that all of this gear is marked up well into the thousands of percent."

You seem to be confusing hardware with software. Juniper's gross margin was 64.25% for the quarter ending Dec. 31, 2012, and in that ballpark for previous quarters back to inception.

Re: Today's Outage Post Mortem

#109

Post by graycat » Sun, Mar 03, 2013, 11:19 PM UTC

scoot wrote: post #102

graycat wrote: post #90

Yes, case number 384,449,194 of systems management causing a system problem. Also case number 439,224 of what looked like a localized problem quickly causing a huge system, e.g., all 23 data centers around the world, to crash. They have my sympathy: So, they typed in a 'rule'. At one time I was working in 'artificial intelligence' (AI), actually 'expert systems', based on using 'rules' to implement real time manageme…

'in the words of the HAL 9000, "human error".' Except that it wasn't human error, at least not in the sense that the decision to enter the rule, or the rule itself was in error. The human error was with the bug in the Juniper firmware that caused this rule to crash this router, and arguably with the CloudFlare process that allows rules to be propagated to all routers concurrently, rather than segmenting the network a…

"or the rule itself was in error"

I thought that the rule was in error: I couldn't read the rule clearly on the screen, but it seemed, or I guessed, that the problem with the rule was that the "humans" omitted the decimal points and, thus, asked for blocking packets with lengths 1000 larger than intended. The Juniper software got sick, i.e., allocated too much memory, only because it was trying to swallow working with such absurdly large packet sizes. But, then, I couldn't clearly read the screen capture with the rule.

Re: Today's Outage Post Mortem

#110

Post by cbsmith » Sun, Mar 03, 2013, 11:24 PM UTC

naww wrote: post #57

noselasd wrote: post #52

> attack packets were between 99,971 and 99,985 bytes long. This should raise a red flag, as it must be impossible. Ethernet NICs would just bail out on packets longer than what you've set the MTU to, and ethernet frames would just come from the next hop in most cases. And IP packets have a max length field of 16 bit.

http://en.wikipedia.org/wiki/Jumbogram > An optional feature of IPv6, the jumbo payload option, allows the exchange of packets with payloads of up to one byte less than 4 GiB

People are praising the transparency of this report, but I am not sure I agree because of this point. when I read the report, I had to stop to think when I read the part about packet size to conclude that they had to be talking about an IPv6 packet using the hop-by-hop extension for fragmented packets. That is a special case, because you don't actually know the length of the packet until you receive the last fragment.

As a consequence, fragmented ipv6 packets are error for use in DoS attacks. This is not a "weird" occurrence, but rather an expected one, and since end points are not required to accept such huge packets, I am surprised Cloud flare want already doing all it could to advertise to upstream sources that IPv6 fragments longer than a much smaller than 90K should be dropped, at least if rooted to their DNS. I am also surprised that when their software came up with that kind of a response without first validating that it wouldn't cause the exact memory problem it did. Rules on v6 fragmented packets that can't match on a single fragment are inherently dangerous. It is only reasonable to have safe guards already in place for them.

I am also not sure this is really a bug in Juniper software. I imagine the memory problem only shows up with high traffic and in the midst of a DoS attack. That is kind of a given when you put a rule like that in that kind of a situation.