Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Today's Outage Post Mortem

blog.cloudflare.com

Post reply on HN
Print view View original

11–20 of 159 posts

Re: Today's Outage Post Mortem

#11

Post by eastdakota »

rdl wrote: post #7
lazyjones wrote: post #6

So what are they going to change as a consequence? It seems logical to not rely on a single router vendor anymore, or to test new rules on a staging setup at least for a very short time before pushing them to all routers.

Running Vendor J and Vendor C routers together means you get exposed to the weird bugs in either's open/interoperability code, and lose out on all the advanced features (since most of the good stuff isn't well supported in true cross-platform vendor independent fashion). It's probably more reasonable to split your network into a few more independent sections and never do updates which affect everything, but unless yo…

While we'll discuss it more at length and after a bit of sleep, my hunch is this will be closer to our approach.

Re: Today's Outage Post Mortem

#12

Post by rdl »

eastdakota wrote: post #11
rdl wrote: post #7

Earlier quoted context omitted.

Running Vendor J and Vendor C routers together means you get exposed to the weird bugs in either's open/interoperability code, and lose out on all the advanced features (since most of the good stuff isn't well supported in true cross-platform vendor independent fashion). It's probably more reasonable to split your network into a few more independent sections and never do updates which affect everything, but unless yo…

While we'll discuss it more at length and after a bit of sleep, my hunch is this will be closer to our approach.

The thing which annoyed me the most was losing all DNS. You really need to have the DNS servers in separate infrastructure (ASN, netblock, while anycasted) so there is never a case where both of your DNS are out for a customer domain. The "CNAME" product looks pretty kludgey.

Re: Today's Outage Post Mortem

#13

Post by nathannecro »

eastdakota wrote: post #4
yRetsyM wrote: post #3

I'm not very educated on this end of the spectrum - but I wonder if a process is possible where a rule or router update of some description is applied to one router only, testing the specific schema before pushing to the rest of the routers, thereby failing one router and not failing the rest? I understand the need to respond as quickly as possible - but as stated in this case this was already a manual response. It a…

Don't sell yourself short: our ops team has been on our internal chat talking about how to do something exactly like this for the last hour or so. It's difficult at our scale to truly simulate traffic, but we should be able to roll rules out to just subsets of our network. That's already how we handle router OS upgrades. If a small handful of data centers had crashed, likely no one would have noticed because we've de…

Similar to yRetsyM, my domain knowledge doesn't extend to this side, but if your network is undergoing a DDOS or some other form of attack, taking the time to test rules in a pre-production/test server seems to be quite dangerous. What does the ops team think about using multiple hardware vendors?

Re: Today's Outage Post Mortem

#14

Post by dageshi »

dododo wrote: post #9

if you want to build a reliable system, one useful thing to do is use equipment from multiple vendors. sure it's inconvenient, but by doing this you can often de-correlate failures. especially if you want to improve someone else's reliability. e.g., from simple things like hard drives in a raid from different vendors, to n-version programming in safety critical systems (like airplanes).

Will customers be willing to pay for the additional costs incurred by that inconvenience? I think there are a whole lot of different things to try before you start introducing different routers with different os's/quirks/capabilities into the mix. Frankly that sounds like a recipe for not just inconvenience but chaos.

Re: Today's Outage Post Mortem

#15

Post by rdl »

dododo wrote: post #9

if you want to build a reliable system, one useful thing to do is use equipment from multiple vendors. sure it's inconvenient, but by doing this you can often de-correlate failures. especially if you want to improve someone else's reliability. e.g., from simple things like hard drives in a raid from different vendors, to n-version programming in safety critical systems (like airplanes).

That works when the interfaces are totally standard, but edge/core routers are not like that. Cisco supports one set of protocols for talking to other Cisco products; another set for talking to everything else. The "everything else" protocols suck in a lot of ways (they're ok inter-site, but not really so great intra-site).

Same with Juniper. (there aren't really other viable options besides those two)

You could build the same site fully independently with all-Cisco on one, and all Juniper on another, and potentially get some better isolation from vendor faults, but at very high expense.

You end up with much worse reliability if you have a mixed Cisco/Juniper network without a lot of additional isolation otherwise.

Re: Today's Outage Post Mortem

#17

Post by packetbeats »

senthilnayagam wrote: post #8

So as far the DOS attack was very successful. It took down the site which it intended to and take down the network with lots and lots of the sites. Hope lessons are learnt and your next generation is less prone to these attacks

If the attacker knew about the Juniper bug and thought about a way to convince the network operators to introduce the rule of death themselves, then this is a nice hack indeed. It won't be easy for CloudFare to generically protect against these types of attacks. They could either have mechanisms to revert configurations faster or a way to test new configurations on a single router.

Re: Today's Outage Post Mortem

#18

Post by opendomain »

packetbeats wrote: post #17
senthilnayagam wrote: post #8

So as far the DOS attack was very successful. It took down the site which it intended to and take down the network with lots and lots of the sites. Hope lessons are learnt and your next generation is less prone to these attacks

If the attacker knew about the Juniper bug and thought about a way to convince the network operators to introduce the rule of death themselves, then this is a nice hack indeed. It won't be easy for CloudFare to generically protect against these types of attacks. They could either have mechanisms to revert configurations faster or a way to test new configurations on a single router.

This is exactly my analysis. Why would the packet the attacker using be so large? The only logical reason is that they knew the Juniper bug. I am wondering if Juniper also knew this bug but did not disclose it.

Re: Today's Outage Post Mortem

#19

Post by rurounijones »

packetbeats wrote: post #17
senthilnayagam wrote: post #8

So as far the DOS attack was very successful. It took down the site which it intended to and take down the network with lots and lots of the sites. Hope lessons are learnt and your next generation is less prone to these attacks

If the attacker knew about the Juniper bug and thought about a way to convince the network operators to introduce the rule of death themselves, then this is a nice hack indeed. It won't be easy for CloudFare to generically protect against these types of attacks. They could either have mechanisms to revert configurations faster or a way to test new configurations on a single router.

The idea that the attacker knew about the bug is, I think, a remote but intriguing idea.

Wonder if Cloudflare need to do some tests along the lines of:

A) List up all the types of rules we usually use to mitigate these situations. B) Run those rules on a test router with wildly unusual input values, as was the case in this situation. C) Send test traffic using that wildly unexpected input to see what happens.

Basically a bit of manual fuzzing

Time-consuming and maybe not worthwhile, but it could save against another full system death.

Re: Today's Outage Post Mortem

#20

Post by windexh8er »

rdl wrote: post #15
dododo wrote: post #9

if you want to build a reliable system, one useful thing to do is use equipment from multiple vendors. sure it's inconvenient, but by doing this you can often de-correlate failures. especially if you want to improve someone else's reliability. e.g., from simple things like hard drives in a raid from different vendors, to n-version programming in safety critical systems (like airplanes).

That works when the interfaces are totally standard, but edge/core routers are not like that. Cisco supports one set of protocols for talking to other Cisco products; another set for talking to everything else. The "everything else" protocols suck in a lot of ways (they're ok inter-site, but not really so great intra-site). Same with Juniper. (there aren't really other viable options besides those two) You could buil…

Re: "there aren't really any viable options..."

Total misconception. BGP, OSPF, ISIS, LISP, etc. are all non proprietary. Sure, the root cause of this particular problem is that CF is using something specific to Juniper, however router interoperability is not predicated on components like that. This example was a tool CF operationalized, and likely had little to do with their routing with the exception of it being a metric they may have influenced routes with.

People who have all Cisco or all Juniper shops namely do it from a cost perspective. Sure, there are some reasons outside of that but it's likely the big driver. The more you buy, the more you save. And the network sales realm is royally messed up to begin with. I've seen Juniper give 90% discounts on hardware just to break into a Cisco shop. But, the reality of the situation is that all of this gear is marked up well into the thousands of percent. So if you're not getting, minimally 50% then your probably not doing yourself due diligence.

Post reply on HN