Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Today's Outage Post Mortem

blog.cloudflare.com

Post reply on HN
Print view View original

111–120 of 159 posts

Re: Today's Outage Post Mortem

#111

Post by cbsmith »

eastdakota wrote: post #72
GigabyteCoin wrote: post #61

Earlier quoted context omitted.

Is junpier to blame for the bug in their OS? Or is cloudflare to blame for not testing JunOS enough before relying on that OS?

Buck stops with us. We choose the hardware and software that runs on our network. We test and work around thousands of bugs in it. It was up to us to check range limits before applying them. While we'll never be perfect, one of the things I am most proud of with the CloudFlare team is how quickly we do learn from mistakes.

There is more to this story than meets the eye. This had to be an IPv6 fragment attack. Why weren't you already advertising rejection of such packets, at least for DNS? Why would your analysis software and procedures not already be checking for memory problems with rules that would need yo assemble all the fragments before matching? Seems like there is more to this story than meets the eye.

Re: Today's Outage Post Mortem

#112

Post by 23david »

tlrobinson wrote: post #79
powertower wrote: post #28

> CloudFlare currently runs 23 data centers worldwide. Shouldn't that always say - CloudFlare currently runs in 23 data centers worldwide? Or is that just how one would phrase that if you rent multiple racks or a cage in a datacenter? ...because I've seen that a bunch of times before from just about everyone. Just curious.

The distinction is a bit arbitrary. As a customer you should care that their service is geographically distributed, not whether they own the buildings where the servers are kept.

Gotta disagree. Owning a datacenter doesn't mean that they own the building.

The difference between renting space in a datacenter versus running an entire datacenter is very big, and has ramifications for their uptime, security of their data and disaster recovery. Not sure why they aren't clearer about this.

Re: Today's Outage Post Mortem

#113

Post by scoot »

graycat wrote: post #109
scoot wrote: post #102

Earlier quoted context omitted.

'in the words of the HAL 9000, "human error".' Except that it wasn't human error, at least not in the sense that the decision to enter the rule, or the rule itself was in error. The human error was with the bug in the Juniper firmware that caused this rule to crash this router, and arguably with the CloudFlare process that allows rules to be propagated to all routers concurrently, rather than segmenting the network a…

"or the rule itself was in error" I thought that the rule was in error: I couldn't read the rule clearly on the screen, but it seemed, or I guessed, that the problem with the rule was that the "humans" omitted the decimal points and, thus, asked for blocking packets with lengths 1000 larger than intended. The Juniper software got sick, i.e., allocated too much memory, only because it was trying to swallow working wit…

The rule matched the output of the profiler, so in that sense was correct. It wasn't clear from the article whether the profiler output was correct, a result of intentionally malformed packets, or otherwise.

Regardless, the Juniper should either have rejected the rule, or accommodated it.

Re: Today's Outage Post Mortem

#114

Post by kisielk »

scoot wrote: post #102
graycat wrote: post #90

Yes, case number 384,449,194 of systems management causing a system problem. Also case number 439,224 of what looked like a localized problem quickly causing a huge system, e.g., all 23 data centers around the world, to crash. They have my sympathy: So, they typed in a 'rule'. At one time I was working in 'artificial intelligence' (AI), actually 'expert systems', based on using 'rules' to implement real time manageme…

'in the words of the HAL 9000, "human error".' Except that it wasn't human error, at least not in the sense that the decision to enter the rule, or the rule itself was in error. The human error was with the bug in the Juniper firmware that caused this rule to crash this router, and arguably with the CloudFlare process that allows rules to be propagated to all routers concurrently, rather than segmenting the network a…

Actually this outage report is a good example of compounding systematic errors. A among the things that went wrong: incorrect and impossible packet sizes were detected, the rule generator generated rules matching the impossible packet sizes, the human operator who looked at the rules and entered them in to the router didn't notice any problems, and finally the routers responded to the incorrect rules by starting to crash.

Had any of these steps not gone wrong there likely would not have been an outage. It was a combination of failures that caused it.

Re: Today's Outage Post Mortem

#115

Post by imbriaco »

scoot wrote: post #113
graycat wrote: post #109

Earlier quoted context omitted.

"or the rule itself was in error" I thought that the rule was in error: I couldn't read the rule clearly on the screen, but it seemed, or I guessed, that the problem with the rule was that the "humans" omitted the decimal points and, thus, asked for blocking packets with lengths 1000 larger than intended. The Juniper software got sick, i.e., allocated too much memory, only because it was trying to swallow working wit…

The rule matched the output of the profiler, so in that sense was correct. It wasn't clear from the article whether the profiler output was correct, a result of intentionally malformed packets, or otherwise. Regardless, the Juniper should either have rejected the rule, or accommodated it.

It's pretty clear that it was a failure of CloudFlare's profiler in generating an obviously impossible rule, the engineer who attempted to apply that rule, and Juniper for allowing it. There's no one place to lay blame, nor is it relevant.

I would imagine that while we have seen a public reason for outage statement that there is a lot more work going on inside CloudFlare as far as the post-mortem is concerned. There are a lot of angles to cover here to really understand how the _system_ failed.

Re: Today's Outage Post Mortem

#116

Post by david_shaw »

TazeTSchnitzel wrote: post #67
larrys wrote: post #42

Earlier quoted context omitted.

"The small team" Didn't see this when I posted my comment. So there is a team monitoring not a single person?

Uptime is a fundamental part of their business, of course it's a team.

> Uptime is a fundamental part of their business, of course it's a team

Many businesses call roles a "team," even when it's just one person. I believe this is the case here, since the linked CloudFlare blog post states:

> Someone from our operations team is monitoring our network 24/7.

Re: Today's Outage Post Mortem

#117

Post by scoot »

kisielk wrote: post #114
scoot wrote: post #102

Earlier quoted context omitted.

'in the words of the HAL 9000, "human error".' Except that it wasn't human error, at least not in the sense that the decision to enter the rule, or the rule itself was in error. The human error was with the bug in the Juniper firmware that caused this rule to crash this router, and arguably with the CloudFlare process that allows rules to be propagated to all routers concurrently, rather than segmenting the network a…

Actually this outage report is a good example of compounding systematic errors. A among the things that went wrong: incorrect and impossible packet sizes were detected, the rule generator generated rules matching the impossible packet sizes, the human operator who looked at the rules and entered them in to the router didn't notice any problems, and finally the routers responded to the incorrect rules by starting to c…

I don't disagree with you, but I was calling in to question the suggestion that the creation of the rule was the specific human error. By definition, every error you listed is a human error, as even if ultimately carried out but computers (routers), they were designed by humans.

Re: Today's Outage Post Mortem

#118

Post by DigitalSea »

Rather unfortunate for the credibility of Cloudfare as a network provider, but you've got to admire them for their honesty and it'll work out better for them in the end. It's amazing how a few lines of code managed to bring down Cloudfare, they could have told us anything and nobody would have been able to question it; instead they gave us the truth and I really respect that. They didn't blame the intern, they didn't blame their hardware or make an excuse about a power outage. In terms of honesty Cloudfare seems to be leading the way regardless of their public credibility or image being tainted. Very impressive response time and resolution of the issue as well, good job Cloudfare!

Re: Today's Outage Post Mortem

#119

Post by saurik »

driverdan wrote: post #55
jaequery wrote: post #46

this is the type of reason why i stopped using cloudflare. there are just too many eggs in one basket. it's as if their entire service becomes a SPOF to your infrastructure.

You could say the same thing about almost any of your service providers. Your DNS provider goes out, everything goes out. The routers at the data center with your servers go out, all your servers go out. Your CDN goes out, all of the static assets on your site go out. There will always be potential SPOF.

While your example with the routers at your backend is truly problematic, DNS is designed with built-in redundancy and CDNs (which CloudFlare should not really count as) having a world-wide outage (as opposed to "people accessing from New York are currently having issues, as we lost one PoP") is nigh-unto unheard of... can you imagine Akamai (or CDNetworks or EdgeCast or even Amazon) saying "doh, all of our infrastructure everywhere just disappeared"?

The core problem with CloudFlare is that they seem to have a highly-centralized take on what is normally a massively-decentralized solution-space, with large numbers of value-adds they encourage customers to use without making it clear that they treat in a haphazard manner, doing very little testing before deploying pushing-the-envelope features while simultaneously having very little in-house debugging expertise to handle serious issues.

(As a concrete example of that last complaint, Cydia was crippled for an entire day due to ModMyi turning on CloudFlare's "preloader" transformation, which apparently caused many WebKit-based browsers--including both MobileSafari and Cydia--to entirely lock up; CloudFlare seemed to go the entire day without noticing, which I continue to be utterly shocked by, and it was only after I told them how to fix it that they were able to acknowledge the issue.)

http://www.saurik.com/id/14 <- When "Dumb Pipes" Get Too Smart, an extensive analysis of this bug

Re: Today's Outage Post Mortem

#120

Post by DigitalSea »

larrys wrote: post #41
ryguytilidie wrote: post #22

This is pretty impressive. Keep in mind most of the team is on the west coast so this happened at 1am on a Sunday and they put up a post mortem within hours. Obviously you would prefer it not happen at all, but that is a great response imo.

But this is not impressive: "Someone from our operations team is monitoring our network 24/7." "Someone" seems to indicate "1 person". Not "people are monitoring" but "someone". That's it, one person monitors the network? Like the single night guard at the warehouse?

That is a little harsh. What makes you think it's only one person monitoring the entire network? And even if it is, what's the problem with that as long as they do their job correctly?
Post reply on HN