Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

More details about the October 4 outage

engineering.fb.com

Post reply on HN
Print view View original

111–120 of 306 posts

Re: More details about the October 4 outage

#111

Post by yuliyp »

antocv wrote: post #66
jimmyvalmer wrote: post #56

tldr; a maintenance query was issued that inexplicably severed FB's data centers from the internet, which unnecessarily caused their DNS servers to mark themselves defunct, which made it all but impossible for their guys to repair the problem from HQ, which compelled them to physically dispatch field units whose progress was stymied by recent increased physical security measures.

> caused their DNS servers to mark themselves defunct This is awkward for me too, why should a DNS server withdraw BGP routes? Design fail.

It's a trade-off.

Imagine you have some DNS servers at a POP. They're connected to a peering router there which is connected to a bunch of ISPs. The POP is connected via a couple independent fiber links to the rest of your network. What happens if both of those links fail?

Ideally the rest of your service can detect that this POP is disconnected, and adjust DNS configuration to point users toward POPs which are not disconnected. But you still have that DNS server which can't see that config change (since it's disconnected from the rest of your network) but still reachable from a bunch of local ISPs. That DNS server will continue to direct traffic to the POP which can't handle it.

What if that DNS server were to mark itself unavailable? In that case, DNS traffic from ISPs near that POP would instead find another DNS server from a different POP, and get a response which pointed toward some working POP instead. How would the DNS server mark itself unavailable? One way is to see if it stopped being able to communicate with the source of truth.

Yesterday all of the DNS servers stopped being able to communicate with the source of truth, so marked themselves offline. This code assumes a network partition, so can't really rely on consensus to decide what to do.

Re: More details about the October 4 outage

#112

Post by flutas »

Barrin92 wrote: post #24
HenryKissinger wrote: post #5

> During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally Imagine being this person. Tomorrow on /r/tifu.

of course if one person can knock down an entire global system through a trivial mistake the problem is obviously not the person to begin with, but the architecture of the system.

Or the fact that there was a bug in the tool that should have prevented this.

> Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command.

Re: More details about the October 4 outage

#113

Post by packetslave »

detaro wrote: post #91
packetslave wrote: post #72

Earlier quoted context omitted.

"Okay, we'll pull in a DSL line from a completely separate ISP for the out-of-band access." (guess what else is in that manhole/conduit?) "Okay, we'll use LTE for out-of-band!" (oops, the backhaul for the cell tower goes under the same bridge as the real network) True diversity is HARD (not unsolvable, just hard. especially at scale)!

Although the difference here is that loosing connection and out-of-band for a single data center shouldn't be as catastrophic for Facebook, so your examples would be tolerable?

That's the trick, though: if you don't do that level of planning for all of your datacenters and POPs (and fiber huts out in the middle of nowhere), it's inevitable that the one you most need to access during an outage will be the one where your OOB got backhoe'd.

Murphy is a jerk.

Re: More details about the October 4 outage

#114

Post by nathanyz »

jeffbee wrote: post #55
vitus wrote: post #34

Earlier quoted context omitted.

To expand on why this made me think of the Google outage: It was a global backbone isolation, caused by configuration changes (as they all are...). It was detected fairly early on, but recovery was difficult because internal tools / debugging workflows were also impacted, and even after the problem was identified, it still took time to back out the change. "But wait, a global backbone isolation? Google wasn't totally…

Google also had a runaway automation outage where a process went around the world "selling" all the frontend machines back to the global resource pool. Nobody was alerted until something like 95% of global frontends had disappeared. This was an important lesson for SREs inside and outside Google because it shows the dangers of the antipattern of command line flags that narrow the scope of an operation instead of expa…

I feel like this explains so much about why the gcloud command works the way it does. Sometimes feels overly complicated for minor things, but given this logic, I get it.

Re: More details about the October 4 outage

#115

Post by ghostoftiber »

Incidentally the facebook app itself really handled this gracefully. When the app can't connect to facebook, it displays "updates" from a pool of cached content. It looks and feels like facebook is there, but we know it's not. I didn't notice this until the outage and I thought it was neat.

Re: More details about the October 4 outage

#116

Post by cranekam »

pm2222 wrote: post #71

I'm confused by this. Are the DNS servers inside the backbone, or outside? To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection. In the recent outage the entire backbone was removed from operation, making these locations declare themselves unhealthy and withdraw those BGP adve…

The wording is a bit unclear (rushed, no doubt) but I expect this means the DNS servers stopped announcing themselves as possible targets for the anycasted IPs Facebook uses for its authoritative DNS [1], since they learned that the network was deemed unhealthy. If they all do that nobody will answer traffic sent to the authoritative DNS IPs and nothing works.

[1] See "our authoritative name servers that occupy well known IP addresses themselves" mentioned earlier

Re: More details about the October 4 outage

#117

Post by cnst »

nonbirithm wrote: post #46

DNS seems to be a massive point of failure everywhere, even taking out the tools to help deal with outages themselves. The same thing happened to Azure multiple times in the past, causing complete service outages. Surely there must be some way to better mitigate DNS misconfiguration by now, given the exceptional importance of DNS?

But DNS didn't actually fail. Their design says DNS must go offline if the rest of the network is offline. That's exactly what DNS did.

Sounds like their design was wrong, but you can't just blame DNS. DNS worked 100% here as per the task that it was given.

> To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection.

Re: More details about the October 4 outage

#118

Post by FridayoLeary »

rblion wrote: post #30

Was this a way to delete a lot of evidence before shit really hit the fan?' After reading this, I can't help but feel this was a calculated move. It gives FB a chance to hijack media attention from the whistleblower. It gives them a chance to show the average peson, 'hey, we make mistakes and we have a review process to improve our systems'. The timing is too perfect if you ask me.

Inviting further whistleblowing.

Re: More details about the October 4 outage

#119

Post by bryan_w »

nonbirithm wrote: post #46

DNS seems to be a massive point of failure everywhere, even taking out the tools to help deal with outages themselves. The same thing happened to Azure multiple times in the past, causing complete service outages. Surely there must be some way to better mitigate DNS misconfiguration by now, given the exceptional importance of DNS?

DNS was very much a proximate cause. In most cases you want your anycast dns servers to shoot themselves in the head if they detect their connection to origin to be interrupted. This would have been an big outage anyways just at a different layer.

Oddly enough, one could consider that behavior something that was put in place to "mitigate DNS misconfiguration"

Re: More details about the October 4 outage

#120

Post by jablan »

gist wrote: post #99
Hokusai wrote: post #9

> One of the jobs performed by our smaller facilities is to respond to DNS queries. DNS is the address book of the internet, enabling the simple web names we type into browsers to be translated into specific server IP addresses. What is the target audience of this post? It is too technical for non-technical people, but also it is dumbed down to try to include people that does not know how the internet works. I feel l…

> What is the target audience of this post? Separate point to your question. FB is under no obligation to provide more details than they need to because a small segment of the population (certainly relative to their 'customers') might find it interesting or helpful or entertaining. FB is a business. They can essentially do (and should be able to) do whatever they want. There is no requirement (and there should be no…

> If I go to the store and there is not paper towels there I don't need to know why there are no towels

You don't _need_ to know, but it's human to want to know, and it's also human to want to satisfy other human's curiosity, especially if it doesn't bring any harm to you.

Also, your post is not really answering any of GP's questions. I presume you wanted to say that FB doesn't _owe_ any explanation to us, but the GP asked, as they already provided one, to whom is it addressed.

Post reply on HN