Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

More details about the October 4 outage

engineering.fb.com

Post reply on HN
Print view View original

11–20 of 306 posts

Re: More details about the October 4 outage

#11

Post by mschuster91 »

dmoy wrote: post #4

Interesting bit on recovery w.r.t. the electrical grid > flipping our services back on all at once could potentially cause a new round of crashes due to a surge in traffic. Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems ... I wish there was a bit more detail in here. What'…

One case is automated protection systems in the grid detecting a sudden hop of current and assuming an isolation failure along the path - basically, not enough current to trip the short-circuit breakers, but enough to raise an alarm.

Re: More details about the October 4 outage

#13

Post by jffry »

rvnx wrote: post #3

It's a no apologies messages: "We failed, our processes failed, our recovery process only partially worked, we celebrate failure. Our investors were not happy, our users were not happy, some people probably ended in physically dangerous situations due to WhatsApp being unavailable but it's ok. We believe a tradeoff like this is worth it." - Your engineering team.

Yesterday's blog post (discussion here: https://news.ycombinator.com/item?id=28754824) was a direct apology.

Should that be repeated in a somewhat more technical discussion of why it happened?

Re: More details about the October 4 outage

#14

Post by secguyperson »

> a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally

From a security perspective, I’m blown away that a single person apparently had the technical permissions to do such a thing. I can’t think of any valid reason that a single person would have the ability to disconnect every single data center globally. The fact that such functionality exists seems like a massive foot-gun.

At a minimum I would expect multiple layers of approval, or perhaps regionalized permissions, so that even if this person did run an incorrect command, the system turns around and says “ok we’ll shut down the US data centers but you’re not allowed to issue this command for the EU data centers, so those stay up”.

Re: More details about the October 4 outage

#16

Post by thirtyseven »

Hokusai wrote: post #9

> One of the jobs performed by our smaller facilities is to respond to DNS queries. DNS is the address book of the internet, enabling the simple web names we type into browsers to be translated into specific server IP addresses. What is the target audience of this post? It is too technical for non-technical people, but also it is dumbed down to try to include people that does not know how the internet works. I feel l…

With an outage this big, even a post for a technical audience will get read by non-technical people (including journalists), so I'm sure it helps to include details like this.

Re: More details about the October 4 outage

#17

Post by plorg »

dmoy wrote: post #4

Interesting bit on recovery w.r.t. the electrical grid > flipping our services back on all at once could potentially cause a new round of crashes due to a surge in traffic. Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems ... I wish there was a bit more detail in here. What'…

Brownouts is probably the most proximate concern - a sudden increase in demand will draw down the system frequency in the vicinity, and if there aren't generation units close enough or with enough dispatchable capacity there's a small chance they would trip a protective breaker.

A person I know on the power grid side said at one data center there were step functions when FB went down and then when it came up, equal to about 20% of the load behind the distribution transformer. That quantity is about as much as an aluminum smelter switching on or off.

Re: More details about the October 4 outage

#18

Post by vitus »

Google had a comparable outage several years ago.

https://status.cloud.google.com/incident/cloud-networking/19...

This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust).

I'd be curious to see what systemic changes happen at FB as a result, if any.

Re: More details about the October 4 outage

#19

Post by shadofx »

Hokusai wrote: post #9

> One of the jobs performed by our smaller facilities is to respond to DNS queries. DNS is the address book of the internet, enabling the simple web names we type into browsers to be translated into specific server IP addresses. What is the target audience of this post? It is too technical for non-technical people, but also it is dumbed down to try to include people that does not know how the internet works. I feel l…

Teenagers who are responsible for managing the family router?
Post reply on HN