Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

More details about the October 4 outage

engineering.fb.com

Post reply on HN
Print view View original

151–160 of 306 posts

Re: More details about the October 4 outage

#151

Post by willcipriano »

fabian2k wrote: post #134
narrator wrote: post #90

Earlier quoted context omitted.

This is kind of like Chernobyl where they were testing to see how hot they could run the reactor to see how much power it could generate. Then things went sideways.

As already said the test was about something entirely different. And the dangerous part was not the test itself, but the way they delayed the test and then continued to perform it despite the reactor being in a problematic state and the night shift being on duty, who were not trained on this test. The main problem was that they ran the reactor at reduced power long enough to have significant xenon poisoning, and then…

I'd say the failure at Chernobyl was that anyone who asked questions got sent to a labor camp and the people making the decisions really had no clue about the work being done. Everything else just stems from that. The safest reactor in the world would blow up under the same leadership.

Re: More details about the October 4 outage

#152

Post by pm2222 »

toast0 wrote: post #125
pm2222 wrote: post #71

I'm confused by this. Are the DNS servers inside the backbone, or outside? To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection. In the recent outage the entire backbone was removed from operation, making these locations declare themselves unhealthy and withdraw those BGP adve…

If we make a simplified network map, FB looks more or less like a bunch of PoPs (points of presence) at major peering points around the world, a backbone network that connects those PoPs to the FB datacenters, and the FB operated datacenters themselves. (The datacenters are generally located a bit farther away from population centers, and therefore peering points, so it's sensible to communicate to the outside world…

Ok so the DNS servers at PoPs, outside of backbone, did not go down.

Does it mean they can respond with public IPs meaningful for local PoP only and are not able to respond with IPs as directions to other PoPs or FB's main DCs? So that has to mean different public IPs are handed out at different PoPs, right?

Re: More details about the October 4 outage

#153

Post by miyuru »

antocv wrote: post #66
jimmyvalmer wrote: post #56

tldr; a maintenance query was issued that inexplicably severed FB's data centers from the internet, which unnecessarily caused their DNS servers to mark themselves defunct, which made it all but impossible for their guys to repair the problem from HQ, which compelled them to physically dispatch field units whose progress was stymied by recent increased physical security measures.

> caused their DNS servers to mark themselves defunct This is awkward for me too, why should a DNS server withdraw BGP routes? Design fail.

Most of the large DNS services are anycasted via BGP. (All POPs announce the same IP prefix) It makes sense to stop the BGP routing if the POP is unhealthy. Traffic will flow to the next healthy POP.

In this case if the DNS sevice in the POP is unhealthy and IP address belonging to the DNS service are removed from the POP.

Re: More details about the October 4 outage

#154

Post by spaetzleesser »

vitus wrote: post #18

Google had a comparable outage several years ago. https://status.cloud.google.com/incident/cloud-networking/19... This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust). I'd be curious to see what systemic changes happen at F…

“ mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust “

The same leadership that demanded tighter and tighter deadlines and discouraged thinking things through?

Re: More details about the October 4 outage

#155

Post by chronid »

antocv wrote: post #66
jimmyvalmer wrote: post #56

tldr; a maintenance query was issued that inexplicably severed FB's data centers from the internet, which unnecessarily caused their DNS servers to mark themselves defunct, which made it all but impossible for their guys to repair the problem from HQ, which compelled them to physically dispatch field units whose progress was stymied by recent increased physical security measures.

> caused their DNS servers to mark themselves defunct This is awkward for me too, why should a DNS server withdraw BGP routes? Design fail.

Note those are anycast addresses, my guess is the DNS server gives out addresses for FB names pointing your traffic to the POP the DNS server is part of.

If the POP is not able to connect to the rest of Facebook's network, the POP stops announcing itself as available and that DNS and part of the network goes away so your traffic can go somewhere else.

Re: More details about the October 4 outage

#156

Post by spaetzleesser »

silicon2401 wrote: post #139
vitus wrote: post #18

Google had a comparable outage several years ago. https://status.cloud.google.com/incident/cloud-networking/19... This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust). I'd be curious to see what systemic changes happen at F…

> leadership read out emails from customers telling us how we let them down and lost their trust). That's amazing. I would never have expected my feedback to a company to actually be read, let alone taken seriously. Hopefully more companies do this than I thought.

From my experience this is more done to make leadership feel better and deflect blame from their leadership.

Re: More details about the October 4 outage

#157

Post by topspin »

kube-system wrote: post #8
dmoy wrote: post #4

Interesting bit on recovery w.r.t. the electrical grid > flipping our services back on all at once could potentially cause a new round of crashes due to a surge in traffic. Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems ... I wish there was a bit more detail in here. What'…

Likely tripping breakers or overload protection on UPSes? Often PDUs used in a rack can be configured to start servers up in a staggered pattern to avoid a surge in demand for these reasons. I'd imagine there's more complications when you're doing an entire DC vs just a single rack, though.

Disk arrays have been staggering drive startup for a long time for this reason. Sinking current into hundreds of little starting motors simultaneously is a bad idea.

Re: More details about the October 4 outage

#158

Post by UncleMeat »

agwa wrote: post #150
vitus wrote: post #18

Google had a comparable outage several years ago. https://status.cloud.google.com/incident/cloud-networking/19... This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust). I'd be curious to see what systemic changes happen at F…

> a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust Is that normal at Google? Making people feel bad for an outage doesn't seem consistent with the "blameless postmortem" culture promoted in the SRE book[1]. [1] https://sre.google/sre-book/postmortem-culture/

"Blameless Postmortem" does not mean "No Consequences", even if people often want to interpret it that way. If an organization determines that a disconnect between ground work and a customer's experience is a contributing factor to poor decision making then they might conclude that making engineers more emotionally invested in their customers could be a viable path forward.

Re: More details about the October 4 outage

#159

Post by ro_bit »

AdamHominem wrote: post #138
vitus wrote: post #18

Google had a comparable outage several years ago. https://status.cloud.google.com/incident/cloud-networking/19... This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust). I'd be curious to see what systemic changes happen at F…

> This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust). Bullshit. I'd believe this if it was not completely impossible for 99.999999% of google "customers" to contact anyone at the company. Or for the decade and a half of p…

> You googlers can't even disdain yourselves to talk to other workers at the company who are in a caste lower than you.

We must know different googlers then. It's good to avoid painting a group with the same brush

Re: More details about the October 4 outage

#160

Post by colechristensen »

Ansil849 wrote: post #41

> The backbone is the network Facebook has built to connect all our computing facilities together, which consists of tens of thousands of miles of fiber-optic cables crossing the globe and linking all our data centers. This makes it sound like Facebook has physically laid "tens of thousands of miles of fiber-optic cables crossing the globe and linking all our data centers". Is this in fact true?

Likely a mixture of bought, leased, and self laid fiber. This is not at all uncommon and basically necessary if you have your own data center.
Post reply on HN