More details about the October 4 outage • Hacker News Board

Re: More details about the October 4 outage

#121

Post by Hokusai » Tue, Oct 05, 2021, 7:12 PM UTC

gist wrote: post #99

Hokusai wrote: post #9

> One of the jobs performed by our smaller facilities is to respond to DNS queries. DNS is the address book of the internet, enabling the simple web names we type into browsers to be translated into specific server IP addresses. What is the target audience of this post? It is too technical for non-technical people, but also it is dumbed down to try to include people that does not know how the internet works. I feel l…

> What is the target audience of this post? Separate point to your question. FB is under no obligation to provide more details than they need to because a small segment of the population (certainly relative to their 'customers') might find it interesting or helpful or entertaining. FB is a business. They can essentially do (and should be able to) do whatever they want. There is no requirement (and there should be no…

The air industry has this solved, it's mandatory to report certain kind of incidents to avoid them in the future and inform the aviation community. https://www.skybrary.aero/index.php/Mandatory_Occurrence_Rep...

That the main form of personal communication for hundreds of millions of users is down and there is no mandatory reporting is irresponsible. That Facebook is a business does not mean that they do not have responsibilities towards society.

Facebook is not your local supermarket, it has global impact.

Re: More details about the October 4 outage

#122

Post by formerly_proven » Tue, Oct 05, 2021, 7:13 PM UTC

narrator wrote: post #90

tantalor wrote: post #49

Earlier quoted context omitted.

But the FB outage was not a configuration change. > a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network

This is kind of like Chernobyl where they were testing to see how hot they could run the reactor to see how much power it could generate. Then things went sideways.

The Chernobyl test was not a test to drive the reactor to the limits, but actually a test to verify that the inertia of the main turbines is big enough to drive the coolant pumps for X amount of time in the case of grid failure.

Re: More details about the October 4 outage

#123

Post by divbzero » Tue, Oct 05, 2021, 7:14 PM UTC

ghostoftiber wrote: post #115

Incidentally the facebook app itself really handled this gracefully. When the app can't connect to facebook, it displays "updates" from a pool of cached content. It looks and feels like facebook is there, but we know it's not. I didn't notice this until the outage and I thought it was neat.

I’m curious if the app handled posts or likes gracefully too. Did it accept and cache the updates until it could reconnect to Facebook servers?

Re: More details about the October 4 outage

#124

Post by kube-system » Tue, Oct 05, 2021, 7:14 PM UTC

Johnny555 wrote: post #50

kube-system wrote: post #8

Earlier quoted context omitted.

Likely tripping breakers or overload protection on UPSes? Often PDUs used in a rack can be configured to start servers up in a staggered pattern to avoid a surge in demand for these reasons. I'd imagine there's more complications when you're doing an entire DC vs just a single rack, though.

I don't see how suddenly running more traffic is going to trip datacenter breakers -- I could see how flipping on power to an entire datacenter's worth of servers could cause a spike in electrical demand that the power infrastructure can't handle, but if suddently running CPU's at 100% trips breakers, then it seems like that power infrastructure is undersized? This isn't a case where servers were powered off, they we…

I don't know the answer. But it's not too uncommon, in general, to provision for reasonable use cases plus a margin, rather than provision for worst case scenario.

Re: More details about the October 4 outage

#125

Post by toast0 » Tue, Oct 05, 2021, 7:14 PM UTC

pm2222 wrote: post #71

I'm confused by this. Are the DNS servers inside the backbone, or outside? To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection. In the recent outage the entire backbone was removed from operation, making these locations declare themselves unhealthy and withdraw those BGP adve…

If we make a simplified network map, FB looks more or less like a bunch of PoPs (points of presence) at major peering points around the world, a backbone network that connects those PoPs to the FB datacenters, and the FB operated datacenters themselves. (The datacenters are generally located a bit farther away from population centers, and therefore peering points, so it's sensible to communicate to the outside world through the PoPs only)

The DNS servers run at the PoPs, but only BGP advertise the DNS addresses when the PoP determines it's healthy. If there's no connectivity back to a FB datacenter (or perhaps, no connectivity to the preferred datacenter), the PoP is unhealthy and won't advertise the DNS addresses over BGP.

Since the BGP change that was pushed eliminated the backbone connectivity, none of the PoPs were able to connect to datacenters, and so they all, independently, stopped advertising the DNS addresses.

So that's why DNS went down. Of course, since client access goes through load balancers at the PoPs, and the PoPs couldn't access the datacenters where requests are actually processed, DNS being down wasn't a meaningful impediment to accessing the services. Apparently, it was an issue with management (among other issues).

Disclosure: I worked at WhatsApp until 2019, and saw some of the network diagrams. Network design may have changed a bit in the last 2 years, but probably not too much.

Re: More details about the October 4 outage

#126

Post by closeparen » Tue, Oct 05, 2021, 7:15 PM UTC

Ansil849 wrote: post #73

educationcto wrote: post #68

Earlier quoted context omitted.

The original error was the network command, but the slower response and lengthy outage was partially due to the physical security they put in place to prevent malicious activity. Any event like this has multiple root causes.

Yes, but the fact that the blogpost concludes on this relatively tangential note (which notably also conveniently allows Facebook to brag about their security measures) and not on the note that their audit code was apparently itself not sufficiently audited, is what makes this deceptive spin.

Our postmortems have three sections. Prevention, detection, and mitigation. They all matter.

Shit happens. People ship bugs. People fat-finger commands. An engineering team’s responsibility doesn’t stop there. It also needs to quickly activate responders who know what to do and have the tools & access to fix it. Sometimes the conditions that created the issue are within acceptable bounds; the real need for reform is in why it took so long to fix.

Re: More details about the October 4 outage

#127

Post by dfcowell » Tue, Oct 05, 2021, 7:15 PM UTC

Johnny555 wrote: post #50

kube-system wrote: post #8

Earlier quoted context omitted.

Likely tripping breakers or overload protection on UPSes? Often PDUs used in a rack can be configured to start servers up in a staggered pattern to avoid a surge in demand for these reasons. I'd imagine there's more complications when you're doing an entire DC vs just a single rack, though.

I don't see how suddenly running more traffic is going to trip datacenter breakers -- I could see how flipping on power to an entire datacenter's worth of servers could cause a spike in electrical demand that the power infrastructure can't handle, but if suddently running CPU's at 100% trips breakers, then it seems like that power infrastructure is undersized? This isn't a case where servers were powered off, they we…

All of the components in the supply chain will be rated for greater than max load, however power generation at grid scale is a delicate balancing act.

I’m not an electrical engineer, so the details here may be fuzzy, however in broad strokes:

Grid operators constantly monitor power consumption across the grid. If more power is being drawn than generated, line frequency drops across the whole grid. This leads to brownouts and can cause widespread damage to grid equipment and end-user devices.

The main way to manage this is to bring more capacity online to bring the grid frequency back up. This is slow, since spinning up even “fast” generators like natural gas can take on the order of several minutes.

Notably, this kind of scenario is the whole reason the Tesla battery in South Australia exists. It can respond to spikes in demand (and consume surplus supply!) much faster than generator capacity can respond.

The other option is load shedding, where you just disconnect parts of your grid to reduce demand.

Any large consumers (like data center operators) likely work closely with their electricity suppliers to be good citizens and ramp up and down their consumption in a controlled manner to give the supply side (the power generators) time to adjust their supply as the demand changes.

Note that changes to power draw as machines handle different load will also result in changes to consumption in the cooling systems etc. making the total consumption profile substantially different coming from a cold start.

Re: More details about the October 4 outage

#128

Post by cube00 » Tue, Oct 05, 2021, 7:19 PM UTC

> We’ve done extensive work hardening our systems to prevent unauthorized access, and it was interesting to see how that hardening slowed us down as we tried to recover from an outage caused not by malicious activity, but an error of our own making. I believe a tradeoff like this is worth it — greatly increased day-to-day security vs. a slower recovery from a hopefully rare event like this.

If you correctly design your security with appropriate fall backs you don't need to make this trade off.

If that story of the Facebook campus having no physical key holes on doors is true it just speaks to an arrogance of assuming things can never fail so we don't even need to bother planning for it.

Re: More details about the October 4 outage

#129

Post by trevorishere » Tue, Oct 05, 2021, 7:21 PM UTC

formerly_proven wrote: post #122

narrator wrote: post #90

Earlier quoted context omitted.

This is kind of like Chernobyl where they were testing to see how hot they could run the reactor to see how much power it could generate. Then things went sideways.

The Chernobyl test was not a test to drive the reactor to the limits, but actually a test to verify that the inertia of the main turbines is big enough to drive the coolant pumps for X amount of time in the case of grid failure.

Of possible interest:

https://www.youtube.com/watch?v=Ijst4g5KFN0

This is a presentation to students by an MIT professor that goes over exactly what happened, the sequence of events, mistakes made, and so on.

Re: More details about the October 4 outage

#130

Post by divbzero » Tue, Oct 05, 2021, 7:22 PM UTC

chomp wrote: post #15

So someone ran "clear mpls lsp" instead of "show mpls lsp"?

For context, parent comment is trying to decipher this heavily-PR-reviewed paragraph:

> During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally. Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command.