Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

More details about the October 4 outage

engineering.fb.com

Post reply on HN
Print view View original

71–80 of 306 posts

Re: More details about the October 4 outage

#71

Post by pm2222 »

I'm confused by this. Are the DNS servers inside the backbone, or outside?

  To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection. In the recent outage the entire backbone was removed from operation,  making these locations declare themselves unhealthy and withdraw those BGP advertisements. The end result was that our DNS servers became unreachable even though they were still operational. This made it impossible for the rest of the internet to find our servers.

Re: More details about the October 4 outage

#72

Post by packetslave »

vitus wrote: post #48
louwrentius wrote: post #42

> Our primary and out-of-band network access was down Don't create circular dependencies.

With something as fundamental as the network, no way around it. - Okay, we'll set up a separate maintenance network in case we can't get to the regular network. - Wait, but we need a maintenance network for the maintenance network...

"Okay, we'll pull in a DSL line from a completely separate ISP for the out-of-band access." (guess what else is in that manhole/conduit?)

"Okay, we'll use LTE for out-of-band!" (oops, the backhaul for the cell tower goes under the same bridge as the real network)

True diversity is HARD (not unsolvable, just hard. especially at scale)!

Re: More details about the October 4 outage

#73

Post by Ansil849 »

educationcto wrote: post #68
Ansil849 wrote: post #59

> We’ve done extensive work hardening our systems to prevent unauthorized access, and it was interesting to see how that hardening slowed us down as we tried to recover from an outage caused not by malicious activity, but an error of our own making. I believe a tradeoff like this is worth it — greatly increased day-to-day security vs. a slower recovery from a hopefully rare event like this. From here on out, our job…

The original error was the network command, but the slower response and lengthy outage was partially due to the physical security they put in place to prevent malicious activity. Any event like this has multiple root causes.

Yes, but the fact that the blogpost concludes on this relatively tangential note (which notably also conveniently allows Facebook to brag about their security measures) and not on the note that their audit code was apparently itself not sufficiently audited, is what makes this deceptive spin.

Re: More details about the October 4 outage

#74

Post by temp_praneshp »

Ansil849 wrote: post #67
temp_praneshp wrote: post #61

Earlier quoted context omitted.

No one except you is trying to spin anything.

Going on at lengths about how the trade off between prolonged downtime and strict security protocols is a worthy trade off is erecting a nonsensical strawman, the literal definition of spinning a story. The key issue had nothing to do with Facebook's data center security protocols.

No one except you is erecting a nonsensical strawman.

Re: More details about the October 4 outage

#76

Post by nomel »

dmoy wrote: post #4

Interesting bit on recovery w.r.t. the electrical grid > flipping our services back on all at once could potentially cause a new round of crashes due to a surge in traffic. Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems ... I wish there was a bit more detail in here. What'…

If your system is pulling 500 watts at 120V, that's around 4A of line voltage. If you drop down 20% to 100V, the output will happily still pull its regulated voltage, but now the line components are seeing ~20% more, at 5A. For brown out, you need to overrate your components, and/or shut everything off if the line voltage goes too low.

I used to do electrical compliance testing in a previous life, with brown out testing being one of our safety tests. You would drape a piece of cheese cloth over the power supply and slowly ramp the line voltage down. At the time, the power supplies didn't have good line side voltage monitoring. There was almost always smoke, and sometimes cheese cloth fires. Since this was safety testing, pass/fail was mostly based on if the cheese cloth caught fire, not if the power supply was damaged.

Re: More details about the October 4 outage

#78

Post by net4all »

pm2222 wrote: post #71

I'm confused by this. Are the DNS servers inside the backbone, or outside? To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection. In the recent outage the entire backbone was removed from operation, making these locations declare themselves unhealthy and withdraw those BGP adve…

If I understand it correctly they have DNS servers spread out at different locations. These locations are also BGP peering locations. If the DNS server at a location cannot reach the other datacenters via the backbone it stops advertising their IP prefixes. The hope is that traffic will then instead get routed to some other facebook location that is still operational.

Re: More details about the October 4 outage

#79

Post by [deleted] »

pm2222 wrote: post #71

I'm confused by this. Are the DNS servers inside the backbone, or outside? To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection. In the recent outage the entire backbone was removed from operation, making these locations declare themselves unhealthy and withdraw those BGP adve…

[deleted]

Re: More details about the October 4 outage

#80

Post by cratermoon »

Ansil849 wrote: post #59

> We’ve done extensive work hardening our systems to prevent unauthorized access, and it was interesting to see how that hardening slowed us down as we tried to recover from an outage caused not by malicious activity, but an error of our own making. I believe a tradeoff like this is worth it — greatly increased day-to-day security vs. a slower recovery from a hopefully rare event like this. From here on out, our job…

Simple Testing Can Prevent Most Critical Failures[1], "We found the majority of catastrophic failures could easily have been prevented by performing simple testing on error handling code – the last line of defense – even without an understanding of the software design."

1 https://www.eecg.utoronto.ca/~yuan/papers/failure_analysis_o...

Post reply on HN