Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Update about the October 4th outage

engineering.fb.com

Post reply on HN
Print view View original

221–230 of 239 posts

Re: Update about the October 4th outage

#221

Post by dodobirdlord »

pbhjpbhj wrote: post #125
toast0 wrote: post #60

Earlier quoted context omitted.

The didn't revoke all their routes, FWIW, just a lot of them (including the anycast DNS routes)

What I don't understand is why, when a route is revoked, if there is no other route announced the routing table gets updated? It seems like either it's a black hole or it still works and there was a BGP error (or the route works but the resources aren't present, so traffic would be dropped). What's the reason for designing the system to revoke routes when no new route is announced? It strikes me it's like DNS when yo…

Withdrawing the last route covering an IP range is a legitimate change, indicating that those IP addresses are no longer in use by the ASN. This needs to be supported so that one ASN can withdraw an IP block and transfer it to another ASN.

Re: Update about the October 4th outage

#222

Post by matsur »

EricE wrote: post #212
yuliyp wrote: post #131

Earlier quoted context omitted.

It's less vague than you realize. It points out that the problem was within Facebook's network between its datacenters. This not only suggests it's related to express backbone, but also suggests that the DNS BGP withdrawal which Cloudflare observed was not the primary issue. It's not a full root cause analysis, to be sure, and leaves many open questions, but I definitely wouldn't describe it as painfully vague.

A point of distinction - there is no "DNS BGP withdrawl". DNS is related to BGP only that without the right BGP routes in the routers, no packets can get to the facebook networks and thus the facbook DNS servers. That their DNS servers were taken out was a side affect of the root issue - they withdrew all the routes to their networks from the rest of the Internet. Not picking on you - but there has been a lot of conf…

This is not what we observed. Less specific routes were still present in routing tables, but the more specifics that routed DNS traffic disappeared.

Re: Update about the October 4th outage

#223

Post by toast0 »

can16358p wrote: post #170
toast0 wrote: post #102

Earlier quoted context omitted.

For a status page to be actually independent, it needs to have all it's requirements hosted on other infrastructure. fb.com authoritative DNS is the same as facebook.com, so it's going down when (FB) DNS goes down (and DNS is going down when BGP is broken, apparently). It looks like the status page is hosted on CloudFront though, so it got part of the way. (Of course, the other question is if it was updatable / updat…

Pardon me if it's a stupid question, but out of curiosity: Is there any way to keep DNS up in case BGP goes down for any reason? Like a fallback nameserver hosted elsewhere/not affected by Facebook's ASs? Is it technically impossible or did Facebook just assume something like yesterday would never happen and kept things simple instead of complicating things?

Yes, but...

If it's a FB managed server, run on someone else's network, you still have a lot of the FB software risk (FB's software stack and development mantra make it easy to push changes, some of which break everything, including the ability to push further changes); even if not FB, there's a similar risk.

If it's not a FB managed server, like a 3rd party DNS provider, it's difficult to get that synchronized considering all the fun geographic loadbalancing FB is doing at the DNS level. That's generally hard once you start doing this; and it's why you don't see many dual-provider DNS setups.

Really, the status page should be not on a core domain, so that the DNS can just be external.

FB DNS breaking yesterday almost doesn't matter in the scheme of things, because the BGP breakage broke everything anyway. Would it have been a bit nicer to get http error messages instead of DNS not found messages, sure; but mostly nothing was working anyway.

Re: Update about the October 4th outage

#224

Post by colechristensen »

implements wrote: post #137
colechristensen wrote: post #44

Earlier quoted context omitted.

I entirely believed that the badge readers on doors were broken, I have set systems like that up and they’re network connected. It is not the least bit surprising that BGP chaos would break that. That doesn’t mean people literally couldn’t get in the building, more like security was really annoyed and confused all day.

I briefly worked on access control a long time ago - aren’t they supposed to cache credentials locally so that individual doors still function autonomously during a fire / loss of power and comms?

Eh I would assume that would only be necessary if the lock mechanism kept people in, the only place I’ve been that did that was a defense contractor, and even then there were alarmed emergency doors. Everywhere else had either no exit control or trivially bypassed exit control.

Re: Update about the October 4th outage

#225

Post by quartz »

ethbr0 wrote: post #200
quartz wrote: post #199

Earlier quoted context omitted.

Facebook has a very healthy approach to incident response (one of the reasons it's so rare for the site to go down at all despite the enormous traffic and daily code pushes). Unless there was some kind of nefarious intent, it's very unlikely anyone will be 'punished'. The likely ramifications will be around changes to processes, tests, automations, and fallbacks to 1) prevent the root sequence of events from happenin…

I've never understood companies that fire individuals when policies were followed and an incident happened. Or, when no policies existed. Or, when policies are routinely bypassed. Organizational failures require organizational solutions. That seems pretty obvious.

100%! Unfortunately even with 60+ years of us formally knowing this-- at least since WWII-- it's very difficult to fight the urge to blame and punish people for incidents.

In many ways we're wired to do it and it FEELS GOOD to do it. Even the industries that are championed for focusing on organizational process over human blame (ex: airlines) are often lulled into initially falling back on the emotional knee-jerk of "pilot error" (see: the early days of the 737 max debacle).

Companies have to be very intentional, usually top-down, about focusing on the context that allowed humans to fail instead of the human themself. That's often easier said than done.

Re: Update about the October 4th outage

#227

Post by yuliyp »

EricE wrote: post #212
yuliyp wrote: post #131

Earlier quoted context omitted.

It's less vague than you realize. It points out that the problem was within Facebook's network between its datacenters. This not only suggests it's related to express backbone, but also suggests that the DNS BGP withdrawal which Cloudflare observed was not the primary issue. It's not a full root cause analysis, to be sure, and leaves many open questions, but I definitely wouldn't describe it as painfully vague.

A point of distinction - there is no "DNS BGP withdrawl". DNS is related to BGP only that without the right BGP routes in the routers, no packets can get to the facebook networks and thus the facbook DNS servers. That their DNS servers were taken out was a side affect of the root issue - they withdrew all the routes to their networks from the rest of the Internet. Not picking on you - but there has been a lot of conf…

FB did not totally disconnect from the internet as far as what I observed. They broke their backbone (from the blog post, as well as seeing that their edge was returning 503s during the incident), and FB withdrew specific BGP announcements which covered their DNS servers are what I saw. I think this is consistent with what Cloudflare actually said, although not what has been often repeated since then.

Re: Update about the October 4th outage

#229

Post by TheCoelacanth »

oarsinsync wrote: post #204
dTal wrote: post #197

Earlier quoted context omitted.

Perhaps you're unaware that billions of devices attempting to resolve Facebook's unresolvable domains effectively DDOS-ed the DNS system? It most certainly did slow down big chunks of internet which otherwise had nothing to do with Facebook. https://www.theverge.com/2021/10/4/22709123/facebook-outage-...

> Some applications may have seen increased load and suffered due to server resourcing constraints, caused by applications like the above failing to fail gracefully, and instead polling more aggressively. I had Cloudflare's woes in mind when I wrote that.

So your point is that it didn't slow down the entire Internet, only the parts of the Internet that use DNS (damn near all of them)?

Re: Update about the October 4th outage

#230

Post by cheesecake_luvr »

DevoidSimo wrote: post #124
cheesecake_luvr wrote: post #118

On a side note: when I browse to that page in Firefox (92.0.1) from HN I can't go back to HN - the back arrow is disabled. What gives?

Do you have the facebook container extension? That closes the current tab, opens a new tab with a container, then goes to the facebook link. Reopening the last closed tab works for me, although I haven't noticed this before since I always open links in a new tab.

Tried Edge, it works as expected. Tried turning off Facebook Container, it also works as expected. So you are right good Sir!

Still, a bit unexpected behaviour though.

Post reply on HN