Live data from Hacker News

Update about the October 4th outage

engineering.fb.com

131–140 of 239 posts

Re: Update about the October 4th outage

#131

Gotta love how painfully vague this is. Sounds like a PR piece for investors, not an engineering blog piece.

It's less vague than you realize. It points out that the problem was within Facebook's network between its datacenters. This not only suggests it's related to express backbone, but also suggests that the DNS BGP withdrawal which Cloudflare observed was not the primary issue.

It's not a full root cause analysis, to be sure, and leaves many open questions, but I definitely wouldn't describe it as painfully vague.

Re: Update about the October 4th outage

#132

Earlier quoted context omitted.

Both posts have essentially the same info - the fb one just didn't include an explainer on how the internet works.

The Cloudflare post includes graphs depicting how the issue looked downstream and a rough initial timeline for the incident. The Facebook post says basically nothing more than that there was a networking mistake. I hope we get something more substantial and informative than that over the next couple of weeks, but it doesn't seem (at least from my searching) that Facebook is in the business of publicly posting in-dept…

Cloudflare can scratch the surface of the issue it wouldn't matter, it is a content marketing piece after all. Facebook, otoh, needs to be thorough.

Re: Update about the October 4th outage

#133
post #112

Earlier quoted context omitted.

How would you explain that AWS, GCE, Cloudflare, GitLab publish very detailed post-mortems?

It's a marketing strategy. Their target customer segment is technical. FaceBooks and Twitter's for the most part, aren't.

Yeah, also a chance some eng from AWS/GCP/Azure leaks actual details if they lie or if public statements are inadequate.

Re: Update about the October 4th outage

#134
post #60

Earlier quoted context omitted.

The didn't revoke all their routes, FWIW, just a lot of them (including the anycast DNS routes)

What I don't understand is why, when a route is revoked, if there is no other route announced the routing table gets updated? It seems like either it's a black hole or it still works and there was a BGP error (or the route works but the resources aren't present, so traffic would be dropped). What's the reason for designing the system to revoke routes when no new route is announced? It strikes me it's like DNS when yo…

BGP operates on the principle of using the 'most specific prefix', basically if all else is equal, a route covering fewer IPs is more specific and should be used.

When the announcement is revoked, you fall back to a less specific prefix if present, or your default route.

If you've got a full BGP table, then you tend not to have a useful default route (you should have specific routes for everything) and it might be useful to fallback to the last known value. But many participants have an intentional default route and then get announcements for special traffic --- dropping the announcement would mean to send it on the default route instead. It's hard to know what the right thing to do is, so better to do what you were told by the authority.

DNS is a bit different, but again, the authority told you to use some data and how long to keep it (ttl), if they're not there to tell you a value again later, what else can you do but report an error? Some DNS servers have configurable behavior to continue using old data while fetching new data or when new data is unavailable.

But the expectation is if you can't keep your BGP up and your DNS up, your server probably isn't up either. Note that in this case, bypassing DNS and going to the FB Edge PoPs that were still network available (because of different BGP announcements, that weren't withdrawn) resulted in errors, because they weren't able to connect to the upstream data centers. (Or so it seems)

Re: Update about the October 4th outage

#135
post #60

Earlier quoted context omitted.

The didn't revoke all their routes, FWIW, just a lot of them (including the anycast DNS routes)

What I don't understand is why, when a route is revoked, if there is no other route announced the routing table gets updated? It seems like either it's a black hole or it still works and there was a BGP error (or the route works but the resources aren't present, so traffic would be dropped). What's the reason for designing the system to revoke routes when no new route is announced? It strikes me it's like DNS when yo…

Basically the same problem as tag soup. As soon as mistakes don't stop things working you get many more mistakes.

Re: Update about the October 4th outage

#137
post #41

Earlier quoted context omitted.

- It was government intervention - Facebook was hacked - They did it on purpose to bury the whistleblower story - No one could access Facebook offices - They had to cut open servers with angle grinders - Disgruntled employees changed DNS records - Lots of made up numbers for how much money Facebook/the rest of the economy was losing (or gaining) They probably rushed out this blog post just to dispel some of these rum…

I entirely believed that the badge readers on doors were broken, I have set systems like that up and they’re network connected. It is not the least bit surprising that BGP chaos would break that. That doesn’t mean people literally couldn’t get in the building, more like security was really annoyed and confused all day.

I briefly worked on access control a long time ago - aren’t they supposed to cache credentials locally so that individual doors still function autonomously during a fire / loss of power and comms?

Re: Update about the October 4th outage

#139

The badge story only shows how people are looking for "efficiency" where it doesn't matter, with predictable results. The badge system should be local to the building. There are few actual reasons (sure, besides "efficiency") of why badge control should be centralized. Even less reasons for it to be a subdomain of fb. Another option would be to keep the system but make it failsafe (but it seems the newer generation d…

Having the badge system work from a single point has a lot of advantages for a company like FB: HR can update info from everywhere (they might not be in the same office), you can immediately deny or block a card everywhere, you have an audit log etc.. They're not having this for fun.

Akso, it's likely not on an fb subdomain, but something like office.security.fb-infra.com (example). It just happens to be that fb-infra.com is using the Facebook DNS server.

Post reply on HN