Gotta love how painfully vague this is. Sounds like a PR piece for investors, not an engineering blog piece.
It's not a full root cause analysis, to be sure, and leaves many open questions, but I definitely wouldn't describe it as painfully vague.
131–140 of 239 posts
Gotta love how painfully vague this is. Sounds like a PR piece for investors, not an engineering blog piece.
It's not a full root cause analysis, to be sure, and leaves many open questions, but I definitely wouldn't describe it as painfully vague.
Earlier quoted context omitted.
Both posts have essentially the same info - the fb one just didn't include an explainer on how the internet works.
The Cloudflare post includes graphs depicting how the issue looked downstream and a rough initial timeline for the incident. The Facebook post says basically nothing more than that there was a networking mistake. I hope we get something more substantial and informative than that over the next couple of weeks, but it doesn't seem (at least from my searching) that Facebook is in the business of publicly posting in-dept…
Earlier quoted context omitted.
How would you explain that AWS, GCE, Cloudflare, GitLab publish very detailed post-mortems?
It's a marketing strategy. Their target customer segment is technical. FaceBooks and Twitter's for the most part, aren't.
Earlier quoted context omitted.
The didn't revoke all their routes, FWIW, just a lot of them (including the anycast DNS routes)
What I don't understand is why, when a route is revoked, if there is no other route announced the routing table gets updated? It seems like either it's a black hole or it still works and there was a BGP error (or the route works but the resources aren't present, so traffic would be dropped). What's the reason for designing the system to revoke routes when no new route is announced? It strikes me it's like DNS when yo…
When the announcement is revoked, you fall back to a less specific prefix if present, or your default route.
If you've got a full BGP table, then you tend not to have a useful default route (you should have specific routes for everything) and it might be useful to fallback to the last known value. But many participants have an intentional default route and then get announcements for special traffic --- dropping the announcement would mean to send it on the default route instead. It's hard to know what the right thing to do is, so better to do what you were told by the authority.
DNS is a bit different, but again, the authority told you to use some data and how long to keep it (ttl), if they're not there to tell you a value again later, what else can you do but report an error? Some DNS servers have configurable behavior to continue using old data while fetching new data or when new data is unavailable.
But the expectation is if you can't keep your BGP up and your DNS up, your server probably isn't up either. Note that in this case, bypassing DNS and going to the FB Edge PoPs that were still network available (because of different BGP announcements, that weren't withdrawn) resulted in errors, because they weren't able to connect to the upstream data centers. (Or so it seems)
Earlier quoted context omitted.
The didn't revoke all their routes, FWIW, just a lot of them (including the anycast DNS routes)
What I don't understand is why, when a route is revoked, if there is no other route announced the routing table gets updated? It seems like either it's a black hole or it still works and there was a BGP error (or the route works but the resources aren't present, so traffic would be dropped). What's the reason for designing the system to revoke routes when no new route is announced? It strikes me it's like DNS when yo…
Earlier quoted context omitted.
- It was government intervention - Facebook was hacked - They did it on purpose to bury the whistleblower story - No one could access Facebook offices - They had to cut open servers with angle grinders - Disgruntled employees changed DNS records - Lots of made up numbers for how much money Facebook/the rest of the economy was losing (or gaining) They probably rushed out this blog post just to dispel some of these rum…
I entirely believed that the badge readers on doors were broken, I have set systems like that up and they’re network connected. It is not the least bit surprising that BGP chaos would break that. That doesn’t mean people literally couldn’t get in the building, more like security was really annoyed and confused all day.
The badge story only shows how people are looking for "efficiency" where it doesn't matter, with predictable results. The badge system should be local to the building. There are few actual reasons (sure, besides "efficiency") of why badge control should be centralized. Even less reasons for it to be a subdomain of fb. Another option would be to keep the system but make it failsafe (but it seems the newer generation d…
Akso, it's likely not on an fb subdomain, but something like office.security.fb-infra.com (example). It just happens to be that fb-infra.com is using the Facebook DNS server.