Remember, remember, the 4th of October.
Update about the October 4th outage
51–60 of 239 posts
Re: Update about the October 4th outage
#52Gotta love how painfully vague this is. Sounds like a PR piece for investors, not an engineering blog piece.
Disagree -- it's here to establish something that a lot of people have been speculating about, which is whether it's hacking-related. It doesn't say much because its purpose is to deliver a single bit of information: { hacking: boolean }
Re: Update about the October 4th outage
#53Just out of curiosity, does Facebook have a status page? Like http://status.twitter.com?
Re: Update about the October 4th outage
#54Just out of curiosity, does Facebook have a status page? Like http://status.twitter.com ?
https://status.fb.com/
Although it only covers their API and business apps, not the site itself.
Re: Update about the October 4th outage
#55> configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues This could be anything, potentially. I'm not very knowledgeable in computer networking, but this could be as trivial as an incorrect update to a DNS record, right?
There seems to be nothing uncertain about the immediate cause of the issue - Facebook revoked all of their BGP routes, and all of their IP addresses couldn't receive packets until they were restored.
Understood.
I had question: This is what we can only perceive through internet/routing table entries right?
Internal to FB, we don't know what had caused issues that led to the BGP UPDATE.
That's kind of what has been confusing me - there's a lot of speculation around FB's data center design and what actually happened, but we actually don't know for sure until they post an RCA - please correct me if I'm wrong here.
Re: Update about the October 4th outage
#56Sounds like they could do with some updates to their risk-driven backbone management strategy!
https://engineering.fb.com/2021/08/09/connectivity/backbone-...
Re: Update about the October 4th outage
#57Re: Update about the October 4th outage
#58Just out of curiosity, does Facebook have a status page? Like http://status.twitter.com ?
https://status.fb.com/ Although it only covers their API and business apps, not the site itself.
Looks like it! I remember trying to access this site during the down time and it turns out it’s not accessible at the moment
Re: Update about the October 4th outage
#59> configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues This could be anything, potentially. I'm not very knowledgeable in computer networking, but this could be as trivial as an incorrect update to a DNS record, right?
Backbone routers don't usually deal with hostnames or DNS. This is pretty much saying they done broke BGP. And it sounds like they're saying that they broke it in a way that prevented accessing their data centers from the PoPs, and we know from the long downtime that it prevented accessing the BGP configuration system from darn near anywhere.
It happened to also kill the announcements for anycast DNS.
Re: Update about the October 4th outage
#60> configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues This could be anything, potentially. I'm not very knowledgeable in computer networking, but this could be as trivial as an incorrect update to a DNS record, right?
There seems to be nothing uncertain about the immediate cause of the issue - Facebook revoked all of their BGP routes, and all of their IP addresses couldn't receive packets until they were restored.
The didn't revoke all their routes, FWIW, just a lot of them (including the anycast DNS routes)