31–40 of 280 posts
"Because of this Cloudflare’s 1.1.1.1 DNS resolver could no longer respond to queries asking for the IP address of facebook.com or instagram.com." The instagram.com zone itself uses a third-party DNS service and didn't go down. (But e.g. www.instagram.com is a CNAME to a zone on FB DNS.)
That’s pretty much why during the downtime visit instagram.com showed a 503 from AWS instead.
This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?
Given my experience with DNS issues, I am guessing that they are running into dependencies along the way that assume/require DNS be available to function.
I would find it a bit surprising if Facebook didn't have OOB access to their data centers, however.
This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?
I’m pretty new to BGP, but I’d imagine that cutting off access to an AS is fast because all it takes is for the neighbouring routers update their routes. At which point any traffic that makes it that far is simply dropped. Whereas to make an announcement, the entire internet (or at least all routers between the AS and the user) need to pickup the new announcement. (Note: I still need to read the article)
I think it's not so simple because authoritative DNS systems are involved.
So it's not just a BGP error. It's a BGP error which disconnected authoritative DNS for all facebook. I'm not quite sure why that makes it so slow to fix. is it just because internal difficulties due to having no DNS at all?
> ... but as of 22:28 UTC Facebook appears to be ... Someone assumed London==UTC, when London is 1 hour ahead :) that was actually 21:28 UTC
No matter what time of year it is, people tend to use 'EST' for 'Eastern Time' even when we might be in Eastern Daylight Time rather than Standard. It's especially annoying when dealing with multiple countries that may or may not be using Daylight Saving Time.
A few weeks ago I tried to find out what the current time in CET is. Asking google for "CET" gave me: "23:27 CET". Asking google for "CET time" (I know that "time" is twice in this case) gave me "00:27 CET".
The last one is wrong and should be CEST or even more correct would be just the same result for CET as I asked for
This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?
https://ns1.com/resources/dns-propagation#:~:text=DNS%20prop....
Earlier quoted context omitted.
That’s pretty much why during the downtime visit instagram.com showed a 503 from AWS instead.
Wonder why they're still using AWS given that FB operates its own data centers...
Do you want to update BGP?
No: exit
Yes: type this random 100 character phrase to continue no copy paste
Updating BGP configs should go through a flowchart like this: Do you want to update BGP? No: exit Yes: type this random 100 character phrase to continue no copy paste