Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding how Facebook disappeared from the internet

blog.cloudflare.com

Post reply on HN
Print view View original

41–50 of 280 posts

Re: Understanding how Facebook disappeared from the internet

#41

Post by 0des »

powera wrote: post #18
phoe-krk wrote: post #4

Earlier quoted context omitted.

Do you really think that Facebook is going to write one right now? It's good that we have some coverage from companies that have some stake in the game because they are also affected by the outage, even if only partially.

Facebook has to make some public statement; the shareholders will demand it. I expect the detail level to be roughly "an automated system pushed a broken configuration"; that is to say, there probably won't be any interesting information at all for the Hacker News crowd. I doubt that this was caused by "hackers" or "hostile governments" or "dissident employees upset about Facebook privacy issues", and also doubt that…

>Facebook has to make some public statement

History has shown us they can give us zero response, or an incorrect response, and we (via our representatives) will accept it and continue living life as before.

Re: Understanding how Facebook disappeared from the internet

#42

Post by tester756 »

PeterCorless wrote: post #36
throwdecro wrote: post #6

This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?

If an authoritative DNS entry was removed, it can take up to 72 hours for that change to be propagated around the world, though usually just a few hours for some other authoritative DNS systems to get you mostly back: https://ns1.com/resources/dns-propagation#:~:text=DNS%20prop... .

Why it takes this long?

Re: Understanding how Facebook disappeared from the internet

#44

Post by _joel »

toomuchtodo wrote: post #39
cwilkes wrote: post #38

Updating BGP configs should go through a flowchart like this: Do you want to update BGP? No: exit Yes: type this random 100 character phrase to continue no copy paste

“Are you sure? Can people you need to recover from this change already in the building?”

Maybe a timed rollback with the previous state stored on the device that needs to be rolled back, althogh if you're doing this at facebook scale I'm sure that's a little more difficult than it sounds, perhaps.

Re: Understanding how Facebook disappeared from the internet

#45

Post by withinboredom »

tester756 wrote: post #42
PeterCorless wrote: post #36

Earlier quoted context omitted.

If an authoritative DNS entry was removed, it can take up to 72 hours for that change to be propagated around the world, though usually just a few hours for some other authoritative DNS systems to get you mostly back: https://ns1.com/resources/dns-propagation#:~:text=DNS%20prop... .

Why it takes this long?

Caching

Re: Understanding how Facebook disappeared from the internet

#46

Post by withinboredom »

bink wrote: post #33
kube-system wrote: post #9

Earlier quoted context omitted.

Given my experience with DNS issues, I am guessing that they are running into dependencies along the way that assume/require DNS be available to function.

With routing it's even worse than that. If they had no out-of-band method to connect to these routers and they botched the routing config then they had no way to route any traffic to them at all. At least with DNS you can still connect to the IPs. I would find it a bit surprising if Facebook didn't have OOB access to their data centers, however.

Assuming you don’t need DNS to get authorization to enter the OOB access…

Re: Understanding how Facebook disappeared from the internet

#47

Post by rifkiamil »

dec0dedab0de wrote: post #7
throwdecro wrote: post #6

This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?

I haven't been following closely, but I think once they moved the prefixes they could no longer access the routers. Coupled with barebones staff at the data center due to the pandemic, and all internal communication being disrupted. Though I really expected it to be up within an hour or two.

We have had out-of-band management ports & networks design for decades! I know the feeling of driving 8 hours because I lost connection to the device I was configuring. https://en.wikipedia.org/wiki/Out-of-band_management

Re: Understanding how Facebook disappeared from the internet

#48

Post by Hikikomori »

throwdecro wrote: post #6

This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?

Restoring is just as simple as flipping the switch again, but access to that switch is another matter when your internal network is also down and you cannot even get access to your office or datacenters.

Re: Understanding how Facebook disappeared from the internet

#50

Post by [deleted] »

throwdecro wrote: post #6

This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?

[deleted]
Post reply on HN