Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding how Facebook disappeared from the internet

blog.cloudflare.com

Post reply on HN
Print view View original

51–60 of 280 posts

Re: Understanding how Facebook disappeared from the internet

#51

Post by gorgoiler »

throwdecro wrote: post #6

This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?

At a guess: reconnecting traffic at the billions-of-people scale has the potential for finding all sorts of weird behaviours. For all we know, it has been connected and disconnected 10x already during the outage, with each reconnect overwhelming some new, deeper level of the system each time.

Reconnect and the GLBs fall apart under load as the entire world’s cadre of recursive resolvers hit you.

Fix that. Reconnect again. This time your LBs have marked half the servers as offline because their heartbeats have been failing.

Fix that. Reconnect again. Now all the memcache data is hours old and so the site business logic fetches straight from databases, knocking them over.

Fix the databases. Reconnect again. Ad nauseam.

Re: Understanding how Facebook disappeared from the internet

#52

Post by literallyaduck »

literallyWTF wrote: post #13
AlbertCory wrote: post #12

It sorta sounds like Facebook is Too Big To Fail. Yet another reason to dismantle it.

I would rather purge it from this planet and throw Zuck into jail.

Is it harmful?

https://www.theguardian.com/technology/2021/sep/29/facebook-...

Did Facebook lie to congress?

https://www.forbes.com/sites/zakdoffman/2019/03/02/facebook-...

Also short comments without discussion and evidence do poorly.

I believe it should be halted for public safety which is well within the rights and capacity of our government. I believe a fair trial should happen before anyone receives punishment, but I don't believe we could find an impartial jury as Facebook is ubiquitous.

Re: Understanding how Facebook disappeared from the internet

#53

Post by rifkiamil »

bink wrote: post #33
kube-system wrote: post #9

Earlier quoted context omitted.

Given my experience with DNS issues, I am guessing that they are running into dependencies along the way that assume/require DNS be available to function.

With routing it's even worse than that. If they had no out-of-band method to connect to these routers and they botched the routing config then they had no way to route any traffic to them at all. At least with DNS you can still connect to the IPs. I would find it a bit surprising if Facebook didn't have OOB access to their data centers, however.

I'm sure they got stuck in a security loop. To get access OOB passwords or IP address, they needed to get into a password vault that is under facebook AN.

Next time FB save you passwords in OneDrive and Google Drive as a backup LOL. facebook-oob-password@gmail.com

Re: Understanding how Facebook disappeared from the internet

#54

Post by atok1 »

bilbo0s wrote: post #25
powera wrote: post #18

Earlier quoted context omitted.

Facebook has to make some public statement; the shareholders will demand it. I expect the detail level to be roughly "an automated system pushed a broken configuration"; that is to say, there probably won't be any interesting information at all for the Hacker News crowd. I doubt that this was caused by "hackers" or "hostile governments" or "dissident employees upset about Facebook privacy issues", and also doubt that…

The fact is, it's altogether likely that they could be legally require NOT to make such a statement outlining the cause if it was a hostile actor. I've felt a distinct change recently. The US government is not messing around about cyber security anymore. The guys with the blue windbreakers show up, I'd pretty much say "yes, sir." Of course, I don't have FB's power, but I don't think it matters.

I have some fresh boot soup I can offer you? With a side of steel-toe.

Re: Understanding how Facebook disappeared from the internet

#55

Post by huuuz »

dec0dedab0de wrote: post #7
throwdecro wrote: post #6

This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?

I haven't been following closely, but I think once they moved the prefixes they could no longer access the routers. Coupled with barebones staff at the data center due to the pandemic, and all internal communication being disrupted. Though I really expected it to be up within an hour or two.

Reduced staff because of the pandemic? 18 months in? This looks like a sweet deal for those workers more than anything else.

Re: Understanding how Facebook disappeared from the internet

#56

Post by rifkiamil »

Wolfspirit wrote: post #8
throwdecro wrote: post #6

This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?

I'm not sure if that is true (and I hope it is not cause that would be fatal) but I read somewhere that with facebook being down also means all internal infrastructure of facebook isn't available at the moment (chats, communication) including remote control tools for the BGP Routers. Therefor they require people to get physical access to the router while many people are working from home cause of the pandemic.

Thats why we have patten called https://en.wikipedia.org/wiki/Out-of-band_management

Re: Understanding how Facebook disappeared from the internet

#57

Post by OJFord »

huuuz wrote: post #55
dec0dedab0de wrote: post #7

Earlier quoted context omitted.

I haven't been following closely, but I think once they moved the prefixes they could no longer access the routers. Coupled with barebones staff at the data center due to the pandemic, and all internal communication being disrupted. Though I really expected it to be up within an hour or two.

Reduced staff because of the pandemic? 18 months in? This looks like a sweet deal for those workers more than anything else.

Presumably they're working remotely, but the incident required physical access, so only the reduced number of people present were (immediately (or at all, I don't know their policies)) available to deal with it.

Re: Understanding how Facebook disappeared from the internet

#58

Post by imglorp »

Wolfspirit wrote: post #35
interestica wrote: post #28

Earlier quoted context omitted.

No matter what time of year it is, people tend to use 'EST' for 'Eastern Time' even when we might be in Eastern Daylight Time rather than Standard. It's especially annoying when dealing with multiple countries that may or may not be using Daylight Saving Time.

Even google isn't quite sure about the summer time. Not sure if that is just a Google German thing... A few weeks ago I tried to find out what the current time in CET is. Asking google for "CET" gave me: "23:27 CET". Asking google for "CET time" (I know that "time" is twice in this case) gave me "00:27 CET". The last one is wrong and should be CEST or even more correct would be just the same result for CET as I asked…

I am always forgetting the polarity. Seeing the cut-overs listed helps me:

    $ zdump -v Europe/Berlin | grep 2021
    Europe/Berlin  Sun Mar 28 00:59:59 2021 UT = Sun Mar 28 01:59:59 2021 CET isdst=0 gmtoff=3600
    Europe/Berlin  Sun Mar 28 01:00:00 2021 UT = Sun Mar 28 03:00:00 2021 CEST isdst=1 gmtoff=7200
    Europe/Berlin  Sun Oct 31 00:59:59 2021 UT = Sun Oct 31 02:59:59 2021 CEST isdst=1 gmtoff=7200
    Europe/Berlin  Sun Oct 31 01:00:00 2021 UT = Sun Oct 31 02:00:00 2021 CET isdst=0 gmtoff=3600

Re: Understanding how Facebook disappeared from the internet

#59

Post by WaitWaitWha »

This provides a good set of details (mostly educational) what happened up to but not including the how and why the BGP routes were withdrawn (who sent the UPDATE packets to the neighboring ASes?).

The most "natural" occurrence that I can think of is best-path change. If a "better" route between AS is added, the now-second-best routes are withdrawn.

Correct me if I am wrong, but there is no way of determining the source of the withdraw message (UPDATE)...

Re: Understanding how Facebook disappeared from the internet

#60

Post by shyn3 »

powera wrote: post #18
phoe-krk wrote: post #4

Earlier quoted context omitted.

Do you really think that Facebook is going to write one right now? It's good that we have some coverage from companies that have some stake in the game because they are also affected by the outage, even if only partially.

Facebook has to make some public statement; the shareholders will demand it. I expect the detail level to be roughly "an automated system pushed a broken configuration"; that is to say, there probably won't be any interesting information at all for the Hacker News crowd. I doubt that this was caused by "hackers" or "hostile governments" or "dissident employees upset about Facebook privacy issues", and also doubt that…

Facebook has a special ownership structure which means the shareholders have no input as their shares do not have voting power.
Post reply on HN