Understanding how Facebook disappeared from the internet • Hacker News Board

Re: Understanding how Facebook disappeared from the internet

#61

Post by literallyWTF » Mon, Oct 04, 2021, 11:21 PM UTC

literallyaduck wrote: post #52

literallyWTF wrote: post #13

Earlier quoted context omitted.

I would rather purge it from this planet and throw Zuck into jail.

Is it harmful? https://www.theguardian.com/technology/2021/sep/29/facebook-... Did Facebook lie to congress? https://www.forbes.com/sites/zakdoffman/2019/03/02/facebook-... Also short comments without discussion and evidence do poorly. I believe it should be halted for public safety which is well within the rights and capacity of our government. I believe a fair trial should happen before anyone receives punishment,…

Why would I care if my short comment does poorly or not? Everything’s always a 10 page “discussion” on this site fuck me

Re: Understanding how Facebook disappeared from the internet

#62

Post by ampdepolymerase » Mon, Oct 04, 2021, 11:22 PM UTC

rifkiamil wrote: post #47

dec0dedab0de wrote: post #7

Earlier quoted context omitted.

I haven't been following closely, but I think once they moved the prefixes they could no longer access the routers. Coupled with barebones staff at the data center due to the pandemic, and all internal communication being disrupted. Though I really expected it to be up within an hour or two.

We have had out-of-band management ports & networks design for decades! I know the feeling of driving 8 hours because I lost connection to the device I was configuring. https://en.wikipedia.org/wiki/Out-of-band_management

HN engineers believe out of band admin control planes to be surveillance and backdoor firmware so they are disabled for privacy reasons.

Re: Understanding how Facebook disappeared from the internet

#63

Post by rawoke083600 » Mon, Oct 04, 2021, 11:24 PM UTC

cwilkes wrote: post #38

Updating BGP configs should go through a flowchart like this: Do you want to update BGP? No: exit Yes: type this random 100 character phrase to continue no copy paste

or like an iptables or Windows desktop screen resolution change :)

Re: Understanding how Facebook disappeared from the internet

#64

Post by nimbius » Mon, Oct 04, 2021, 11:25 PM UTC

throwdecro wrote: post #6

This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?

Unpopular opinion but I think a talent exodus and turnover are likely at fault here as well. The people who stood up this juggernaut are no longer here, and Facebooks ability to consistently attract talent that is competent enough to maintain such a formidable beast is hampered by repetitive revelations that it operates at the net-loss of humanity as a whole.

Facebooks no longer an innovator, just a mining operation with a dwindling population of hateful elderly and bots.

Re: Understanding how Facebook disappeared from the internet

#65

Post by robocat » Mon, Oct 04, 2021, 11:26 PM UTC

Also see Krebs’ article on the outage (although he hasn’t yet updated it after the problem was rectified): https://krebsonsecurity.com/2021/10/what-happened-to-faceboo...

Re: Understanding how Facebook disappeared from the internet

#66

Post by Hikikomori » Mon, Oct 04, 2021, 11:29 PM UTC

adamcharnock wrote: post #27

throwdecro wrote: post #6

This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?

I’m pretty new to BGP, but I’d imagine that cutting off access to an AS is fast because all it takes is for the neighbouring routers update their routes. At which point any traffic that makes it that far is simply dropped. Whereas to make an announcement, the entire internet (or at least all routers between the AS and the user) need to pickup the new announcement. (Note: I still need to read the article)

Once they start adversiting again it should only take a few minutes at most for most ISPs to get it.

Re: Understanding how Facebook disappeared from the internet

#67

Post by tremon » Mon, Oct 04, 2021, 11:31 PM UTC

amachefe wrote: post #32

Facebook.com is up now, looks like the issue now is the billions of request that is DDoS on the DNS servers

Why? Do you think there's billions of widely-used recursive resolvers in the world? Each resolver only needs to contact the Facebook DNS servers once per hostname, the end-user requests are all served from cache.

Re: Understanding how Facebook disappeared from the internet

#68

Post by jrochkind1 » Mon, Oct 04, 2021, 11:32 PM UTC

ampdepolymerase wrote: post #62

rifkiamil wrote: post #47

Earlier quoted context omitted.

We have had out-of-band management ports & networks design for decades! I know the feeling of driving 8 hours because I lost connection to the device I was configuring. https://en.wikipedia.org/wiki/Out-of-band_management

HN engineers believe out of band admin control planes to be surveillance and backdoor firmware so they are disabled for privacy reasons.

Who the hell put HN engineers are in charge of facebook, and why haven't we gotten more out of it than a temporary outage??

Re: Understanding how Facebook disappeared from the internet

#69

Post by gcbirzan » Mon, Oct 04, 2021, 11:37 PM UTC

ruoso wrote: post #20

> ... but as of 22:28 UTC Facebook appears to be ... Someone assumed London==UTC, when London is 1 hour ahead :) that was actually 21:28 UTC

Actually, the article seems to confuse the times quite a lot. It's talking about ~16:50 UTC at points, but the outage started at 15:40, which they not only mention in the article but you can also see on the graphs.

Re: Understanding how Facebook disappeared from the internet

#70

Post by ajsnigrutin » Mon, Oct 04, 2021, 11:40 PM UTC

rifkiamil wrote: post #53

bink wrote: post #33

Earlier quoted context omitted.

With routing it's even worse than that. If they had no out-of-band method to connect to these routers and they botched the routing config then they had no way to route any traffic to them at all. At least with DNS you can still connect to the IPs. I would find it a bit surprising if Facebook didn't have OOB access to their data centers, however.

I'm sure they got stuck in a security loop. To get access OOB passwords or IP address, they needed to get into a password vault that is under facebook AN. Next time FB save you passwords in OneDrive and Google Drive as a backup LOL. facebook-oob-password@gmail.com

Security policies are a pain in cases like this...

Laptop + mobile tethering + serial cable to the router + teamvewier for the remote admin to get the access solves problems like this in minutes.

Breaking a gajillion security policies by doing that is a different story though.