Live data from Hacker News

Understanding how Facebook disappeared from the internet

blog.cloudflare.com

201–210 of 280 posts

Re: Understanding how Facebook disappeared from the internet

#201

Earlier quoted context omitted.

I'm sure they got stuck in a security loop. To get access OOB passwords or IP address, they needed to get into a password vault that is under facebook AN. Next time FB save you passwords in OneDrive and Google Drive as a backup LOL. facebook-oob-password@gmail.com

Security policies are a pain in cases like this... Laptop + mobile tethering + serial cable to the router + teamvewier for the remote admin to get the access solves problems like this in minutes. Breaking a gajillion security policies by doing that is a different story though.

Don’t you think that losing thousands, millions, or billions of dollars is a pain?

Re: Understanding how Facebook disappeared from the internet

#202
post #197

Those short TTLs that FB likely has on its DNS records probably bit it in the ass today. If it had longer TTLs, caching would have helped it. Curious though that recursive resolvers won't serve an expired cached record when they can't reach the authoritative server. (I know that unbound can be configured to do so, but not sure about others.)

I don't think so. The root cause (as I understand) was that FB stopped advertising it's IP address space to the world. Even if you had the IP addresses of the FB servers, you would not find any route to access them.

The fact that DNS was also not resolving was a symptom of the DNS servers also being unavailable since they were part of the same IP address space that was unadvertised.

Re: Understanding how Facebook disappeared from the internet

#203
post #57
post #55

Earlier quoted context omitted.

Reduced staff because of the pandemic? 18 months in? This looks like a sweet deal for those workers more than anything else.

Presumably they're working remotely, but the incident required physical access, so only the reduced number of people present were (immediately (or at all, I don't know their policies)) available to deal with it.

Not sure Im buying this, facebook have 131 data centers according to one post yesterday, I think they still had some form of OOB

Re: Understanding how Facebook disappeared from the internet

#204

This provides a good set of details (mostly educational) what happened up to but not including the how and why the BGP routes were withdrawn (who sent the UPDATE packets to the neighboring ASes?). The most "natural" occurrence that I can think of is best-path change. If a "better" route between AS is added, the now-second-best routes are withdrawn. Correct me if I am wrong, but there is no way of determining the sour…

That info. would probably come from FB though, not Cloudflare. At least that's how I understand the incident.

Re: Understanding how Facebook disappeared from the internet

#205
post #64

Earlier quoted context omitted.

Unpopular opinion but I think a talent exodus and turnover are likely at fault here as well. The people who stood up this juggernaut are no longer here, and Facebooks ability to consistently attract talent that is competent enough to maintain such a formidable beast is hampered by repetitive revelations that it operates at the net-loss of humanity as a whole. Facebooks no longer an innovator, just a mining operation…

Isn’t Facebook huge in some Asian countries? Supposedly 3.5 billion people use one or more of its services. WhatsApp, Instagram and Oculus certainly aren’t just used by the elderly.

Sounds plausible, but how does that change the ethical calculus of working there?

Re: Understanding how Facebook disappeared from the internet

#206
How can you explain yesterday's outage (Facebook, Instagram, WhatsApp) to your parents?

You are feeling hungry and went to food court. The food court (open area) has a lot of options. You sit down in front of Domino's (Facebook), since you want to eat garlic bread. Now, you can't order from the counter directly. The waiter will come to your seat and ask for the order. You ordered garlic bread from the waiter, but the guy at Domino's counter went missing. Your order was not reaching to the chef in kitchen as Domino's counter guy was not present.

This explains why Domino's (Facebook) ecosystem was down, but what about other vendors? They had nothing to do with Facebook.

To understand this, we need to go back to our food court again. Now, there are a lot of hungry people sitting outside Domino. Since they were not getting answer from one waiter as why their food is not on their table, they started disturbing all the waiters. Due to this, majority of the waiters were trying to figure out where the Domino's counter guy went and other food joints (read websites) were not able to fulfil their own orders.

So although only Domino's was down, it appeared as if whole Food Court (Internet) was facing issues.

Counter Guy at Domino's - Facebook Nameservers Waiters - DNS Servers (Cloudflare, Google, Akamai)

Re: Understanding how Facebook disappeared from the internet

#208
It's been 20+ years since I had reason to know the internals of BGP, but I still carry a slight nervousness whenever I think about it.

In many ways, it's beautiful in that it allows Autonomous Systems to come together and independently make the Internet.

However, I always felt it was brittle in that it relied every AS to have edge routers aware of the entire AS routing map, and so when those routers went down hard, when they came back up they were blank and needed to learn the Internet again. I noted back then that on some routers a single spoofed UDP packet to a router getting a table from its peers could cause it to stop doing that and start again. In a time when those updates could take hours... well... that kept me awake at night sometimes.

Propagation of routes was global and error prone. Even today, you can see on the spy glass websites rogue AS routers advertising crazy routes either by accident or on purpose as a hostile act (to try and get traffic for a target network through itself).

There are tens of thousands of engineers globally nursing and managing this stuff to keep the whole thing going.

Like I said, it's been a while since I last looked closely, and I imagine multiple improvements have been made in this space since that time, but if there was ever a protocol that needs a long, cold hard look for a replacement, it's quite possibly BGP.

It's also the one - and only - non-crypto situation where if somebody asked me if a blockchain could be really very useful, I'd say "probably, yeah".

Re: Understanding how Facebook disappeared from the internet

#209

How can you explain yesterday's outage (Facebook, Instagram, WhatsApp) to your parents? You are feeling hungry and went to food court. The food court (open area) has a lot of options. You sit down in front of Domino's (Facebook), since you want to eat garlic bread. Now, you can't order from the counter directly. The waiter will come to your seat and ask for the order. You ordered garlic bread from the waiter, but the…

I think that this may be more confusing than the actual explanation.

Re: Understanding how Facebook disappeared from the internet

#210

How can you explain yesterday's outage (Facebook, Instagram, WhatsApp) to your parents? You are feeling hungry and went to food court. The food court (open area) has a lot of options. You sit down in front of Domino's (Facebook), since you want to eat garlic bread. Now, you can't order from the counter directly. The waiter will come to your seat and ask for the order. You ordered garlic bread from the waiter, but the…

I think that this may be more confusing than the actual explanation.

Thanks for the feedback. I will simplify it further!
Post reply on HN