Understanding how Facebook disappeared from the internet • Hacker News Board

Re: Understanding how Facebook disappeared from the internet

#261

Post by alfalfasprout » Tue, Oct 05, 2021, 5:36 PM UTC

rifkiamil wrote: post #47

dec0dedab0de wrote: post #7

Earlier quoted context omitted.

I haven't been following closely, but I think once they moved the prefixes they could no longer access the routers. Coupled with barebones staff at the data center due to the pandemic, and all internal communication being disrupted. Though I really expected it to be up within an hour or two.

We have had out-of-band management ports & networks design for decades! I know the feeling of driving 8 hours because I lost connection to the device I was configuring. https://en.wikipedia.org/wiki/Out-of-band_management

This. And if you're really worried about it you can go crazy with security with individually issued hardware security tokens and one time use access tokens.

It's pretty inexcusable that FB wasn't able to use OOB management.

Re: Understanding how Facebook disappeared from the internet

#262

Post by alfalfasprout » Tue, Oct 05, 2021, 5:41 PM UTC

Stuff like this happens... but the real question is why FB had to send employees to physically go to datacenters to fix this. Sounds like their OOB management had a dependency on FB being online which seems like a bad design. If you're that paranoid about security issue revokable one time pads to a select set of employees.

Re: Understanding how Facebook disappeared from the internet

#263

Post by ampdepolymerase » Tue, Oct 05, 2021, 5:45 PM UTC

jrochkind1 wrote: post #68

ampdepolymerase wrote: post #62

Earlier quoted context omitted.

HN engineers believe out of band admin control planes to be surveillance and backdoor firmware so they are disabled for privacy reasons.

Who the hell put HN engineers are in charge of facebook, and why haven't we gotten more out of it than a temporary outage??

Previously they had physical access to the data centers and weren't locked out.

Re: Understanding how Facebook disappeared from the internet

#264

Post by WorldMaker » Tue, Oct 05, 2021, 6:13 PM UTC

oblio wrote: post #242

nroets wrote: post #224

Earlier quoted context omitted.

So much of the internet is about translating addresses from one format to another: Human readable to IP: DNS Private IP to Public IP: NAT MAC to Private IP: ARP Public IP to AS: BGP With a block chain DNS it may be possible to replace all of them with a single protocol, but it may well turn out to be an academic exercise.

> So much of the internet is about translating addresses from one format to another Most of computing is about mapping A to B and maintaining databases of those mappings :-)

As the saying goes there are two hard problems in computing: naming things, cache invalidation, and buffer overflows.

Most of these systems are about naming things and cache invalidation.

Re: Understanding how Facebook disappeared from the internet

#265

Post by WorldMaker » Tue, Oct 05, 2021, 6:18 PM UTC

Hikikomori wrote: post #223

p4bl0 wrote: post #212

Earlier quoted context omitted.

But then it is the DNS that had to propagate from the now accessible authoritative servers.

Shouldn't take long once they are up and responding again?

I'd assume it is a cache invalidation problem at that point: from my lay understanding BGP probably needs to busts caches on a withdrawal to keep traffic from going to black holes and prevent DDoS attempts, but probably has to wait for TTL timeouts to cache new routes (and those TTLs are going to vary by whatever cache systems the other ASes are running not the timing of Facebook's AS sending the new [old] routes.).

Re: Understanding how Facebook disappeared from the internet

#266

Post by WorldMaker » Tue, Oct 05, 2021, 6:29 PM UTC

pitkali wrote: post #222

phtrivier wrote: post #216

Earlier quoted context omitted.

That would have been accurate for a DNS outage ; but with my layman understanding of BGP, I would say the analogy would be something between "...but their phone line is broken" and "...but they disappeared from the phone book because they don't have a phone line any more" . Is that right ?

Is that an interesting distinction for the target audience, though?

Actually, it probably is, especially if you dial the analogy back a couple decades before the "We're sorry that number has been disconnected" automated responses: Facebook's phone line went down and when you call the Operator even if you have the phone number, they can't connect you, but this is weird and you aren't the only one trying to call Facebook so now they are calling in other Operators to diagnose the problem because surely someone has heard from Facebook recently.

That analogy includes the snowball impact on the other websites and services as the Switchboard Operators get more over-utilized into puzzling out Facebook's problem than servicing calls for still working phone numbers.

Re: Understanding how Facebook disappeared from the internet

#267

Post by WaitWaitWha » Tue, Oct 05, 2021, 6:56 PM UTC

WaitWaitWha wrote: post #59

This provides a good set of details (mostly educational) what happened up to but not including the how and why the BGP routes were withdrawn (who sent the UPDATE packets to the neighboring ASes?). The most "natural" occurrence that I can think of is best-path change. If a "better" route between AS is added, the now-second-best routes are withdrawn. Correct me if I am wrong, but there is no way of determining the sour…

> "Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. "

and

> "... its root cause was a faulty configuration change on our end."

From https://engineering.fb.com/2021/10/04/networking-traffic/out...

Re: Understanding how Facebook disappeared from the internet

#268

Post by Hikikomori » Tue, Oct 05, 2021, 7:35 PM UTC

WorldMaker wrote: post #265

Hikikomori wrote: post #223

Earlier quoted context omitted.

Shouldn't take long once they are up and responding again?

I'd assume it is a cache invalidation problem at that point: from my lay understanding BGP probably needs to busts caches on a withdrawal to keep traffic from going to black holes and prevent DDoS attempts, but probably has to wait for TTL timeouts to cache new routes (and those TTLs are going to vary by whatever cache systems the other ASes are running not the timing of Facebook's AS sending the new [old] routes.).

BGP doesn't really cache routes, though you can configure routers to hold a route for some time if a peer times out, but this is only used in special cases and usually not something you would want your router to do with routes to other networks. If a router gets a withdrawal for a route it will remove the route from the table without waiting. This is a feature and important part of how BGP acts in case of problems and how it can self heal quickly, there's no point in caching a route when that path is not working anymore and usually there's a backup path.

Re: Understanding how Facebook disappeared from the internet

#269

Post by [deleted] » Tue, Oct 05, 2021, 8:00 PM UTC

asadjb wrote: post #202

js2 wrote: post #197

Those short TTLs that FB likely has on its DNS records probably bit it in the ass today. If it had longer TTLs, caching would have helped it. Curious though that recursive resolvers won't serve an expired cached record when they can't reach the authoritative server. (I know that unbound can be configured to do so, but not sure about others.)

I don't think so. The root cause (as I understand) was that FB stopped advertising it's IP address space to the world. Even if you had the IP addresses of the FB servers, you would not find any route to access them. The fact that DNS was also not resolving was a symptom of the DNS servers also being unavailable since they were part of the same IP address space that was unadvertised.

[deleted]

Re: Understanding how Facebook disappeared from the internet

#270

Post by xtat » Tue, Oct 05, 2021, 8:07 PM UTC

BFG-One wrote: post #238

xtat wrote: post #102

Earlier quoted context omitted.

Cloudflare are also good at disappearing people from the Internet

I don't trust any analysis from CloudFlare. These are the same people preventing people with vpns or tor browsers from reaching sites while screaming and yelling OMG DDoS!! then blaming the customer of CloudFlare for not knowing how to configure it.

big +1 here