Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding how Facebook disappeared from the internet

blog.cloudflare.com

Post reply on HN
Print view View original

271–280 of 280 posts

Re: Understanding how Facebook disappeared from the internet

#271

Post by allarm »

rifkiamil wrote: post #47
dec0dedab0de wrote: post #7

Earlier quoted context omitted.

I haven't been following closely, but I think once they moved the prefixes they could no longer access the routers. Coupled with barebones staff at the data center due to the pandemic, and all internal communication being disrupted. Though I really expected it to be up within an hour or two.

We have had out-of-band management ports & networks design for decades! I know the feeling of driving 8 hours because I lost connection to the device I was configuring. https://en.wikipedia.org/wiki/Out-of-band_management

It’s likely that their oobm systems are depending on dns in some way - aaa for instance. It’s not possible that they don’t have oobm at all, really.

Re: Understanding how Facebook disappeared from the internet

#272

Post by allarm »

kelp wrote: post #92
foobarian wrote: post #74

Earlier quoted context omitted.

What I don't get is why there is no dead-man switch mechanism in place to roll back the configuration automatically unless someone confirms it positively. Kind of how screen resolution rolls back if you don't ack it. I used to always run a "(sleep 600; iptables -F) &" when messing with remote personal stuff just in case I lock myself out. I suppose with something like BGP it would be very difficult to get such a fall…

This is a key feature of Junos on Juniper devices. It's called 'commit confirmed' and it will apply the config and then auto-rollback if you don't confirm it within a certain amount of time. https://www.juniper.net/documentation/us/en/software/junos/c... But Facebook's network is certainly much more complex and automated than just doing one commit on one device. But I think they do still uses Juniper devices at the e…

Cisco have the rollback feature as well. It’s not implemented as smoothly as in Juniper though.

Re: Understanding how Facebook disappeared from the internet

#273

Post by RustyConsul »

WorldMaker wrote: post #264
oblio wrote: post #242

Earlier quoted context omitted.

> So much of the internet is about translating addresses from one format to another Most of computing is about mapping A to B and maintaining databases of those mappings :-)

As the saying goes there are two hard problems in computing: naming things, cache invalidation, and buffer overflows. Most of these systems are about naming things and cache invalidation.

but if it's base zero, you still wouldn't get a buffer overflow ;)

Re: Understanding how Facebook disappeared from the internet

#274

Post by collinmanderson »

nroets wrote: post #224
PaulRobinson wrote: post #208

It's been 20+ years since I had reason to know the internals of BGP, but I still carry a slight nervousness whenever I think about it. In many ways, it's beautiful in that it allows Autonomous Systems to come together and independently make the Internet. However, I always felt it was brittle in that it relied every AS to have edge routers aware of the entire AS routing map, and so when those routers went down hard, w…

So much of the internet is about translating addresses from one format to another: Human readable to IP: DNS Private IP to Public IP: NAT MAC to Private IP: ARP Public IP to AS: BGP With a block chain DNS it may be possible to replace all of them with a single protocol, but it may well turn out to be an academic exercise.

I believe IPv6 intended to replace both ARP/MAC and NAT, but has mostly failed to achieve that.

Re: Understanding how Facebook disappeared from the internet

#275

Post by WorldMaker »

RustyConsul wrote: post #273
WorldMaker wrote: post #264

Earlier quoted context omitted.

As the saying goes there are two hard problems in computing: naming things, cache invalidation, and buffer overflows. Most of these systems are about naming things and cache invalidation.

but if it's base zero, you still wouldn't get a buffer overflow ;)

Unless it needs to be a null pointer terminated array.

Re: Understanding how Facebook disappeared from the internet

#276

Post by ksaj »

BFG-One wrote: post #250
ffwszgf wrote: post #139

Earlier quoted context omitted.

You would think those in charge of security would know better with their mid-six figure salaries. But then again these days they stopped hiring based on merit but rather certain other “metrics” no wonder they were so incompetent...

was not them. was an intern.

The intern wasn't the single point of failure. They just triggered the failure. This should have been obvious at the design stage and pre-mitigated.

And if they don't mitigate it now, every hacker with access into their network now knows how to bring the whole show down in an instant.

Re: Understanding how Facebook disappeared from the internet

#277

Post by atf104 »

Godel_unicode wrote: post #133
gorgoiler wrote: post #51

Earlier quoted context omitted.

At a guess: reconnecting traffic at the billions-of-people scale has the potential for finding all sorts of weird behaviours. For all we know, it has been connected and disconnected 10x already during the outage, with each reconnect overwhelming some new, deeper level of the system each time. Reconnect and the GLBs fall apart under load as the entire world’s cadre of recursive resolvers hit you. Fix that. Reconnect a…

I find this kind of uninformed conjecture amusing on a thread full of people complaining about cloudflare doing the same (they didn't). There's no evidence of this kind of flapping behavior in any of the telemetry I've seen posted by network engineer friends, and the blog post explicitly calls out when they saw the BGP updates that brought the site back online.

For me, the hardest part to believe is there was literally no one on-site at their datacenters. Really? No one? At this scale, literally, there has to be a security guard there who can kick the door open.

Re: Understanding how Facebook disappeared from the internet

#280

Post by pinix »

Let's say that when Facebook cannot announce their DNS prefixes, the DNS resolvers from other ASes cannot reach their DNS servers. But why? Does Facebook change the IP addresses of their DNS servers and other web endpoints constantly? Shoudn't Facebook IP addresses be cached by DNS resolvers?
Post reply on HN