Live data from Hacker News

Understanding how Facebook disappeared from the internet

blog.cloudflare.com

251–260 of 280 posts

Re: Understanding how Facebook disappeared from the internet

#251

Is anyone else kinda put off by how Cloudflare keeps interjecting themselves into this situation? First with the Twitter posts and now this blogpost. They're a passive observer, not involved in this event, and yet they keep putting themselves in the middle of it. Cloudflare staff are not Facebook staff, they do not know how things actually went down to trigger the events that transpired. They are essentially doing gl…

YES! THIS!

Re: Understanding how Facebook disappeared from the internet

#252

Earlier quoted context omitted.

They've done this in previous outages and people didn't like it then, either. Generally big companies don't talk smack about other big companies having internet-wide issues, unless those issues are directly caused by the other company. For instance, when Google talked about Cloudbleed, which was when Cloudflare vomited millions of secrets all over Google's caching heirarchy and Google had to manually clean it up. I t…

I still remember Cloudflare's PR efforts to downplay Cloudbleed. A key reason why I meticulously avoid using any of their services.

ty

Re: Understanding how Facebook disappeared from the internet

#253
post #176

Earlier quoted context omitted.

I thought of this at first, but after reading FB official "tech" explanation on the outage ( https://engineering.fb.com/2021/10/04/networking-traffic/out... ) I very much prefer Cloudflare's take on this.

I'dnt expect fb employees to write up about BGP while they are in middle of a fire drill.

or an actual fire?

Re: Understanding how Facebook disappeared from the internet

#254
post #186
post #182

Earlier quoted context omitted.

Yes well lets have this outage again in that flavour in say... 11 months?

Taking a bet that complex systems will fail is usually free money :). Having been a part of Google the only thing more awe inspiring than the sheer complexity of production is the fact that it all worked so well. This is not a dig at current Googlers but entropy is cruel and uncaring. Perhaps parts of the stack which have been kept fit & fresh in people's minds due to constant rewrites will last longer but there are…

Google (at least the good parts) has a pretty good culture of documentation, which helps a bit. While it drove me batty to need to spend a month writing up 10 page specs and getting sign off from directors for a minor feature that nobody would see and would take 3 days to code up, it was nice to be able to trace through the historical evolution of abandoned features when trying to figure out how they worked.

No idea if Facebook is similar.

Re: Understanding how Facebook disappeared from the internet

#255

Question about the WARP map - I assume the grey countries are places where Cloudflare doesn't have any presence, but what about Egypt/Oman? Why are they green? And why is Australia orange and not red?

We didn't have enough data from those locations.

Egypt has over 100 million people. Facebook/WhatsApp is quite popular in that part of the world, so I would image it had significant traffic..

Re: Understanding how Facebook disappeared from the internet

#256

Earlier quoted context omitted.

Given my experience with DNS issues, I am guessing that they are running into dependencies along the way that assume/require DNS be available to function.

Yeah as part of my job I often have to work with our DNS team to provision say a subdomain or get some domain verified. They’ve got like…three people…trying to service thousands of teams across the enterprise. I do not envy their job at all.

Your guys need BP Diamond IP http://www.diamondipam.com

It's an IP address management (IPAM) solution that also just happens to be a fantastic, federated (if you want) DNS management system too. Indeed a previous org I worked at bought it strictly to tame the DNS beast - local sys admins could control DNS for their subnets but not affect anything else. If we wanted, we could have had approval processes on top of the change requests - the system supported that too.

I think the security teams finally woke up to the IP address management functionality and were slowly starting to integrate that into the rest of the infrastructure - but I was leaving around then. It was a fantastic system. One of the best hierarchical role-based access control systems in an application I have ever seen; the granularity was amazing yet it was easy to understand/administer. Not an easy trick!

Re: Understanding how Facebook disappeared from the internet

#258
post #242
post #224

Earlier quoted context omitted.

So much of the internet is about translating addresses from one format to another: Human readable to IP: DNS Private IP to Public IP: NAT MAC to Private IP: ARP Public IP to AS: BGP With a block chain DNS it may be possible to replace all of them with a single protocol, but it may well turn out to be an academic exercise.

> So much of the internet is about translating addresses from one format to another Most of computing is about mapping A to B and maintaining databases of those mappings :-)

Thanks for not pushing back too hard on my (elderly?), ignorance, and giving me a chance to catch up with the current state of the art. This looks very interesting, and I am looking forward to spending time diving in for no more than a sense of curiosity. Thank you.

Re: Understanding how Facebook disappeared from the internet

#259

How can you explain yesterday's outage (Facebook, Instagram, WhatsApp) to your parents? You are feeling hungry and went to food court. The food court (open area) has a lot of options. You sit down in front of Domino's (Facebook), since you want to eat garlic bread. Now, you can't order from the counter directly. The waiter will come to your seat and ask for the order. You ordered garlic bread from the waiter, but the…

I'd just say that you had an address for Facebook in your address book. The page somehow vanished and you don't know their address any more. So you start phoning other people and knocking on their door to try and find what their address is. Everyone else is doing this and no one knows what their address is. So you've got millions of people phoning each other and knocking on doors. Facebook being down was already an i…

If your parent is startrek fan, tell him "Commander Data" transmitted a "sleep" command to BGP (Borg Gateway Protocol) collective.

As result, the borg collectives can access to network, even 7of9 can't enter the campus.

Re: Understanding how Facebook disappeared from the internet

#260
post #224

It's been 20+ years since I had reason to know the internals of BGP, but I still carry a slight nervousness whenever I think about it. In many ways, it's beautiful in that it allows Autonomous Systems to come together and independently make the Internet. However, I always felt it was brittle in that it relied every AS to have edge routers aware of the entire AS routing map, and so when those routers went down hard, w…

So much of the internet is about translating addresses from one format to another: Human readable to IP: DNS Private IP to Public IP: NAT MAC to Private IP: ARP Public IP to AS: BGP With a block chain DNS it may be possible to replace all of them with a single protocol, but it may well turn out to be an academic exercise.

Problem is the blockchain stuff usually ends up being computationally expensive for what it's trying to do.

I could maybe see a use case for DNS but then either each computer has to have the full blockchain locally or rely on providers to do that for them in which case we're basically at the same model we have now (with root servers and DNS servers).

Obviously you don't want to publish private IPs, etc. to other computers... so inside a local network you're not buying yourself much.

For routing tables I could see more of a use case for blockchain technology since you can actually verify the routing tables and prevent BGP spoofing, etc.

Post reply on HN