Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding how Facebook disappeared from the internet

blog.cloudflare.com

Post reply on HN
Print view View original

191–200 of 280 posts

Re: Understanding how Facebook disappeared from the internet

#191

Post by acomjean »

gorgoiler wrote: post #51
throwdecro wrote: post #6

This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?

At a guess: reconnecting traffic at the billions-of-people scale has the potential for finding all sorts of weird behaviours. For all we know, it has been connected and disconnected 10x already during the outage, with each reconnect overwhelming some new, deeper level of the system each time. Reconnect and the GLBs fall apart under load as the entire world’s cadre of recursive resolvers hit you. Fix that. Reconnect a…

Reminds me of when I was working at a startup we DOSed ourselves. We had devices that monitored power minute by minute. A load balancer was mis configured and we went down for a some hours. We came back up and the devices all saw that and flooded us with all the data they’d been storing since we were down.. down again… bring it up and down again…. we needed a better fix. I was up till 2 am with the other developer coding a fix.. the next morning we talked to the CEO (CTO was on vacation)who told us upgrading the was database was going to be too expensive… good times. We did get a firmware fix ( to be honest we were running out of cash..)

I can’t imagine trying to restart something as big as facebook…

Re: Understanding how Facebook disappeared from the internet

#192

Post by Tostino »

paul7986 wrote: post #161
crmd wrote: post #115

Earlier quoted context omitted.

When I hear Facebook leadership talking about the VR vision, I’m reminded of IBM in the early 2000s encouraging employees to claim their second life avatars and setup cyber customer briefing centers.

Exactly ... like I am going to strap on a headset and work with my co-workers memojis for 8 hours or even less. Laughable and VR has been around since the 90s with the same form factor .. it hasn't evolved as it will always be strap this bulky thing to your head & isolate yourself from the world. Maybe it will catch on for videogames but to co-work in a virtually reality world HA. AR Glasses at least their form facto…

Eh, I could see myself doing development in that type of environment if the tools were get better. I've worked remote for the past 6 years, so it wouldn't quite be the same as in an office environment though.

Re: Understanding how Facebook disappeared from the internet

#193

Post by geocrasher »

How I explained this to friends and family: Imagine that the only way to get to Los Angeles is to use a GPS enabled device. All of the maps know how to get there from anywhere. But imagine that every GPS took Los Angeles off the map. Los Angeles is still there, but nobody knows how to get there. That's what happened to Facebook. We don't yet know why, however.

Re: Understanding how Facebook disappeared from the internet

#194

Post by Jamie9912 »

Wolfspirit wrote: post #35
interestica wrote: post #28

Earlier quoted context omitted.

No matter what time of year it is, people tend to use 'EST' for 'Eastern Time' even when we might be in Eastern Daylight Time rather than Standard. It's especially annoying when dealing with multiple countries that may or may not be using Daylight Saving Time.

Even google isn't quite sure about the summer time. Not sure if that is just a Google German thing... A few weeks ago I tried to find out what the current time in CET is. Asking google for "CET" gave me: "23:27 CET". Asking google for "CET time" (I know that "time" is twice in this case) gave me "00:27 CET". The last one is wrong and should be CEST or even more correct would be just the same result for CET as I asked…

In this case, Google is providing a wrong answer to not confuse people. People don't understand that a time zone can exist but not be observed at the same time

Re: Understanding how Facebook disappeared from the internet

#195

Post by goatlover »

nimbius wrote: post #64
throwdecro wrote: post #6

This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?

Unpopular opinion but I think a talent exodus and turnover are likely at fault here as well. The people who stood up this juggernaut are no longer here, and Facebooks ability to consistently attract talent that is competent enough to maintain such a formidable beast is hampered by repetitive revelations that it operates at the net-loss of humanity as a whole. Facebooks no longer an innovator, just a mining operation…

Isn’t Facebook huge in some Asian countries? Supposedly 3.5 billion people use one or more of its services. WhatsApp, Instagram and Oculus certainly aren’t just used by the elderly.

Re: Understanding how Facebook disappeared from the internet

#196

Post by selcuka »

threeseed wrote: post #178
selcuka wrote: post #143

Earlier quoted context omitted.

They state that [1] this only happens if you are using more than 3 Page Rules that the free plan allows, which is fair. The alternative would be to randomly delete some of your page rules, which is worse: > If you do not want to be charged for the additional page rules, you should ensure you only have 3 active rules before you downgrade from Pro to Free. [1] https://community.cloudflare.com/t/i-canceled-my-subscripti…

The alternative is to ask the user what they want to do instead of defaulting to billing you. Or provide the option of refunds. And pretty sure that explanation is incorrect or has changed. I have never used Page Rules and was still billed.

> The alternative is to ask the user what they want to do

Fair point.

Re: Understanding how Facebook disappeared from the internet

#197

Post by js2 »

Those short TTLs that FB likely has on its DNS records probably bit it in the ass today. If it had longer TTLs, caching would have helped it. Curious though that recursive resolvers won't serve an expired cached record when they can't reach the authoritative server. (I know that unbound can be configured to do so, but not sure about others.)

Re: Understanding how Facebook disappeared from the internet

#198

Post by dijit »

throwdecro wrote: post #6

This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?

The kind of tail you’re talking about is baked into DNS at least.

I don’t know enough about BGP to make an informed decision; but at the point the outage is noticed it’s entirely possible that the system has been unavailable for quite some time already.

Re: Understanding how Facebook disappeared from the internet

#199

Post by withinboredom »

mnordhoff wrote: post #118
withinboredom wrote: post #45

Earlier quoted context omitted.

Caching

Resolvers typically cache successful "does not exist" responses for no more than 1-3 hours. (And authoritative servers often have a lower negative TTL.) (There's a corner case related to DNSSEC that can make it go higher, but that's being worked on, and isn't relevant here.) In this situation, the nameservers were just down. I haven't done exhaustive research, but the resolvers I'm aware of cache that kind of thing f…

If there’s a chain of caches 3 deep, a 15 minute cache on bad responses will take 45 minutes to clear.
Post reply on HN