Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding how Facebook disappeared from the internet

blog.cloudflare.com

Post reply on HN
Print view View original

111–120 of 280 posts

Re: Understanding how Facebook disappeared from the internet

#111

Post by lawwantsin17 »

DantesKite wrote: post #107
Ansil849 wrote: post #72

Is anyone else kinda put off by how Cloudflare keeps interjecting themselves into this situation? First with the Twitter posts and now this blogpost. They're a passive observer, not involved in this event, and yet they keep putting themselves in the middle of it. Cloudflare staff are not Facebook staff, they do not know how things actually went down to trigger the events that transpired. They are essentially doing gl…

No. I don’t mind at all. It’s a great marketing opportunity and they would be stupid not to take advantage of it.

You like it when people take advantage and kick a competitor when they're down? How ruthlessness has been normalized.

Re: Understanding how Facebook disappeared from the internet

#112

Post by _aj8o »

Ansil849 wrote: post #72

Is anyone else kinda put off by how Cloudflare keeps interjecting themselves into this situation? First with the Twitter posts and now this blogpost. They're a passive observer, not involved in this event, and yet they keep putting themselves in the middle of it. Cloudflare staff are not Facebook staff, they do not know how things actually went down to trigger the events that transpired. They are essentially doing gl…

So ... cloudflare has inserted themselves far more deeply into the discussion awhile back.

With the new DoH Chrome setting turned on by default (DNS-over-HTTPS), AFAIK they are now the default resolver bypassing your network settings.

[1] https://developers.cloudflare.com/1.1.1.1/encrypted-dns/dns-...

[2] https://duo.com/decipher/google-makes-dns-over-https-default...

Re: Understanding how Facebook disappeared from the internet

#113

Post by NikolaNovak »

Ansil849 wrote: post #72

Is anyone else kinda put off by how Cloudflare keeps interjecting themselves into this situation? First with the Twitter posts and now this blogpost. They're a passive observer, not involved in this event, and yet they keep putting themselves in the middle of it. Cloudflare staff are not Facebook staff, they do not know how things actually went down to trigger the events that transpired. They are essentially doing gl…

I kinda see your perspective; but I also mostly view them similarly to e.g. BackBlaze blogs - sure it can be viewed as taking smack about Samsung and Seagate drives failing, but I seem them as "things fail; what can we learn from it / here's our observations". And yes a little bit of self-advertising, but again, most tech blogs are, one way or another.

Re: Understanding how Facebook disappeared from the internet

#114

Post by Ansil849 »

NikolaNovak wrote: post #113
Ansil849 wrote: post #72

Is anyone else kinda put off by how Cloudflare keeps interjecting themselves into this situation? First with the Twitter posts and now this blogpost. They're a passive observer, not involved in this event, and yet they keep putting themselves in the middle of it. Cloudflare staff are not Facebook staff, they do not know how things actually went down to trigger the events that transpired. They are essentially doing gl…

I kinda see your perspective; but I also mostly view them similarly to e.g. BackBlaze blogs - sure it can be viewed as taking smack about Samsung and Seagate drives failing, but I seem them as "things fail; what can we learn from it / here's our observations". And yes a little bit of self-advertising, but again, most tech blogs are, one way or another.

It just comes off as crass. Next time Cloudflare's CDN eats itself and starts vomiting up private customer data, Facebook can do a blogpost titled 'Understanding how Cloudflare exposed the private information of untold numbers of its customers'.

Re: Understanding how Facebook disappeared from the internet

#115

Post by crmd »

14 wrote: post #103
unityByFreedom wrote: post #94

Earlier quoted context omitted.

Ah, so no playing oculus either?

Curious about this as well could not find anything on initial searches but I honestly hope so. The one reason I have not grabbed the oculus is because the Facebook requirement.

When I hear Facebook leadership talking about the VR vision, I’m reminded of IBM in the early 2000s encouraging employees to claim their second life avatars and setup cyber customer briefing centers.

Re: Understanding how Facebook disappeared from the internet

#116

Post by bagels »

foobarian wrote: post #74
Hikikomori wrote: post #48

Earlier quoted context omitted.

Restoring is just as simple as flipping the switch again, but access to that switch is another matter when your internal network is also down and you cannot even get access to your office or datacenters.

What I don't get is why there is no dead-man switch mechanism in place to roll back the configuration automatically unless someone confirms it positively. Kind of how screen resolution rolls back if you don't ack it. I used to always run a "(sleep 600; iptables -F) &" when messing with remote personal stuff just in case I lock myself out. I suppose with something like BGP it would be very difficult to get such a fall…

How long do you wait?

How do you know if you waited long enough to see negative effects (think caches, your own and caches of others).

Waiting too long with a bad config also costs you.

Re: Understanding how Facebook disappeared from the internet

#117

Post by WaitWaitWha »

WaitWaitWha wrote: post #59

This provides a good set of details (mostly educational) what happened up to but not including the how and why the BGP routes were withdrawn (who sent the UPDATE packets to the neighboring ASes?). The most "natural" occurrence that I can think of is best-path change. If a "better" route between AS is added, the now-second-best routes are withdrawn. Correct me if I am wrong, but there is no way of determining the sour…

Love to hear the reason for the downs.

Re: Understanding how Facebook disappeared from the internet

#118

Post by mnordhoff »

withinboredom wrote: post #45
tester756 wrote: post #42

Earlier quoted context omitted.

Why it takes this long?

Caching

Resolvers typically cache successful "does not exist" responses for no more than 1-3 hours. (And authoritative servers often have a lower negative TTL.)

(There's a corner case related to DNSSEC that can make it go higher, but that's being worked on, and isn't relevant here.)

In this situation, the nameservers were just down. I haven't done exhaustive research, but the resolvers I'm aware of cache that kind of thing for no more than 15 minutes.

Re: Understanding how Facebook disappeared from the internet

#119

Post by [deleted] »

Ansil849 wrote: post #101
hinkley wrote: post #100

Earlier quoted context omitted.

They've done this in previous outages and people didn't like it then, either. Generally big companies don't talk smack about other big companies having internet-wide issues, unless those issues are directly caused by the other company. For instance, when Google talked about Cloudbleed, which was when Cloudflare vomited millions of secrets all over Google's caching heirarchy and Google had to manually clean it up. I t…

I still remember Cloudflare's PR efforts to downplay Cloudbleed. A key reason why I meticulously avoid using any of their services.

[deleted]

Re: Understanding how Facebook disappeared from the internet

#120

Post by dilyevsky »

Ansil849 wrote: post #72

Is anyone else kinda put off by how Cloudflare keeps interjecting themselves into this situation? First with the Twitter posts and now this blogpost. They're a passive observer, not involved in this event, and yet they keep putting themselves in the middle of it. Cloudflare staff are not Facebook staff, they do not know how things actually went down to trigger the events that transpired. They are essentially doing gl…

Probably re-enabling it gradually to avoid hammering DNS servers too much due to client caches being totally cold
Post reply on HN