Live data from Hacker News

Facebook-owned sites were down

facebook.com

611–620 of 1001 posts

Re: Facebook-owned sites were down

#611
post #169

Is it just me or HN also feels kinda laggy?

Yep. I am the developer of HN client HACK for iOS and Android and a bunch of users emailed me asking if my app was broken. Looks like something bigger is afoot.

Something's wrong with your app. It's not working at all, while Harmonic works perfectly.

Re: Facebook-owned sites were down

#612

Earlier quoted context omitted.

Also, equally important to note, there was a massive expose on FaceBook yesterday that is reverberating across social media and news networks, and today, when I tried to make a post including the tag #deletefacebook, my post mysteriously could not be published and the page refreshed, mysteriously wiping my post... This is possibly the equivalent of a corporate watergate if you ask me... Just my personal opinion as a…

So what you're saying is facebook... deleted itself? The singularity is happening. It realized it would end society, so it ended itself.

They decided that they publish too much misinformation and self censored ;)

Re: Facebook-owned sites were down

#614

Earlier quoted context omitted.

General tip: If HN is being laggy and you're determined you want to waste some time here, open it in a private window. HN works extremely quickly if it doesn't know who you are.

Wow this really works, thank you. What actually is the reason for it being much faster in a private window? Is there so much tracking going on in a normal window?

its faster because the pages are cached, they are effectively static. It's slower when logged in because the pages are created dynamically as it has your username, tracking favourites, upvotes etc, and much of it cannot be quickly cached.

Re: Facebook-owned sites were down

#616
Reminds me of a story Jack Fresco use to tell were financial workers were unable to get to work because a bridge was not usable. People were worried about terrible consequences if all these important people were unable to do their work. To their surprise life just continued as if nothing changed.

Re: Facebook-owned sites were down

#617

Earlier quoted context omitted.

I can't fathom how they didn't plan for this. In any business of size, you have to change configuration remotely on a regular basis, and can easily lock yourself out on a regular basis. Every single system has a local user with a random password that we can hand out for just this kind of circumstance...

Organizational complexity grows super-linearly; in general, the number of people a company can hire per unit time is either constant or grows linearly. Google once had a very quiet big emergency that was, ironically(1), initiated by one of their internal disaster-recovery tests. There's a giant high-security database containing the 'keys to the kingdom', as it were... Passwords, salts, etc. that cannot be represented…

Thanks for telling this story as it was more amusing than my experiences of being locked in a security corridor with a demagnetised access card, looooong ago.

Re: Facebook-owned sites were down

#618
post #317

Is anyone else seeing knock-on effects at the other major public DNS providers? I'm seeing nslookups sent to 4.2.2.2 and 8.8.8.8 intermittently timeout if the hostname does not belong to a major website. CloudFlare DNS (1.1.1.1) doesn't appear to be impacted. For example: [root@app ~]# nslookup downforeveryoneorjustme.com 4.2.2.2 ;; connection timed out; trying next origin ;; connection timed out; no servers could be…

See this thread (with replies):

>Now, here's the fun part. @Cloudflare runs a free DNS resolver, 1.1.1.1, and lots of people use it. So Facebook etc. are down... guess what happens? People keep retrying. Software keeps retrying. We get hit by a massive flood of DNS traffic asking for http://facebook.com

https://twitter.com/jgrahamc/status/1445066136547217413

>Our small non profit also sees a huge spike in DNS traffic. It’s really insane.

https://twitter.com/awlnx/status/1445072441886265355

>This is frontend DNS stats from one of the smaller ISPs I operate. DNS traffic has almost doubled.

https://twitter.com/TheodoreBaschak/status/14450732299707637...

Re: Facebook-owned sites were down

#619

Earlier quoted context omitted.

The account has been deleted as well.

What are they afraid of? While they are sharing information that's internal/proprietary to the company, it isn't anything particularly sensitive and having some transparency into the problem is good for everyone. Who'd want to work for a company that might take disciplinary action because an SRE posted a reddit comment to basically say "BGP's down lol" - If I was in charge I'd give them a modest EOY bonus for being h…

FB takes confidentiality very seriously. He crossed a major red line.

Re: Facebook-owned sites were down

#620
post #493

Earlier quoted context omitted.

And there his account went poof, thanks for archiving.

They were quoted on multiple news sites including Ars Technica. I would imagine they were not authorized to post that information. I hope they don't lose their job. Shareholders and other business leaders I'm sure are much happier reporting this as a series of unfortunate technical failures (which I'm sure is part of it) rather than a company-wide organizational failure. The fact they can't physically badge in the pe…

> an organization that hasn't actually thought through all its failure modes

Move Fast and Break Things!

Post reply on HN