Getting everything back up again will probably be a nightmare. Imagine all the internal services trying to reach a consistent state after such a long outage.
Facebook-owned sites were down
751–760 of 1001 posts
Re: Facebook-owned sites were down
#752Earlier quoted context omitted.
I can't fathom how they didn't plan for this. In any business of size, you have to change configuration remotely on a regular basis, and can easily lock yourself out on a regular basis. Every single system has a local user with a random password that we can hand out for just this kind of circumstance...
Another Monday morning at a boring datacenter job, i bet they weren't even there yet at 830 when the phones started ringing.
Assuming anyone can actually look up the phone numbers to call.
Re: Facebook-owned sites were down
#753Earlier quoted context omitted.
There are various non-FB fallback measures, including IRC as a last-ditch method. The IRC fallback is usually tested once a year for each engineer.
I just heard from a contact that the fallback/backup IRC is also down.
Bet it was located at irc.facebook.com ;)
Joking aside, I can see how an IRC network has potential to be used in these situations. Maybe FAMANG should work together to set something like this up. The problem is, a single IRC server is not fail safe, but a network of multiple servers would just see a netsplit, in which case users would switch servers.
Also, I remember back in the IRCnet days using simply telnet to connect to IRCnet just for fun and sending messages, so its a very easy protocol that can be understood in a global desaster scenario (just the PING replys where annoying in telnet).
Re: Facebook-owned sites were down
#754Re: Facebook-owned sites were down
#755There is a global outage this morning starting from 8:00 AM. This list has Google, Tiktok, Zoom, Slack and of course FB products and services.
Re: Facebook-owned sites were down
#756Earlier quoted context omitted.
You mean the same problem as when GMail goes down and Googlers can't reach each other? I guess good decentralized public communication services could solve those issues for everybody.
I can assure you that Google has a procedure in place for that.
For those who don't know who he is: l9i would know this. Just clarifying that this is not an Internet nobody guessing.
Re: Facebook-owned sites were down
#757From this tweet: https://twitter.com/BlazejKrajnak/status/1445063232486531099
"Because of missing DNS records for http://Facebook.com, every device with FB app is now DDoSing recursive DNS resolvers. And it may cause overloading ..."
Re: Facebook-owned sites were down
#758Facebook being down makes me think of all of those small businesses who never built websites. They rely on traffic and publicity from their Facebook pages only. It's so important to diversify, such as building a website.
Well assuming they do that, likely they will host websites on AWS or Azure. Then AWS is down, what are you gonna do?
Move your resources to the one you didn't pick and point your domain registrar there.
Re: Facebook-owned sites were down
#759Earlier quoted context omitted.
I can't fathom how they didn't plan for this. In any business of size, you have to change configuration remotely on a regular basis, and can easily lock yourself out on a regular basis. Every single system has a local user with a random password that we can hand out for just this kind of circumstance...
Organizational complexity grows super-linearly; in general, the number of people a company can hire per unit time is either constant or grows linearly. Google once had a very quiet big emergency that was, ironically(1), initiated by one of their internal disaster-recovery tests. There's a giant high-security database containing the 'keys to the kingdom', as it were... Passwords, salts, etc. that cannot be represented…
That's not quite how it happened. ;)
We used this story as the opening of "Building Secure and Reliable Systems" (chapter 1). You can check it out for free at https://sre.google/static/pdf/building_secure_and_reliable_s... (size warning: 9 MB).
Re: Facebook-owned sites were down
#760Earlier quoted context omitted.
And there his account went poof, thanks for archiving.
They were quoted on multiple news sites including Ars Technica. I would imagine they were not authorized to post that information. I hope they don't lose their job. Shareholders and other business leaders I'm sure are much happier reporting this as a series of unfortunate technical failures (which I'm sure is part of it) rather than a company-wide organizational failure. The fact they can't physically badge in the pe…
They must have been moving very fast!