now if only tiktok would fail
Facebook-owned sites were down
891–900 of 1001 posts
Re: Facebook-owned sites were down
#892Earlier quoted context omitted.
Wondering how Facebook communicates now internally - most of their work streams likely depend on Facebooks systems which are all down. Can engineers and security teams even access prod systems anymore? Like, would "Bastion" hosts be reachable? Wonder if they use Signal and Slack now?
There are various non-FB fallback measures, including IRC as a last-ditch method. The IRC fallback is usually tested once a year for each engineer.
A lot of the core 'ops like' teams at FB use IRC on a daily basis.
When I worked there, I wasn't aware of any 'test once per year' concept or directive.
Of course, FB is a really big place, so things are different in different areas.
Re: Facebook-owned sites were down
#893Earlier quoted context omitted.
You mean the same problem as when GMail goes down and Googlers can't reach each other? I guess good decentralized public communication services could solve those issues for everybody.
I can assure you that Google has a procedure in place for that.
I unfortunately cannot edit the parent comment anymore but several people pointed out that I didn't back up my claim or provided any credentials so here they are:
Google has multiple independent procedures for coordination during disasters. A global DNS outage (mentioned in https://news.ycombinator.com/item?id=28751140) was considered and has been taken into account.
I do not attempt to hide my identity here, quite the opposite: my HN profile contains my real name. Until recently a part of my job was to ensure that Google is prepared for various disasterous scenarios and that Googlers can coordinate the response independently from Google's infrastructure. I authored one of the fallback communication procedures that would likely be exercised today if Google's network experienced a global outage. Of course Google has a whole team of fantastic human beings who are deeply involved in disaster preparedness (miss you!). I am pretty sure they are going to analyze what happened to Facebook today in light of Google's emergency plans.
While this topic is really fascinating, I am unfortunately not at liberty to disclose the details as they belong to my previous employer. But when I stumble upon factually incorrect comments on HN that I am in a position to correct, why not do that?
Re: Facebook-owned sites were down
#894Earlier quoted context omitted.
I can't fathom how they didn't plan for this. In any business of size, you have to change configuration remotely on a regular basis, and can easily lock yourself out on a regular basis. Every single system has a local user with a random password that we can hand out for just this kind of circumstance...
Most likely they did plan for this. Then, something happened that the failsafe couldn't handle. E.g. if something overwrites /etc/passwd, having a local user won't help. I'm not saying that specific thing happened here -- it's actually vanishingly unlikely -- but your plan can't cover every contingency.
Agreed, it’s also worth mentioning that at the end of every cloud is real physical hardware, and that is decidedly less flexible than cloud, if you locked yourself out of a physical switch or router you have many fewer options.
Re: Facebook-owned sites were down
#895FB seems to be finally loading for me, after nearly 6 hours.
This will be a highly discussed topic for a bit.
Re: Facebook-owned sites were down
#896Its alive! drill @1.1.1.1 www.facebook.com ;; ->>HEADER ;; ANSWER SECTION: www.facebook.com. 3401 IN CNAME star-mini.c10r.facebook.com. star-mini.c10r.facebook.com. 3403 IN A 31.13.72.36
Kinda sorta. There are four DNS servers for Facebook: 129.134.30.12, 129.134.31.12, 185.89.218.12, and 185.89.219.12.
Of those, only 185.89.219.12 is up right now (Edit All four DNS servers are now up). For people who want to add Facebook to hosts.txt, the A record (IP) I’m getting right now is 157.240.11.35 (it was 31.13.70.36)
Re: Facebook-owned sites were down
#897Earlier quoted context omitted.
LOL - score one against building out all tooling internally (a la Amazon and apparently Facebook too)
The rate at which some amazon services lately go done because other AWS services went down proves that this is an unsustainable house of cards anyways.
Netflix knows how to build on top of a house of cards.
Re: Facebook-owned sites were down
#898If it bleeds we can kill it!
Re: Facebook-owned sites were down
#899I'm curious if this extended outage will do anything to curb the dopamine addition caused by facebook.
For example, will FB addicts experience a day of repeated failed attempts to get their FB fix, which will then condition them to stop trying.
Re: Facebook-owned sites were down
#900Posting this comment will be like farting into a hurricane, but here goes. Company like Facebook has a serious problem and their stock drops ... precipitously. CEO of said company instead of selling their equity in their company has taken out loans against their equity in order to decrease their tax burden and cash in on the value of their equity. What amount of decrease would cause a margin call from lenders for the…
Margin calls don't really exist as far as loans are concerned. Once you have agreed collateral, and an agreed schedule of payment, you only get in trouble if you miss a payment, regardless of how the collateral fluctuates in value.