Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook-owned sites were down

facebook.com

Post reply on HN
Print view View original

881–890 of 1001 posts

Re: Facebook-owned sites were down

#881

Post by tablespoon »

cududa wrote: post #712
okwubodu wrote: post #682

Earlier quoted context omitted.

I don’t know how true it is but a few reports claim employees can’t get into the building with their badges.

I remember my first time having a meeting at Facebook and observing none of the doors had keyholes and thinking "hope their badge system never goes down"

> I remember my first time having a meeting at Facebook and observing none of the doors had keyholes and thinking "hope their badge system never goes down"

Every internet-connected physical system needs to have a sensible offline fallback mode. They should have had physical keys, or at least some kind of offline RFID validation (e.g. continue to validate the last N badges that had previously successfully validated).

Re: Facebook-owned sites were down

#882

Post by skeeter2020 »

cududa wrote: post #712
okwubodu wrote: post #682

Earlier quoted context omitted.

I don’t know how true it is but a few reports claim employees can’t get into the building with their badges.

I remember my first time having a meeting at Facebook and observing none of the doors had keyholes and thinking "hope their badge system never goes down"

maybe they're open by default, like old 7-11 stores when they went 24hrs and had no locks on the doors :)

Re: Facebook-owned sites were down

#883

Post by shreezus »

It's quite a coincidence for this to coincide with the whistleblower report + rumors of Peter Thiel (perhaps via Palantir?) involved in leveraging FB for the 2022 midterm elections.

I'm not suggesting that this is the case, but a failure of this scale (with internal systems also down) could allow scrubbing of evidence without leaving traces.

Re: Facebook-owned sites were down

#884

Post by still_grokking »

l9i wrote: post #705
still_grokking wrote: post #669

Earlier quoted context omitted.

You mean the same problem as when GMail goes down and Googlers can't reach each other? I guess good decentralized public communication services could solve those issues for everybody.

I can assure you that Google has a procedure in place for that.

I've read here on HN that exactly this was the issue as they had one of the bigger outages (I think it was due to some auth service failure) and GMail didn't accept incoming mail.

Re: Facebook-owned sites were down

#885

Post by drdeadringer »

suyash wrote: post #686
pmlnr wrote: post #314

Earlier quoted context omitted.

> the people with physical access is separate from the people with knowledge of [...] Welcome to the brave new world of troubleshooting. This will seriously bite us one day.

folks with physical access are also denied. source - https://twitter.com/YourAnonOne/status/1445100431181598723

IT: "Please do this fix."

Person 1: "I can't, I don't have physical access."

IT: "Please do this fix."

Person 2: "I can't, I don't have digital access."

Why? It's [IT's?] policy.

Re: Facebook-owned sites were down

#886

Post by mrep »

ThinkBeat wrote: post #846
kerng wrote: post #380

Earlier quoted context omitted.

Wondering how Facebook communicates now internally - most of their work streams likely depend on Facebooks systems which are all down. Can engineers and security teams even access prod systems anymore? Like, would "Bastion" hosts be reachable? Wonder if they use Signal and Slack now?

I haven't worked for a FAANG but it would be unthinkable that FB does not have backup measures in place for communications entirely outside of Facebook. Hmm well I mean for key people, ops and so on. Not for every employee. Only a few people need that type of access, and they should have it ready. They need to bring more people there should be an easy way to do it. Maybe the internal FB Messenger app has a slide butt…

> Maybe the internal FB Messenger app has a slide button to switch to the backup network for those in need.

Having worked for 2 FAANG companies, I can tell you most core services like which FB Messenger would be using internal database services and relying on those which would be ineffective in a case like this as it would not work and the engineering cost to design them to support an external database would be a lot more than just paying for like 5 different external backup products for your SRE team.

Re: Facebook-owned sites were down

#887

Post by gunshai »

Posting this comment will be like farting into a hurricane, but here goes.

Company like Facebook has a serious problem and their stock drops ... precipitously. CEO of said company instead of selling their equity in their company has taken out loans against their equity in order to decrease their tax burden and cash in on the value of their equity.

What amount of decrease would cause a margin call from lenders for the forced sale of said equity and subsequently the loss of majority stake in their own company? Now obviously only the lenders know this information and assuming I have the rough order of operations correct.

Could this be a potential chink in the armor of founders / CEOs / anyone who takes out low interest loans against the equity they hold in their company? Maybe my understanding of this is too simplified.

Re: Facebook-owned sites were down

#888

Post by londons_explore »

cududa wrote: post #712
okwubodu wrote: post #682

Earlier quoted context omitted.

I don’t know how true it is but a few reports claim employees can’t get into the building with their badges.

I remember my first time having a meeting at Facebook and observing none of the doors had keyholes and thinking "hope their badge system never goes down"

Breaking the glass to get in to fix the service is totally a good business move.

A few hundred bucks of glass Vs a billion wiped off the share price if the service is down for a day and all the user's go find alternatives.

Re: Facebook-owned sites were down

#889

Post by xorcist »

mov31tmov31t wrote: post #347
snickersnee11 wrote: post #307

Earlier quoted context omitted.

Just imagine the amount of stress on this people, hope the money really worth it.

It shouldn't be too stressful. Well-managed companies blame processes rather than people, and have systems set up to communicate rapidly when large-scale events occur. It can be sort of exciting, but it's not like there is one person typing at a keyboard with a hundred managers breathing down their neck. These resolutions are collaborative, shared efforts.

> Well-managed companies blame processes rather than people,

We're six hours without a route to their network, and counting. I think we can safely rule out well-managed.

Re: Facebook-owned sites were down

#890

Post by alexvoda »

Animats wrote: post #628

There's still no connectivity to Facebook's DNS servers: > traceroute a.ns.facebook.com traceroute to a.ns.facebook.com (129.134.30.12), 30 hops max, 60 byte packets 1 dsldevice.attlocal.net (192.168.1.254) 0.484 ms 0.474 ms 0.422 ms 2 107-131-124-1.lightspeed.sntcca.sbcglobal.net (107.131.124.1) 1.592 ms 1.657 ms 1.607 ms 3 71.148.149.196 (71.148.149.196) 1.676 ms 1.697 ms 1.705 ms 4 12.242.105.110 (12.242.105.110)…

Can someone explain why it is also down when trying to access it via Tor using its onion address: http://facebookwkhpilnemxj7asaniu7vnjjbiltxjqhye3mhbshg7kx5t...

Or when trying ips directly: https://www.lifewire.com/what-is-the-ip-address-of-facebook-...

I would have expected a DNS issue to not affect either of these.

I can understand the onionsite being down if facebook implemented it the way a thirdparty would (a proxy server accessing facebook.com) instead of actually having it integrated into its infrastructure as a first class citizen.

Post reply on HN