Facebook-owned sites were down • Hacker News Board

Re: Facebook-owned sites were down

#451

Post by rexreed » Mon, Oct 04, 2021, 5:40 PM UTC

kossTKR wrote: post #291

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

I am sure this is not what they specifically mean by fail fast and break things often.

Re: Facebook-owned sites were down

#452

Post by kaustubhvp » Mon, Oct 04, 2021, 5:40 PM UTC

alasdair_ wrote: post #403

kerng wrote: post #380

Earlier quoted context omitted.

Wondering how Facebook communicates now internally - most of their work streams likely depend on Facebooks systems which are all down. Can engineers and security teams even access prod systems anymore? Like, would "Bastion" hosts be reachable? Wonder if they use Signal and Slack now?

There are various non-FB fallback measures, including IRC as a last-ditch method. The IRC fallback is usually tested once a year for each engineer.

I just heard from a contact that the fallback/backup IRC is also down.

Re: Facebook-owned sites were down

#453

Post by aldanor » Mon, Oct 04, 2021, 5:41 PM UTC

apexalpha wrote: post #268

jliptzin wrote: post #215

Earlier quoted context omitted.

Of all the big tech companies Facebook is the only one where it can completely disappear overnight and my life would be completely unaffected (or possibly improved by not having to explain to people I don't use facebook, please email or text me your invitations rather than use messenger). If Google, Amazon, Netflix, Apple disappeared the story would be completely different.

Weird, because over here WhatsApp is ingrained into the social fabric of your life. Couldn't imagine ever going back to texting/iMessage.

After using Telegram, WhatsApp is a complete piece of garbage, if it disappeared from the face of the earth it would be sure for the best as people would move on to alternative messengers.

Re: Facebook-owned sites were down

#454

Post by pbhjpbhj » Mon, Oct 04, 2021, 5:41 PM UTC

bennyp101 wrote: post #346

kossTKR wrote: post #291

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

Interesting that they published stuff about their BGP setup and infrastructure a few months ago - maybe a little tweak to roll backs is needed. "... We demonstrate how this design provides us with flexible control over routing and keeps the network reliable. We also describe our in-house BGP software implementation, and its testing and deployment pipelines. These allow us to treat BGP like any other software componen…

Surely Facebook don't update routing systems between data centres (IIRC the situation) when they don't have people present to fix things if they go wrong? Or have an out-of-band connection (satellite, or dial-up (?), or some other alternate routing?).

I must be misunderstanding this situation here.

[Aside: I recall updating wi-fi settings on my laptop and first checking I had direct Ethernet connection working ... and that when I didn't have anything important to do (could have done a reinstall with little loss). Is that a reasonable analogy?]

Re: Facebook-owned sites were down

#455

Post by cecilpl2 » Mon, Oct 04, 2021, 5:41 PM UTC

kossTKR wrote: post #291

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

User has now deleted the update.

Re: Facebook-owned sites were down

#456

Post by georgeecollins » Mon, Oct 04, 2021, 5:43 PM UTC

drcongo wrote: post #66

Let's hope it's permanent.

The problem for me is Oculus. I really love their headset and I appreciate the investment Facebook has made in that.

I hate the stupid strategy tax that makes me have an FB account to use their headset, and has it go down when they have an outage. I hope they can learn from MSFT that "Facebook Everywhere" is ultimately a self defeating strategy.

Re: Facebook-owned sites were down

#457

Post by intricatedetail » Mon, Oct 04, 2021, 5:43 PM UTC

standardUser wrote: post #289

jliptzin wrote: post #215

Earlier quoted context omitted.

Of all the big tech companies Facebook is the only one where it can completely disappear overnight and my life would be completely unaffected (or possibly improved by not having to explain to people I don't use facebook, please email or text me your invitations rather than use messenger). If Google, Amazon, Netflix, Apple disappeared the story would be completely different.

Facebook is an unparalleled titan in the realm of advertising and WhatsApp is basically a utility-level communication system for a big chunk of the globe. Instagram is a key cultural driver of the Western world. You many not feel any direct firsthand consequences, but the overall impact would transform the world around you.

Facebook is an unparalleled titan in the realm of consumer manipulation

There I fixed it for you.

Re: Facebook-owned sites were down

#458

Post by rvnx » Mon, Oct 04, 2021, 5:43 PM UTC

mopierotti wrote: post #372

rvnx wrote: post #349

Earlier quoted context omitted.

I like how FB decided to send "ramenporn" as their spokesperson.

I'm worried about that person. I doubt Facebook will look kindly on breaking incident news being shared on reddit.

Well, seems like FB shutdowned his post...

Re: Facebook-owned sites were down

#459

Post by qnsi » Mon, Oct 04, 2021, 5:44 PM UTC

kossTKR wrote: post #291

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

he started deleting the comments

Re: Facebook-owned sites were down

#460

Post by iso1210 » Mon, Oct 04, 2021, 5:44 PM UTC

daper wrote: post #410

cotillion wrote: post #371

Earlier quoted context omitted.

So, does anyone know where to one can buy an LTE gateway with a serial port interface? Asking for a friend.

Our security team complained that we have some services like monitoring or SSH access to some Jump Hosts accessible without a VPN because VPN should be mandatory to access all internal services. I'm afraid once comply we could be in similar situation where Facebook is now...

But you have two independent VPNs right, using different technologies on different internet handoffs in very different parts of your network, right?