Live data from Hacker News

Facebook-owned sites were down

facebook.com

401–410 of 1001 posts

Re: Facebook-owned sites were down

#401
post #363

Earlier quoted context omitted.

Unfortunately Whatsapp replaced texting for around 80 % of the world.

Not a fan of FB, but the main reason for WhatsApp's success was SMS sucking hairy balls.

It didn't help that telecoms used to use SMS as an extreme profit center. I don't think WhatsApp would have taken over the way it did if SMS was always included in all plans for free. This is similar to the way "local" long distance used to be such a racket.

Re: Facebook-owned sites were down

#402

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

> Even in the biggest of organizations, they still have to wait for somebody to race down to the datacenter and plug his laptop into a router. I love this comment.

Imagine having the a huge portion of the digital world internationally riding on your shoulders...

Re: Facebook-owned sites were down

#403
post #380

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

Wondering how Facebook communicates now internally - most of their work streams likely depend on Facebooks systems which are all down. Can engineers and security teams even access prod systems anymore? Like, would "Bastion" hosts be reachable? Wonder if they use Signal and Slack now?

There are various non-FB fallback measures, including IRC as a last-ditch method. The IRC fallback is usually tested once a year for each engineer.

Re: Facebook-owned sites were down

#404

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

Facebook is likely scrambling private jets as we speak to get the right people to the right places.

Reminds me of that episode in Mr Robot

Re: Facebook-owned sites were down

#406
post #314

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

> the people with physical access is separate from the people with knowledge of [...] Welcome to the brave new world of troubleshooting. This will seriously bite us one day.

I can't fathom how they didn't plan for this. In any business of size, you have to change configuration remotely on a regular basis, and can easily lock yourself out on a regular basis. Every single system has a local user with a random password that we can hand out for just this kind of circumstance...

Re: Facebook-owned sites were down

#407

Facebook down, WhatsApp down, but Signal still works. Time for a change? EDIT: Yes, Signal is not federated, but that's what people are at least ready to consider as a WhatsApp alternative. I also created Matrix / Element account, and had 0 contacts using it already.

Signal had their own share of downtime. How about going to a federated system instead of repeating the same mistakes? ( https://matrix.org )

But still a part would be down, if a server has an outage. How about a system, where every device that is used for chatting is a server at the same time? I wonder whether something like that already exists. Bundle it together with bigger servers to handle the load. If the bigger servers experience outages, the service can still continue, although a bit slower

Re: Facebook-owned sites were down

#408

Earlier quoted context omitted.

> Even in the biggest of organizations, they still have to wait for somebody to race down to the datacenter and plug his laptop into a router. I love this comment.

Imagine having the a huge portion of the digital world internationally riding on your shoulders...

[deleted]

Re: Facebook-owned sites were down

#409
post #273

Earlier quoted context omitted.

Of all the big tech companies Facebook is the only one where it can completely disappear overnight and my life would be completely unaffected (or possibly improved by not having to explain to people I don't use facebook, please email or text me your invitations rather than use messenger). If Google, Amazon, Netflix, Apple disappeared the story would be completely different.

All of the big tech companies you mentioned could completely disappear overnight and my life would be completely unaffected or possibly improved.

si es muy posible, de otro lado se daría la oportunidad a empresas mas cercanas con la gente y que les paguen por los usuarios por los datos. finalmente los usuarios son su activo para generar muy importantes ingresos, estaría muy bien que compartan sus beneficios!

Re: Facebook-owned sites were down

#410

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

So, does anyone know where to one can buy an LTE gateway with a serial port interface? Asking for a friend.

Our security team complained that we have some services like monitoring or SSH access to some Jump Hosts accessible without a VPN because VPN should be mandatory to access all internal services. I'm afraid once comply we could be in similar situation where Facebook is now...
Post reply on HN