Facebook-owned sites were down • Hacker News Board

Re: Facebook-owned sites were down

#561

Post by outworlder » Mon, Oct 04, 2021, 6:33 PM UTC

jfrunyon wrote: post #406

pmlnr wrote: post #314

Earlier quoted context omitted.

> the people with physical access is separate from the people with knowledge of [...] Welcome to the brave new world of troubleshooting. This will seriously bite us one day.

I can't fathom how they didn't plan for this. In any business of size, you have to change configuration remotely on a regular basis, and can easily lock yourself out on a regular basis. Every single system has a local user with a random password that we can hand out for just this kind of circumstance...

> I can't fathom how they didn't plan for this

Maybe because they were planning for a million other possible things to go wrong, likely with higher probability than this. And busy with each day's pressing matters.

Re: Facebook-owned sites were down

#562

Post by hamburglar » Mon, Oct 04, 2021, 6:34 PM UTC

pmlnr wrote: post #489

formerly_proven wrote: post #420

Earlier quoted context omitted.

This sounds like something that might have been done with security in mind. Although generally speaking, remote hands don't have to be elite hackors.

Have you ever tried to remotely troubleshoot THROUGH another person?!

My company runs copies of all our internal services in air-gapped data centers for special customers. The operators are just people with security clearance who have some technical skills. They have no special knowledge of our service inner workings. We (the dev team) aren’t allowed to see screenshots or get any data back. So yeah, I have done that sort of troubleshooting many times. It’s very reminiscent of helping your grandma set up her printer over the phone.

Re: Facebook-owned sites were down

#563

Post by chadlavi » Mon, Oct 04, 2021, 6:34 PM UTC

I recognize that for WhatsApp users around the globe this is probably more than an inconvenience, but the rest of humanity is getting something of a reprieve here.

Re: Facebook-owned sites were down

#564

Post by glenneroo » Mon, Oct 04, 2021, 6:34 PM UTC

adamredwoods wrote: post #525

larntz wrote: post #312

Earlier quoted context omitted.

This tweet seems to confirm it is a bgp issue... https://twitter.com/GossiTheDog/status/1445063880963674121?s...

Cloudflare also confirmed it: https://twitter.com/jgrahamc/status/1445068309288951820 Also, the Domain name is for sale??? https://whois.domaintools.com/facebook.com

Weird banner at the top, seems like false advertising as it says a couple lines down: Expires on 2030-03-29

Re: Facebook-owned sites were down

#565

Post by hnaccy » Mon, Oct 04, 2021, 6:34 PM UTC

shadowgovt wrote: post #554

jfrunyon wrote: post #406

Earlier quoted context omitted.

I can't fathom how they didn't plan for this. In any business of size, you have to change configuration remotely on a regular basis, and can easily lock yourself out on a regular basis. Every single system has a local user with a random password that we can hand out for just this kind of circumstance...

Organizational complexity grows super-linearly; in general, the number of people a company can hire per unit time is either constant or grows linearly. Google once had a very quiet big emergency that was, ironically(1), initiated by one of their internal disaster-recovery tests. There's a giant high-security database containing the 'keys to the kingdom', as it were... Passwords, salts, etc. that cannot be represented…

what if the executive had been pick-pocketed

Re: Facebook-owned sites were down

#566

Post by tedmiston » Mon, Oct 04, 2021, 6:34 PM UTC

deadalus wrote: post #225

eeegnu wrote: post #199

Earlier quoted context omitted.

Probably people flooding in to see if anyone knows why things are down. Even Google speed test was down, presumably from too many people testing if it's their internet that's at issue.

https://www.speedtest.net is down too

working for me

Re: Facebook-owned sites were down

#567

Post by yawnxyz » Mon, Oct 04, 2021, 6:35 PM UTC

IceWreck wrote: post #342

kossTKR wrote: post #291

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

> Even in the biggest of organizations, they still have to wait for somebody to race down to the datacenter and plug his laptop into a router. I love this comment.

for something as distributed as Facebook, do multiple somebodys all have to race down each individual datacenter and plug their laptops into the routers?

As someone with no experience in this, it sounds like a terrifying situation for the admins...

Re: Facebook-owned sites were down

#568

Post by swayson » Mon, Oct 04, 2021, 6:36 PM UTC

treesknees wrote: post #523

Ueland wrote: post #493

Earlier quoted context omitted.

And there his account went poof, thanks for archiving.

They were quoted on multiple news sites including Ars Technica. I would imagine they were not authorized to post that information. I hope they don't lose their job. Shareholders and other business leaders I'm sure are much happier reporting this as a series of unfortunate technical failures (which I'm sure is part of it) rather than a company-wide organizational failure. The fact they can't physically badge in the pe…

> I hope they don't lose their job.

FB has such poor integrity, I'd not be surprised if they take such extreme measures.

Re: Facebook-owned sites were down

#569

Post by polote » Mon, Oct 04, 2021, 6:36 PM UTC

treesknees wrote: post #523

Ueland wrote: post #493

Earlier quoted context omitted.

And there his account went poof, thanks for archiving.

They were quoted on multiple news sites including Ars Technica. I would imagine they were not authorized to post that information. I hope they don't lose their job. Shareholders and other business leaders I'm sure are much happier reporting this as a series of unfortunate technical failures (which I'm sure is part of it) rather than a company-wide organizational failure. The fact they can't physically badge in the pe…

> an organization that hasn't actually thought through all its failure modes

Thinking about any potential things that can happen is impossible

Re: Facebook-owned sites were down

#570

Post by TomSwirly » Mon, Oct 04, 2021, 6:36 PM UTC

standardUser wrote: post #289

jliptzin wrote: post #215

Earlier quoted context omitted.

Of all the big tech companies Facebook is the only one where it can completely disappear overnight and my life would be completely unaffected (or possibly improved by not having to explain to people I don't use facebook, please email or text me your invitations rather than use messenger). If Google, Amazon, Netflix, Apple disappeared the story would be completely different.

Facebook is an unparalleled titan in the realm of advertising and WhatsApp is basically a utility-level communication system for a big chunk of the globe. Instagram is a key cultural driver of the Western world. You many not feel any direct firsthand consequences, but the overall impact would transform the world around you.

> Facebook is an unparalleled titan in the realm of advertising

Not unparalleled - Google exists.

And we need less advertising, not more.

> and WhatsApp is basically a utility-level communication system for a big chunk of the globe.

Many other such systems exist - Telegram, Signal, Google Chat.

> Instagram is a key cultural driver of the Western world

Western culture will get along just fine without Instagram.

> the overall impact would transform the world around you.

For the better.