Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook-owned sites were down

facebook.com

Post reply on HN
Print view View original

531–540 of 1001 posts

Re: Facebook-owned sites were down

#531

Post by treesknees »

DaiPlusPlus wrote: post #522
meragrin_ wrote: post #497

Earlier quoted context omitted.

The account has been deleted as well.

What are they afraid of? While they are sharing information that's internal/proprietary to the company, it isn't anything particularly sensitive and having some transparency into the problem is good for everyone. Who'd want to work for a company that might take disciplinary action because an SRE posted a reddit comment to basically say "BGP's down lol" - If I was in charge I'd give them a modest EOY bonus for being h…

Mentioned in another reply

Shareholders and other business leaders I'm sure are much happier reporting this as a series of unfortunate technical failures (which I'm sure is part of it) rather than a company-wide organizational failure. The fact they can't physically badge in the people who know the router configuration speaks to an organization that hasn't actually thought through all its failure modes. People aren't going to like that. It's not uncommon to have the datacenter techs with access and the actual software folks restricted, but that being the reason one of the most popular services in the world has been down for nearly 3 hours now will raise a lot of questions.

Re: Facebook-owned sites were down

#534

Post by chasd00 »

foobaw wrote: post #250

This is taking a longer time than expected for a company like Facebook - must be serious where a rollback isn't possible or trivial.

from what i understand (take with grain of salt) remote access to the routers affected is down. So they need to be physically plugged in to address the issue. hence some of the other "scrambling private jets" comments referring to getting the right people physically plugged in to the right routers.

Re: Facebook-owned sites were down

#536

Post by handmodel »

DaiPlusPlus wrote: post #522
meragrin_ wrote: post #497

Earlier quoted context omitted.

The account has been deleted as well.

What are they afraid of? While they are sharing information that's internal/proprietary to the company, it isn't anything particularly sensitive and having some transparency into the problem is good for everyone. Who'd want to work for a company that might take disciplinary action because an SRE posted a reddit comment to basically say "BGP's down lol" - If I was in charge I'd give them a modest EOY bonus for being h…

Seems reasonable that at a company of 60k, with hundreds who specialize in PR, you do not want a random engineer making the choice himself to be the first to talk to the press by giving a PR conference on a random forum.

Re: Facebook-owned sites were down

#537

Post by jl6 »

pmlnr wrote: post #489
formerly_proven wrote: post #420

Earlier quoted context omitted.

This sounds like something that might have been done with security in mind. Although generally speaking, remote hands don't have to be elite hackors.

Have you ever tried to remotely troubleshoot THROUGH another person?!

Yes, and it works if both parties are able to communicate using precise language. The onus is on the remote SME to exactly articulate steps, and on the local hands to exactly follow instructions and pause for clarifications when necessary.

Re: Facebook-owned sites were down

#538

Post by blntechie »

deadalus wrote: post #225
eeegnu wrote: post #199

Earlier quoted context omitted.

Probably people flooding in to see if anyone knows why things are down. Even Google speed test was down, presumably from too many people testing if it's their internet that's at issue.

https://www.speedtest.net is down too

The site is working fine for me. Speedtest CLI also is useful but doubt when DNS is down.

Re: Facebook-owned sites were down

#539

Post by 1_player »

alexellisuk wrote: post #281
rocho wrote: post #194

Earlier quoted context omitted.

I can confirm, HN, GitHub and Slack are very slow for me as well. Google is very fast, on the other hand.

Also slow here. I can't see anything on the AWS Service Dashboard https://status.aws.amazon.com

In my experience, any service dashboard is useless unless the problem has been going on for so long (i.e. hours) that it is obvious something's wrong.

Re: Facebook-owned sites were down

#540

Post by xgme »

kerng wrote: post #380
kossTKR wrote: post #291

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

Wondering how Facebook communicates now internally - most of their work streams likely depend on Facebooks systems which are all down. Can engineers and security teams even access prod systems anymore? Like, would "Bastion" hosts be reachable? Wonder if they use Signal and Slack now?

Facebook does use IRC and Zoom as a fallback.
Post reply on HN