Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook-owned sites were down

facebook.com

Post reply on HN
Print view View original

741–750 of 1001 posts

Re: Facebook-owned sites were down

#741

Post by jug »

ddalex wrote: post #726
still_grokking wrote: post #669

Earlier quoted context omitted.

You mean the same problem as when GMail goes down and Googlers can't reach each other? I guess good decentralized public communication services could solve those issues for everybody.

Googler here - my opinions are my own, not representing the company at the lowest level in case of severe outage we resort to IRC, Plain Old Telephone Service and, sometimes, stick-it notes taped to windows...

Some people here say their fallback IRC doesn't work due to DNS reliance. :|

Re: Facebook-owned sites were down

#742

Post by korethr »

okwubodu wrote: post #682
kiernanmcgowan wrote: post #643

Earlier quoted context omitted.

My suspicion is that since a lot of internal comms runs through the FB domain and since everyone is still WFH, then its probably a massive issue just to get people talking to each other to solve the problem.

I don’t know how true it is but a few reports claim employees can’t get into the building with their badges.

Link to such claims here: https://news.ycombinator.com/item?id=28750894

I have no doubt that the publicly published post-mortem report (if there even is one) will be heavily redacted in comparison to the internal-only version. But I very much want to see said hypothetical report anyway. This kind of infrastructural stuff fascinates me. And I would hope there would be some lessons in said report that even small time operators such as myself would do well to heed.

Re: Facebook-owned sites were down

#743

Post by cranekam »

yoelo wrote: post #653
john37386 wrote: post #637

Earlier quoted context omitted.

Yeah the patch to fix BGP to reach the DNS is sent by email to @facebook.com. Ooops no DNS to resolve the MX records to send the patch to fix the BGP routers.

Seriously? Is that how it works?

No. A network like Facebook's is vast and complicated and managed by higher-level configuration systems, not people emailing patches around.

If this issue is even to do with BGP it's much more likely the root of the problem is somewhere in this configuration system and that fixing it is compounded by some other issues that nobody foresaw. Huge events like this are always a perfect storm of several factors, any one or two of which would be a total noop alone.

Re: Facebook-owned sites were down

#744

Post by z-nexx »

wtf-is-ur-prblm wrote: post #716
yholio wrote: post #648

Earlier quoted context omitted.

Essentially, they locked themselves out with an uninspired command line at the exact moment the datacenter was being hijacked by ape-people. Yup, corporate comms won't love these status updates.

Sorry, are you referring to data center technicians as “ape people”?

As a former data center technician, I wouldn't say it's too far off

Re: Facebook-owned sites were down

#745

Post by smrtinsert »

outworlder wrote: post #561
jfrunyon wrote: post #406

Earlier quoted context omitted.

I can't fathom how they didn't plan for this. In any business of size, you have to change configuration remotely on a regular basis, and can easily lock yourself out on a regular basis. Every single system has a local user with a random password that we can hand out for just this kind of circumstance...

> I can't fathom how they didn't plan for this Maybe because they were planning for a million other possible things to go wrong, likely with higher probability than this. And busy with each day's pressing matters.

Haha sure. They were too busy implementing php compilers to figure out that "whole DR DNS thing"

rotflmao. I'd remove Facebook from my resume.

Re: Facebook-owned sites were down

#746

Post by _joel »

john37386 wrote: post #667
yoelo wrote: post #653

Earlier quoted context omitted.

Seriously? Is that how it works?

I don't know. I doubt. It's just funny to think that you need email to fix BGP, but DNS is down because of BGP. You need DNS to send email which needs BGP. It's a kind of chicken and egg problem but at a massive scale this time.

You'd think they'd have worked that into their DR plans for a complete P1 outage of the domain/DNS, but perhaps not, or at least they didn't add removal of BGP announcements to the mix.

Re: Facebook-owned sites were down

#748

Post by ticklemyelmo »

wtf-is-ur-prblm wrote: post #716
yholio wrote: post #648

Earlier quoted context omitted.

Essentially, they locked themselves out with an uninspired command line at the exact moment the datacenter was being hijacked by ape-people. Yup, corporate comms won't love these status updates.

Sorry, are you referring to data center technicians as “ape people”?

But we're all ape people.

Re: Facebook-owned sites were down

#749

Post by mdtancsa »

Animats wrote: post #628

There's still no connectivity to Facebook's DNS servers: > traceroute a.ns.facebook.com traceroute to a.ns.facebook.com (129.134.30.12), 30 hops max, 60 byte packets 1 dsldevice.attlocal.net (192.168.1.254) 0.484 ms 0.474 ms 0.422 ms 2 107-131-124-1.lightspeed.sntcca.sbcglobal.net (107.131.124.1) 1.592 ms 1.657 ms 1.607 ms 3 71.148.149.196 (71.148.149.196) 1.676 ms 1.697 ms 1.705 ms 4 12.242.105.110 (12.242.105.110)…

Its partially there. C and D are still not in the global tables according to routeviews ie. 185.89.219.12 is still not being advertised to anyone. My peers to them in Toronto have routes from them, but not sure how far they are supposed to go inside their network. (past hop 2 is them)

% traceroute -q1 -I a.ns.facebook.com

traceroute to a.ns.facebook.com (129.134.30.12), 64 hops max, 48 byte packets 1 torix-core1-10G (67.43.129.248) 0.133 ms

2 facebook-a.ip4.torontointernetxchange.net (206.108.35.2) 1.317 ms

3 157.240.43.214 (157.240.43.214) 1.209 ms

4 129.134.50.206 (129.134.50.206) 15.604 ms

5 129.134.98.134 (129.134.98.134) 21.716 ms

6 *

7 *

% traceroute6 -q1 -I a.ns.facebook.com

traceroute6 to a.ns.facebook.com (2a03:2880:f0fc:c:face:b00c:0:35) from 2607:f3e0:0:80::290, 64 hops max, 20 byte packets

1 toronto-torix-6 0.146 ms

2 facebook-a.ip6.torontointernetxchange.net 17.860 ms

3 2620:0:1cff:dead:beef::2154 9.237 ms

4 2620:0:1cff:dead:beef::d7c 16.721 ms

5 2620:0:1cff:dead:beef::3b4 17.067 ms

6 *

7 *

8 *

Post reply on HN