Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook-owned sites were down

facebook.com

Post reply on HN
Print view View original

511–520 of 1001 posts

Re: Facebook-owned sites were down

#511

Post by lstodd »

pbhjpbhj wrote: post #454
bennyp101 wrote: post #346

Earlier quoted context omitted.

Interesting that they published stuff about their BGP setup and infrastructure a few months ago - maybe a little tweak to roll backs is needed. "... We demonstrate how this design provides us with flexible control over routing and keeps the network reliable. We also describe our in-house BGP software implementation, and its testing and deployment pipelines. These allow us to treat BGP like any other software componen…

Surely Facebook don't update routing systems between data centres (IIRC the situation) when they don't have people present to fix things if they go wrong? Or have an out-of-band connection (satellite, or dial-up (?), or some other alternate routing?). I must be misunderstanding this situation here. [Aside: I recall updating wi-fi settings on my laptop and first checking I had direct Ethernet connection working ... an…

> don't update routing systems between data centres (IIRC the situation) when they don't have people present

Ha. You put too much faith into people.

Re: Facebook-owned sites were down

#512

Post by qnsi »

shmiga wrote: post #504

A lot of addict's will now feel how is it to live in the reality not scrolling dump images. I hope this will be tradition at least once a month.

I am an addict. I refresh this thread waiting for information: fb is back online.

I need help but this is too hard for me. I uninstall social media once a week but install two days later.

I should probably go to therapy with this, but I am not sure I wouldnt be laughed at

Re: Facebook-owned sites were down

#513

Post by jaywalk »

iso1210 wrote: post #460
daper wrote: post #410

Earlier quoted context omitted.

Our security team complained that we have some services like monitoring or SSH access to some Jump Hosts accessible without a VPN because VPN should be mandatory to access all internal services. I'm afraid once comply we could be in similar situation where Facebook is now...

But you have two independent VPNs right, using different technologies on different internet handoffs in very different parts of your network, right?

Still wouldn't help if your configuration change wipes you clear off the Internet like Facebook's apparently has. The only way to have a completely separate backup is to have a way in that doesn't rely on "your network" at all.

Re: Facebook-owned sites were down

#514

Post by mindcrime »

dgb23 wrote: post #396
mindcrime wrote: post #355

Okay, let me tell you the difference between Facebook and everyone else, we don't crash EVER! If those servers are down for even a day, our entire reputation is irreversibly destroyed! Users are fickle, Friendster has proved that. Even a few people leaving would reverberate through the entire userbase. The users are interconnected, that is the whole point. College kids are online because their friends are online, and…

Is this a quote?

Yes, it's from "The Social Network". It's a scene where Mark Z. is explaining to Eduardo how important it is that the servers stay up all the time.

Of course it was, as far as I know, ficitonalized in the first place, although it rings true (in context) to some extent. What I wonder is, how much is that true now? That is, how much downtime would FB have to experience for enough users to start leaving, to the point that it might prompt a serious exodus.

Re: Facebook-owned sites were down

#515

Post by poetaster »

zrail wrote: post #319

hugops for the engineers having to deal with this. It's incredibly stressful and I personally feel like they deserve some empathy, even if I don't like Facebook. I wonder if maybe part of the lesson will be to run the root of your authoritative DNS hierarchy on separate infrastructure with a separate domain name. Using facebook.com as your root is cool and all but when that label disappears it causes huge issues.

I used to do this properly. One vanity got the better of me. Got some work to do. TGF SQL.

Re: Facebook-owned sites were down

#517

Post by Scarblac »

huevosabio wrote: post #365
rvnx wrote: post #349

Earlier quoted context omitted.

I like how FB decided to send "ramenporn" as their spokesperson.

A particular facet I love of the internet era is journalists reporting serious events while having to use the completely absurd usernames... "A Facebook engineer in the response team, ramenporn..."

I remember some huge DDOS attacks like a decade ago, and people were speculating who could be behind it. The three top theories were Russian intelligence, the Mossad, and this guy on 4chan who claimed to have a Botnet doing it.

That was the start of living in the future for me.

Re: Facebook-owned sites were down

#519

Post by rodgerd »

Ueland wrote: post #493
guidopallemans wrote: post #476

Earlier quoted context omitted.

He just deleted all his updates. user: https://old.reddit.com/user/ramenporn some messages: * This is a global outage for all FB-related services/infra (source: I'm currently on the recovery/investigation team). * Will try to provide any important/interesting bits as I see them. There is a ton of stuff flying around right now and like 7 separate discussion channels and video calls. * Update 1440 UTC: \ As many of you…

And there his account went poof, thanks for archiving.

If it was actually someone in Facebook, their job is gone by now, too.

Re: Facebook-owned sites were down

#520

Post by bink »

chasd00 wrote: post #449
jfrunyon wrote: post #406

Earlier quoted context omitted.

I can't fathom how they didn't plan for this. In any business of size, you have to change configuration remotely on a regular basis, and can easily lock yourself out on a regular basis. Every single system has a local user with a random password that we can hand out for just this kind of circumstance...

Another Monday morning at a boring datacenter job, i bet they weren't even there yet at 830 when the phones started ringing.

You mean the VOIP phones that could no longer receive incoming calls?
Post reply on HN