Live data from Hacker News

Facebook, Instagram go down around the world in an apparent outage

usatoday.com

241–250 of 351 posts

Re: Facebook, Instagram go down around the world in an apparent outage

#241
post #237
post #233

Earlier quoted context omitted.

The heavy traffic is due to sports events - champions league last-16 matchday live streaming: Bayern Munich vs. Liverpool, FC Barcelona vs Olympique Lyon. The heatmap matches the clubs' home countries UK, Germany, France & Spain quite well. Dont conflate that with fb/insta problems.

Indeed. The games are streamed like crazy! Lots of streams are in HD.

Yep, the actual bandwidth consumption is mindblowing :)

Re: Facebook, Instagram go down around the world in an apparent outage

#242
post #76

Could this be related to the storm? I was out shoveling, and came back in to my phone blowing up. Our systems at IronMountain (formerly Fortrust) in Denver all rebooted at once. These are all on redundant power, each systems redundant power supplies connecting to different circuits entering the cabinet, and those two circuits fed from 3 PDUs (two separate, one share). Each of those is supposed to be fed by a separate…

Do they do this to get around and 99.9% uptime agreements?

0.1% is still almost an hour per month

Re: Facebook, Instagram go down around the world in an apparent outage

#243

Earlier quoted context omitted.

Last time I dealt with a cloud provider outage the status page was unresponsive during the outage because the status page had some kind of dependency on the resources that were down...

I remember that, the status page for AWS stored the red icon on s3 which was the service that was down.

Actually in this case it was Azure related but it's pretty funny that the same thing has happened to the two biggest cloud platforms

Re: Facebook, Instagram go down around the world in an apparent outage

#244

I'm interviewing for a Production Engineer role at Facebook on Monday, thanks for providing relevant "do you have any questions for us" content.

Good luck! If you’re interviewing in MPK make sure your recruiter takes you to the barbecue shack for lunch

Re: Facebook, Instagram go down around the world in an apparent outage

#245
post #58

Earlier quoted context omitted.

Private post Morten: The NSA middleware we are required to run (that took time to deploy to each of our social partners) is breaking something so let’s revert. Public post Mortem: Entirely believable technical cause.

I imagine the NSA uses an optical tap device. These devices create identical copies and require no power or management.

They would still need the encryption keys since Google encrypts their inter-data center fiber

Re: Facebook, Instagram go down around the world in an apparent outage

#246

I'm interviewing for a Production Engineer role at Facebook on Monday, thanks for providing relevant "do you have any questions for us" content.

Good luck! If you’re interviewing in MPK make sure your recruiter takes you to the barbecue shack for lunch

But be prepared to stand in line for longer than it takes you to actually eat.

Re: Facebook, Instagram go down around the world in an apparent outage

#247

Could this be related to the storm? I was out shoveling, and came back in to my phone blowing up. Our systems at IronMountain (formerly Fortrust) in Denver all rebooted at once. These are all on redundant power, each systems redundant power supplies connecting to different circuits entering the cabinet, and those two circuits fed from 3 PDUs (two separate, one share). Each of those is supposed to be fed by a separate…

No, FB datacenters are geographically diverse.

They do run quarterly 'storms' where a datacenter is shut down to test failover and resiliency. I have no idea if today is one of those days, since I left last year.

Re: Facebook, Instagram go down around the world in an apparent outage

#248
post #9

So yesterday Google had a major (and out of character) outage across its apps, and today Facebook has a major (and also out of character) outage across its apps. I can't wait to see the RCA for both of these and if they're related.

Remember when youtube was down for like six hours a few months ago and we still haven't heard why?

Re: Facebook, Instagram go down around the world in an apparent outage

#249

Earlier quoted context omitted.

Good luck! If you’re interviewing in MPK make sure your recruiter takes you to the barbecue shack for lunch

But be prepared to stand in line for longer than it takes you to actually eat.

#worthit (although I’ll admit my afternoon interviews were a bit... challenging... after 5lbs of brisket)

Re: Facebook, Instagram go down around the world in an apparent outage

#250

Could this be related to the storm? I was out shoveling, and came back in to my phone blowing up. Our systems at IronMountain (formerly Fortrust) in Denver all rebooted at once. These are all on redundant power, each systems redundant power supplies connecting to different circuits entering the cabinet, and those two circuits fed from 3 PDUs (two separate, one share). Each of those is supposed to be fed by a separate…

A company as large and sophisticated as FB has data centers and cloud services in multiple countries, and in the US, probably colocated data centers. Certainly nothing localized to where you are.
Post reply on HN