331–340 of 351 posts
Earlier quoted context omitted.
Gmail and other Google products went down last night. Close though. Thankfully not on Twitter or FB.
I don't believe Gmail was ever fully down. For me, I was just having problems with attachments. I also noticed app icons in the play store failing to load.
It was not fully down, there were issues with their blobstore.
Admittedly networking is not my strength, so perfectly happy for someone to shoot down this hypothesis.
Earlier quoted context omitted.
Why? When Facebook is down, you cannot use Facebook to communicate about the issue. That is also reason why FB still uses* irc instead of messenger for coordinating how to resolve such issues. * Or at least used 1 year ago, when I was working there.
Hmm, I wonder if we could leverage that to make irc more popular again. "FB uses X" is often all the justification small startups need for picking a tech.
:/ Something fun happening in Germany? https://www.akamai.com/us/en/resources/visualizing-akamai/re... And Level3 traffic going to Argentina? https://twitter.com/bgpstream/status/1105819050968580096 And GreatBritain going to cambodia? https://bgpstream.com/event/197968
Major sporting events in the EU.
BGP fuckups appear to happen regularly, based on the tweet history of that account.
Earlier quoted context omitted.
I work for a smaller but comparably large platform. "If everything is down check the DB" is at the top of one of our internal monitoring websites in red. Screw ups related to data loss are rare (I've been here years and haven't seen one with the DBs that the stuff I work with uses) but failures at this scale tend to cascade a little ways and it takes time to dig out of the hole. They probably have the problem solved…
Shouldn't the monitoring websites be able to check the DB status for you before you even look at that red text? :)
Earlier quoted context omitted.
So that engineers can be moved between product groups while carrying relevant knowledge and experience with them.
If that’s the goal, why not start incorporating more Erlang into the rest of Facebook instead? It proved its mettle at WhatsApp.
https://www.quora.com/Why-was-Erlang-chosen-for-use-in-Faceb...
I'm interviewing for a Production Engineer role at Facebook on Monday, thanks for providing relevant "do you have any questions for us" content.
When I interviewed for SRE at Google, they'd had a non-trivial cross product outage days before. Good conversation starter, but I couldn't get many details out of them.
Could this be related to the storm? I was out shoveling, and came back in to my phone blowing up. Our systems at IronMountain (formerly Fortrust) in Denver all rebooted at once. These are all on redundant power, each systems redundant power supplies connecting to different circuits entering the cabinet, and those two circuits fed from 3 PDUs (two separate, one share). Each of those is supposed to be fed by a separate…
"The storm"? It's sunny in the Bay Area for the first time in I don't know how long. I imagine it's nice in other parts of the world as well, other than where this localized "storm" is.
https://www.denverpost.com/2019/03/13/colorado-weather-bomb-...
Could this be related to the storm? I was out shoveling, and came back in to my phone blowing up. Our systems at IronMountain (formerly Fortrust) in Denver all rebooted at once. These are all on redundant power, each systems redundant power supplies connecting to different circuits entering the cabinet, and those two circuits fed from 3 PDUs (two separate, one share). Each of those is supposed to be fed by a separate…
No, FB datacenters are geographically diverse. They do run quarterly 'storms' where a datacenter is shut down to test failover and resiliency. I have no idea if today is one of those days, since I left last year.