121–130 of 187 posts
Earlier quoted context omitted.
Google's DC is underwater OVH's caught fire What's next, us-east-1 gets hit by Godzilla?
Lol us-east-1 already went down for a day back in 2017 when an intern accidentally took down the whole DC. We could call him “Godzilla” Source: my startup (stupidly) hosted our entire infra in us-east-1 at the time. Was a …tough day
Earlier quoted context omitted.
> The fire department is always going to prioritize safety of life, and after all it’s not their stuff getting soaked. They won't hesitate to smash your stuff or break down your walls either. Being in a fire is no joke. You've got to be crazy to think that your stuff is important. It's not.
Anyone talking up firefighters like this clearly hasn't been around them much. They're boys with toys that they don't frequently get to use and they work for the government. Follow the incentives. They'll do their jobs but they don't give a lot of fucks about things like "unnecessary property damage" and "other people's financial well being" and anything else not written in their KPIs. I used to drive tow truck. I ca…
Good. I want you to save me and my family from dying in a fucking fire. This thread is obsessed with saving Funko Pop collections for some reason??
A perfect example of when "cloud just means someone else's computers". It's literally a leaky abstraction.
Sure is leaky. And a cloud is a bunch of water vapour that eventually comes crashing down to earth. I'll never understand how we decided it was a good metaphor for a place we run our services. Startlingly accurate in this case.
Earlier quoted context omitted.
This doesn't sound as bad as OVH's 2021 fire.
Well, we had pictures very quickly of the OVH fire. Google seems to be not very transparent on what is exactly happening...
Earlier quoted context omitted.
zones and regions have different definition in google cloud than AWS. Multiple zones are physically co-located and are not truly availability zones because the physical proximity causes shared fates even when they have independent systems (network, power) that should allow one to fail while another doesn't. Even two datacenters in the same city are prey to the same meteor.
Then why do they charge extra for cross-zonal traffic?
B4: https://storage.googleapis.com/pub-tools-public-publication-...
Earlier quoted context omitted.
AWS treats its availability zones very seriously, each zone has its own independent power substation, air conditioning, and fiber lines. It's incredibly rare for multiple AZs to go down at once, especially since they are more than a few miles apart from each other.
Unless they’re in us-east-1 and it’s an Amazon software/service fault.
Not sure what kind of fire there was there, but once those automatic sprinkler systems get going, they are very difficult to stop. Someone in my freshman college dorm decided to use one as a clothes hanger hook and broke the thermometer in there. The sprinkler damaged the entire floor with water and the floor below had spotty rain as well. The fire department came and was mainly concerned about evacuating everyone ra…
datacenters dont use sprinkler systems (or at least they should not).
But for the service team I worked for, our AZ-evacuation story wasn't great at the time and it took us tens of minutes to manually move out of the AZ, but at least there wasn't a customer-visible availability impact. Once we did it was just monitoring and baby-sitting until we got the word to move back in, I think it was 1-2 days later.
If you operate on AWS you work with the assumption that an AZ is a failure domain, and can die at any time. Surprisingly many service teams at AWS still operate services that don't handle AZ failure that well (at the time). But if you operate services in the cloud you have to know what the failure domain is.
Earlier quoted context omitted.
Yes. Azure and GCPs numbers on the size of their AZs and such are more marketing spin than hard engineering. AWS keeps these in separate physical locations to provide true separation. While there have been tech related regional incidents at AWS a physical event disabling multiple AZs would be extremely unlikely given their much more robust and geographically distributed design. If such a physical event had happened i…
Googler, opinions are my own. I think you misunderstand Google's infrastructure. I'm guessing that each GCP zone is actually a Borg Cell (see: https://storage.googleapis.com/pub-tools-public-publication-... ). Borg cells tend to be isolated from eachother in many ways in the physical layer (networking and management being a big one, not sure about power). So networking or machine management for an entire zone could g…
Obviously every cloud vendor recommends replicating data between multiple regions, but fact of the matter is that a lot of cloud services work much easier with redundancy within a single region than multi-region redundancy.
Earlier quoted context omitted.
Always worth tracking down the sprinkler shut off valve in your residence/place of work. If you’re in a high rise it‘ll be the big red wheel on the sprinkler main in the fire stairs. If it’s a spurious activation you can just shut it off yourself, you don’t need to ask anybody’s permission. The fire department is always going to prioritize safety of life, and after all it’s not their stuff getting soaked.
> The fire department is always going to prioritize safety of life, and after all it’s not their stuff getting soaked. They won't hesitate to smash your stuff or break down your walls either. Being in a fire is no joke. You've got to be crazy to think that your stuff is important. It's not.
https://www.youtube.com/watch?v=rO0ioCCiEe8
Firefighters arrive in 5 minutes.