121–130 of 153 posts
Earlier quoted context omitted.
Starting up a cloud region after a total shutdown is likely an untested procedure with no well known timeframe, even if the hardware is ok.
If you're in the business of being a massive cloud provider, hopefully restarting a region is not an untested procedure for you. You could always test this in a live environment before a region becomes open to customers.
Earlier quoted context omitted.
AWS has similarly suffered outages from an entire datacenter being taken out like this. No one is immune. If you want true fault-tolerance you need to be multi-regional (everyone says as much), ideally, multi-continental. europe-west9 is the only large Google datacenter in France afaik. Building more would cost lots more money, and it seems like the market isn't there for it. Workloads that require data locality in F…
Eh source for that. AWS has had issues where a single Zone caused such a lack of capacity in the region that some multi-zone services degraded to the point of a domino fail-over. However I've not heard of any AWS event where a fire/flood in AZ A also caused a fire/flood in AZ B.
Being in the same building is an "implementation detail" from a customer perspective, what matters is the consequences of this decision.
For example, maybe this decision allows for better network connectivity at a lower cost for inter-zones traffic, while, on the other hand, not protecting against some classes of risks.
In the end, you can have a similar multi-zone outage keeping the region down for an extended period of time just because of a bad network config push (see the massive facebook outage in 2021). As a customer, I don't care if it's a flood or a network outage.
Imho, what matters the most is a clear documentation of how these abstractions work for users and the corresponding contractual agreements (costs, SLAs, etc). Users can thus decide if they are ready to pay the price of protecting themselves against an extended outage impacting a single region.
Earlier quoted context omitted.
GCP doesn't operate the same way as Google consumer products. We are a paid customer for over 5 years and I also have only good things to say about GCP and their support
>We are a paid customer To be fair, so were Stadia users.
Title is incorrect, this is not a general outage. There are two separate issues: europe-west-9 (Paris) has been physically flooded with water somehow and is hard down. This is obviously bad if you're using the region in question, but has zero impact elsewhere. https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf... There is a separate issue stopping changes to HTTP load balancers across most of GCP, but it ha…
So this is probably too soon, thoughts and prayers for the datacenter operators and staff out there, but are they going to auction off the flooded hardware? Trying to restore a flooded Google rack sounds like a super fun project. Anyone experience with losing an entire DC to flooding? edit: I just Googled it (lol) and this DC has to be brand spanking new ( https://cloud.google.com/blog/products/infrastructure/google.…
Our DC was intact, but the building and access was cut-off. We lost the backup diesel power generators in the flooding. Of course, grid power was cut-off.
Our DC operating team managed to shutdown all the servers and racks cleanly before UPS power was completely drained. The 4 engineers and 2 security guards then swam out of the compound in chest high waters. (I am not kidding).
When the rains subsided and the flood waters receded after a couple of days, we had to plan the restart. The facility still had to be certified by health and safety, but we needed to get the datacenter back up.
A secondary operations site that would remote-connect to the DC was brought up in 1 week since we estimated the rains to potentially continue for a few more days and cause interruptions. But the critical item for the plan to work was getting a new backup power setup. We rolled in a truck-mounted diesel generator and positioned it in the highest point in the campus (also close to our building tower that had the DC) and ran power cables to it (we had to source this and it was a challenge to do it with the time crunch and the rains).
We moved staff to other cities by bus (airport was shutdown) as part of our recovery plan, but we still needed connectivity to our DC for some of the critical processes.
Long story short, it worked.
I'll never forget the experience and the scars from this war story.
Earlier quoted context omitted.
> It's intriguing to me that AFAIK load-balancer changes in AWS don't suffer from this problem. (Though, on the other hand, CDN changes do.) The architecture is a lot different. Using google means working with the load balancer in some form. It's all interconnected. AWS is all separate parts that are stitched together thinly. E.g. you can have a single global load balancer in Google that handles your whole infrastruc…
> AWS is all separate parts that are stitched together thinly. Yeah I always hate this when I have to work with AWS. All their services feel like they were designed by completely different companies. Every management interface looks and feels different, and there are tons of services that do almost the same thing so it's not clear which would be best to use. It's a maze to me. Luckily I don't have to work with cloud…
> I really prefer Azure where everything is in the same console and there isn't a lot of overlap. But cloud guys seem to hate it, not sure why.
Also, everything is a Wizard because MS doesn't want to expose the sausage factory.
Earlier quoted context omitted.
But europe-west9-a is only one zone, why does the whole region fall over as a consequence?
GCP has multiple zones in the same physical building. Not all cloud providers have distinct physical buildings for each Availability Zone.
Back in the days when we had our own data centers a zone was defined as a "fire section" meaning that it should not be impacted if any other zone of the data center had a fire. This obviously means that you can't call 3 floors of a building a zone.
Edit: The information on this site https://cloud.google.com/docs/geography-and-regions#regions_... clearly states that a zone is "physically distinct" so they have some explaining to do.
Edit 2: Sneaky... They changed the status page to say "europe-west9" instead of "europe-west9-a".
Earlier quoted context omitted.
Starting up a cloud region after a total shutdown is likely an untested procedure with no well known timeframe, even if the hardware is ok.
If you're in the business of being a massive cloud provider, hopefully restarting a region is not an untested procedure for you. You could always test this in a live environment before a region becomes open to customers.
I’m sure they have checklist and procedures, but an unknowable laundry list of things will go wrong.
Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region https://twitter.com/GCP_Incidents
But europe-west9-a is only one zone, why does the whole region fall over as a consequence?
Earlier quoted context omitted.
But europe-west9-a is only one zone, why does the whole region fall over as a consequence?
AFIK all zones (a, b, and c) have been reported to be down. I'd love to understand ehat happened.
Earlier quoted context omitted.
TBH, GCP isn't operating like Google does. I'm a long time customer and have only good things to tell so far.
There's some exceptions, like Google Cloud Debugger which is getting shut down at the end of the month (although to be fair there was a long notice period). My team is pretty sad about it going away: https://cloud.google.com/debugger/docs
It appears based on my playing around with it that the data is actually traveling into Firebase successfully, but there is not the slightest shred of UI nor onboarding docs for "hello, I would like one cloud debugger, please"