21–30 of 37 posts
On a more serious note, how would your entire network, worldwide go down? Are there really no independent zones (that are unaware of each other)? That can’t be good.
When I tell my clients Azure had another outage they're going to demand we move to another cloud service. Looks like I'm in for a looong couple weeks.
What are you going to tell them when the other cloud services have their inevitable outages?
Things have been back up for several minutes.
Earlier quoted context omitted.
I'm guessing he inherited it. I'm guessing most people on this thread are in that boat.
In my case, we actually chose it, as we´re a Microsoft shop. Hasn´t been that bad, but we did experience 2 major outages in 1 year. Not sure how many outages are there on AWS.
I've run most everything in us-west-2 Oregon the last 5+ years and I can't remember a similar sort of outage there in that time-frame.
A widespread world-wide outage like is happening now on Azure is a red flag imo.
Earlier quoted context omitted.
In my case, we actually chose it, as we´re a Microsoft shop. Hasn´t been that bad, but we did experience 2 major outages in 1 year. Not sure how many outages are there on AWS.
AWS has had a couple cascading EBS failures in us-east-1 years ago which affected a lot of services since it's a foundational building block of the whole system. It's been a reason to prefer instance storage for quite awhile imo. I've run most everything in us-west-2 Oregon the last 5+ years and I can't remember a similar sort of outage there in that time-frame. A widespread world-wide outage like is happening now on…
Yuck. Please tell me you don't do that anymore (unless for specialized workloads where you don't care if an instance loses data due to a shutdown).
Earlier quoted context omitted.
In my case, we actually chose it, as we´re a Microsoft shop. Hasn´t been that bad, but we did experience 2 major outages in 1 year. Not sure how many outages are there on AWS.
AWS has one or two a year from what I've seen... IIRC gcp has had outages too... Unless you're designing with several safeguards in place across multiple regions and cloud providers, there's no getting around it. Everyone has down time, it's just often coordinated to a lot of people when it does happen. It's still generally less than when you try to self host on a cloud provider, it's just not your mistake that did i…
Worldwide outage? That's really not the case. Over thousands of machines in 3 years I haven't seen a single failure that spanned more than one region. Or even a region going entirely offline. At most you'll see one service affected.
Seems like they have tons of global dependencies within their services which cause these cascading failures rather often... Seems like only a few months ago we were reading about a global outage that affected auth?
Regardless: Godspeed to the engineers working to fix this.
On a more serious note, how would your entire network, worldwide go down? Are there really no independent zones (that are unaware of each other)? That can’t be good.
I'm not in the cloud provider game, but it seems like it would be important to audit and ensure that there are no critical cross-region dependencies. I assume GCP and AWS do this regularly?
It seems like there are some things that have to be somewhat global (IAM comes to mind), but minimization of that seems important.
IMO this is the most embarrassing non-security thing that can happen to you as a cloud provider.
Earlier quoted context omitted.
In my case, we actually chose it, as we´re a Microsoft shop. Hasn´t been that bad, but we did experience 2 major outages in 1 year. Not sure how many outages are there on AWS.
AWS has one or two a year from what I've seen... IIRC gcp has had outages too... Unless you're designing with several safeguards in place across multiple regions and cloud providers, there's no getting around it. Everyone has down time, it's just often coordinated to a lot of people when it does happen. It's still generally less than when you try to self host on a cloud provider, it's just not your mistake that did i…
I'm not sure what the SLA is on a single region, but going down 0-2 times per year is a reasonable expectation, depending on the length of each one. If you want more, you have to have regional failover.
Azure is burning error budget in every region today and you would need to failover to a different cloud provider or your own datacenter.
If I'm interpreting this correctly, there was no plan you could implement solely in Azure that could have helped you today.
When I tell my clients Azure had another outage they're going to demand we move to another cloud service. Looks like I'm in for a looong couple weeks.
What are you going to tell them when the other cloud services have their inevitable outages?
1. The other providers have a better track record 2. The other providers don't go down globally multiple times a year