21–30 of 113 posts
Visual Studio Online has been offline all day. They say it is due to the same Azure outage. This has had a productivity impact. If Microsoft didn't own GitHub, this may have prompted a move, but since they do it seems a little redundant given that Github will likely be on Azure too before long. https://blogs.msdn.microsoft.com/vsoservice/?p=17405
The Azure status page has more information. I suggest updating the link. > A severe weather event, including lightning strikes, occurred near one of the South Central US datacenters. This resulted in a power voltage increase that impacted cooling systems. Automated datacenter procedures to ensure data and hardware integrity went into effect and critical hardware entered a structured power down process. https://azure.…
Edit: out my rant. It's been a long day because of this. Just going to leave it at that.
Honestly it's hard to imagine a good mitigation for this. "Build more datacenters" is already happening as fast as it can. "Keep enough spare capacity around to handle losing one of the biggest datacenters in the world" is pretty unreasonable.
If you, as a customer, are uptime focused enough that it's worth paying extra, then the sensible practice has always been Cross-Cloud infrastructhre/failovers. At least since the Amazon Easter failure of 2011. That's what giants like Netflix do.
The Visual Studio Marketplace is also down https://marketplace.visualstudio.com/ .
Edit: out my rant. It's been a long day because of this. Just going to leave it at that.
They have some services that are "global", ie not tied to a given region. Those services' requests are actually processed all over the place, but south central is a big datacenter. The 9th biggest in the world, apparently. When it lost cooling and shut down, everything routed around it as planned... But it caused so much extra traffic that it overwhelmed the connections to other datacenters. The backlog of requests i…
Err what?
It's entirely reasonable to expect Azure to handle the loss of a single DC and not have a 14+ hour global outage. I don't care how big the DC is, losing one should not take out the world, especially not for the length of time this one has been going on.
The Azure status page has more information. I suggest updating the link. > A severe weather event, including lightning strikes, occurred near one of the South Central US datacenters. This resulted in a power voltage increase that impacted cooling systems. Automated datacenter procedures to ensure data and hardware integrity went into effect and critical hardware entered a structured power down process. https://azure.…
I wonder if that's one of their facilities down here in San Antonio. Was getting flash flood alerts on my phone all night and morning.
Earlier quoted context omitted.
They have some services that are "global", ie not tied to a given region. Those services' requests are actually processed all over the place, but south central is a big datacenter. The 9th biggest in the world, apparently. When it lost cooling and shut down, everything routed around it as planned... But it caused so much extra traffic that it overwhelmed the connections to other datacenters. The backlog of requests i…
> "Keep enough spare capacity around to handle losing one of the biggest datacenters in the world" is pretty unreasonable. Err what? It's entirely reasonable to expect Azure to handle the loss of a single DC and not have a 14+ hour global outage. I don't care how big the DC is, losing one should not take out the world, especially not for the length of time this one has been going on.
https://perspectives.mvdirona.com/2017/04/how-many-data-cent...
Earlier quoted context omitted.
You made the same mistake as Microsoft, who seemed to have put their Azure status page on Azure: https://www.reddit.com/r/AZURE/comments/9cvgn2/is_there_a_st... Their status page is back up now, but my stuff's still broken. :\
And Amazon. I recall that during their most recent s3 outage Amazon's status page was green across the board, because somehow all the assets that were supposed to be displayed when things went wrong were themselves hosted on the thing that was down.
Edit: out my rant. It's been a long day because of this. Just going to leave it at that.
So AWS has had some big outages, as has Azure. Has GCP had any big outages yet?