Update on Azure Storage Service Interruption
azure.microsoft.com
Update on Azure Storage Service Interruption
1–10 of 18 posts
Re: Update on Azure Storage Service Interruption
#2404 - Article Not Found
Re: Update on Azure Storage Service Interruption
#3 404 - Article Not Found
The article you were looking for was not found, but maybe try looking again!
---
Yesterday evening Pacific Standard Time, Azure storage services experienced a service interruption across the United States, Europe and parts of Asia, which impacted multiple cloud services in these regions. I want to first sincerely apologize for the disruption this has caused. We know our customers put their trust in us and we take that very seriously. I want to provide some background on the issue that has occurred.
As part of a performance update to Azure Storage, an issue was discovered that resulted in reduced capacity across services utilizing Azure Storage, including Virtual Machines, Visual Studio Online, Websites, Search and other Microsoft services. Prior to applying the performance update, it had been tested over several weeks in a subset of our customer-facing storage service for Azure Tables. We typically call this “flighting,” as we work to identify issues before we broadly deploy any updates. The flighting test demonstrated a notable performance improvement and we proceeded to deploy the update across the storage service. During the rollout we discovered an issue that resulted in storage blob front ends going into an infinite loop, which had gone undetected during flighting. The net result was an inability for the front ends to take on further traffic, which in turn caused other services built on top to experience issues.
Once we detected this issue, the change was rolled back promptly, but a restart of the storage front ends was required in order to fully undo the update. Once the mitigation steps were deployed, most of our customers started seeing the availability improvement across the affected regions. While services are generally back online, a limited subset of customers are still experiencing intermittent issues, and our engineering and support teams are actively engaged to help customers through this time.
When we have an incident like this, our main focus is rapid time to recovery for our customers, but we also work to closely examine what went wrong and ensure it never happens again. We will continually work to improve our customers’ experiences on our platform. We will update this blog with a RCA (root cause analysis) to ensure customers understand how we have addressed the issue and the improvements we will make going forward.
---
Re: Update on Azure Storage Service Interruption
#4I hope to read more in the post-mortem RCA but I am curious what their flighting missed, is flighting so limited that is does not see the cross region scale or something? I also had the feeling from watching Mark Russinovich discuss previous failures that their patch rollouts were much more controlled.
Re: Update on Azure Storage Service Interruption
#5I hope to read more in the post-mortem RCA but I am curious what their flighting missed, is flighting so limited that is does not see the cross region scale or something? I also had the feeling from watching Mark Russinovich discuss previous failures that their patch rollouts were much more controlled.
What I've seen from their patching of ordinary machines, I would say its pretty far from controlled or well thought through. Their patching has led to our machines becoming unavailable before, despite that we have multiple machines in the same availability set. We've been in contact with support to describe what happens and have gotten an Oh, its by design-response back.
Re: Update on Azure Storage Service Interruption
#6How about not rolling out a patch to all data centers at once?
Re: Update on Azure Storage Service Interruption
#7So they rolled out a performance update (not a critical security fix) to all their datacenters at once?
This sounds incredibly amateur for a provider the size of Azure.
Re: Update on Azure Storage Service Interruption
#8This video http://channel9.msdn.com/events/Build/2014/3-615; start at 39 minutes. Mark Russinovich explains their update rollout procedure (or at least part of it).
Willing to bet the rollout infrastructure depended on storage and so their ability to control or stop the rollout was broken once the storage failures began.
Re: Update on Azure Storage Service Interruption
#9How about not rolling out a patch to all data centers at once?
Hi, this is Corey Sanders, an engineer on the Azure compute team. Yes, our normal policy for updates is to roll them in incremental batches. In this case, due to an operational error, we did not apply the changes as per normal policy.
Re: Update on Azure Storage Service Interruption
#10So they rolled out a performance update (not a critical security fix) to all their datacenters at once? This sounds incredibly amateur for a provider the size of Azure.
Hey nnx, this is Corey from the Azure engineering team. We have a standard protocol in the team of applying production changes in incremental batches. Due to an operational error, this update was made across most regions in a short period of time. I really apologize for the disruption.