EC2 Maintenance Update II

Doesn't matter why they did the maintenance, so your first four sentences are simply rationalizations. The parent was complaining all the zones were restarted AT THE SAME TIME. Jeff explains why they had to do the maintenance, and nobody is disputing why they had to do it. What is being disputed is that they did it all at once, without rolling the restarts. This makes ZERO sense given their advice to 're-examine your (AWS based) architecture' for fault tolerance. Don't patronize me while you're doing something that affects me in a way I can't control.

I've had direct experience with AWS in this regard, and was equally disappointed in the outcome. If you want to see a class act in taking ownership of issues arising from such matters, have a look at Rackspace's response:

  Dear Rackspace Customer,

  I’m writing to apologize for the downtime and inconvenience that you and others of our customers have suffered in recent days.  Like other major cloud providers, we were forced to reboot some of our customers’ servers to patch a security vulnerability affecting certain versions of XenServer, a popular open-source hypervisor.  This maintenance was especially difficult for many of you because it had to be performed on short notice, and over the weekend.

  Now that this issue has been fully remediated, without any reports of compromised data among our customers, I’d like to explain what happened, and why.

  Whenever we at Rackspace become aware of a security vulnerability, whether in our systems or (as in this case) in third-party software, we face a balancing act.  We want to be as transparent as possible with you, our customers, so you can join us in taking actions to secure your data.  But we don’t want to advertise the vulnerability before it’s fixed — lest we, in effect, ring a dinner bell for the world’s cyber criminals.

  That’s the dilemma that we faced over the Xen bug. Such vulnerabilities are regularly found in software, whether proprietary or open source. The key, once a bug is identified, is to fix it swiftly and quietly.  This particular vulnerability could have allowed bad actors who followed a certain series of memory commands to read snippets of data belonging to other customers, or to crash the host server. We wanted to flag the issue as quickly as possible to those of you using our Standard, Performance 1, and Performance 2 Cloud Servers, and our Hadoop Cloud Big Data service.  But we didn’t want to do so until we had a software patch in place to address the vulnerability.

  When we learned of the security issue and realized its significance early last week, our engineers worked with our Xen partners to develop and test a patch, and organize a reboot plan.  The patch wasn’t ready until the evening of Friday, Sept. 26.  And the technical details of the vulnerability were scheduled to be publicly released on Wednesday, Oct. 1.  We were faced with the difficult decision of whether to start our reboots over the weekend, with short notice to our customers, or postpone it until Monday. The latter course would not allow us to sufficiently stagger the reboots.  It would jeopardize our ability to fully patch all the affected servers before the vulnerability became public, thus exposing our customers to heightened risk.

  We decided the lesser evil was to proceed immediately, at which time we notified you, and our partners in the Xen community, of the need for an urgent server reboot.  Even then, to avoid alerting cyber criminals, we didn’t mention Xen as the reason for the reboot. Another major cloud provider did attribute its reboot to security problems with Xen, which put all users of the affected versions of that hypervisor at heightened risk.  But we’re relieved to report that, as of now, we’ve learned of no data compromise among Rackspace customers.  Now that the vulnerability has been fully remediated, the Xen community has lifted its embargo on talking about it.

  Those of you who are longtime Rackspace customers know that we have a strong record of open, timely communication with you. We reach out to you whenever there’s an issue.  We answer the phone whenever you call. We do everything we can to find a solution.  This past weekend, our engineers worked tirelessly with customers and partners to remediate the Xen vulnerability.  

  This maintenance affected nearly a quarter of our 200,000-plus customers, and in the course of it, we dropped a few balls.  Some of our reboots, for example, took much longer than they should.  And some of our notifications were not as clear as they should have been. We are making changes to address those mistakes.  And we welcome your feedback on how we can better serve you.

  As a veteran Racker who is proud of our commitment to our customers and their businesses, I am personally sorry for any inconvenience or downtime that we caused you during this incident.

  Sincerely, 

  Taylor Rhodes
  CEO and President
  Rackspace

  taylor.rhodes@rackspace.com

We'd definitely like to use it too - please share! :D

Don't lie. I had ~120 instances I had to juggle between 3 availability zones, and never were two AZs down/rebooted at once. Our environment suffered no downtime, as we had at least 2-3 days notice per AZ to move instances around.

Rackspace's handling of the situation was a joke. They sent notification emails out at 9:30pm on a Friday night, and then proceeded to do the reboots Saturday at peak traffic times.

> Their philosophy, as is pervasive in the hosting industry, is that they have paying customers so whatever they are doing must be right. Obviously Amazon often also seems to act this way, but this particular maintenance was handled well afaict, and availability zones showed their value.

This could explain why Rackspace was shopped around by Morgan Stanley. They may be profitable now, but Amazon and Google are going to eat their lunch.

Shopped around, with no takers. At last count it was reported they've given up on that.