Live data from Hacker News

AWS issues unavoidable reboot schedules with short notice on many EC2 instances

forums.aws.amazon.com

41–50 of 160 posts

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#41
post #27

There's a possibility that this issue might be related to HVM. All of our AWS systems are on older, non-hvm instance types, and none have been rebooted, and there are no maintenance events listed. Friends who are using newer instances (which are all hvm) are reporting the reboot issues. Anecdata, but a possible explanation.

this has been true for us so far as well. But I wouldn't rule out that they have more scheduled maintenances that haven't been put out yet.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#43

While people may be painting Amazon in a bad light here, the business-level risk of wholly committing to a single infrastructure provider (cloud or otherwise, across multiple 'availability zones' or data centers or countries or continents, or otherwise) is real. There is a clear need for many service authors to work with disparate infrastructure in a cloud provider and platform abstracted manner, and arguably no soli…

Curious, what do large scale companies in the cloud currently do for this?

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#44

On AWS you should be prepared for an instance to disappear at any time, for any reason. Why is a scheduled reboot such a big deal?

It's a big deal because about half of my 100 instances are all going down at roughly the same time. Distribution and replication save you if you have 3 boxes and 1 dies. If all 3 die at the same time, you're still screwed.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#45

Pretty crappy that you cannot immediately check if an instance has restarted on a patched host or not. I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option. Having to wait possibly hours for an email to see if your vm migrated to patched host or not is a terrible solution.

> I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option. I would expect Amazon to mark unpatched hosts as bad and not permit new instances to be deployed to them, similar to queue draining. Not cool Amazon.

This is normally what they do when they roll out forced reboots like this.

Note the forum posting says there isn't guarantee of being on an updated host...that is because the patching isn't complete across the region yet.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#46
post #44

On AWS you should be prepared for an instance to disappear at any time, for any reason. Why is a scheduled reboot such a big deal?

It's a big deal because about half of my 100 instances are all going down at roughly the same time. Distribution and replication save you if you have 3 boxes and 1 dies. If all 3 die at the same time, you're still screwed.

Each Availability Zone is being rebooted on a different day. Best practices dictate HA clusters with >=1 instance in each AZ. So, in theory well-designed EC2 systems can withstand this without interruption.

But yeah, this still sucks.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#47

While people may be painting Amazon in a bad light here, the business-level risk of wholly committing to a single infrastructure provider (cloud or otherwise, across multiple 'availability zones' or data centers or countries or continents, or otherwise) is real. There is a clear need for many service authors to work with disparate infrastructure in a cloud provider and platform abstracted manner, and arguably no soli…

Is Rightscale's multi-cloud offering a step in the direction you're advocating?

http://assets.rightscale.com/uploads/pdfs/RightScale-Technic...

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#48
post #44

Earlier quoted context omitted.

It's a big deal because about half of my 100 instances are all going down at roughly the same time. Distribution and replication save you if you have 3 boxes and 1 dies. If all 3 die at the same time, you're still screwed.

Each Availability Zone is being rebooted on a different day. Best practices dictate HA clusters with >=1 instance in each AZ. So, in theory well-designed EC2 systems can withstand this without interruption. But yeah, this still sucks.

That does seem to be the case, that is a relief.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#50
post #27

There's a possibility that this issue might be related to HVM. All of our AWS systems are on older, non-hvm instance types, and none have been rebooted, and there are no maintenance events listed. Friends who are using newer instances (which are all hvm) are reporting the reboot issues. Anecdata, but a possible explanation.

Concur, though also anecdotal.
Post reply on HN