Will new instances created now not have to be rebooted? A lot of people on EC2 can probably re-create their vms now instead of having to wait for the reboot, no?
AWS issues unavoidable reboot schedules with short notice on many EC2 instances
51–60 of 160 posts
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#52Pretty crappy that you cannot immediately check if an instance has restarted on a patched host or not. I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option. Having to wait possibly hours for an email to see if your vm migrated to patched host or not is a terrible solution.
> I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option. I would expect Amazon to mark unpatched hosts as bad and not permit new instances to be deployed to them, similar to queue draining. Not cool Amazon.
There is not enough capacity. I think Amazon did not make this decision lightly.
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#53Pretty crappy that you cannot immediately check if an instance has restarted on a patched host or not. I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option. Having to wait possibly hours for an email to see if your vm migrated to patched host or not is a terrible solution.
And having dealt with ec2 support, I would have expected amazon to half-ass this just the way they did. Another reason to consider owned & operated hardware. Or at least something like monthly rentals.
How would owned & operated hardware or monthly rentals be better?
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#54While people may be painting Amazon in a bad light here, the business-level risk of wholly committing to a single infrastructure provider (cloud or otherwise, across multiple 'availability zones' or data centers or countries or continents, or otherwise) is real. There is a clear need for many service authors to work with disparate infrastructure in a cloud provider and platform abstracted manner, and arguably no soli…
Is Rightscale's multi-cloud offering a step in the direction you're advocating? http://assets.rightscale.com/uploads/pdfs/RightScale-Technic...
Kind of. At a glance, it's new, commercial and they gloss over the complexities... therefore I'm skeptical it really works as well as they say it does, and is leaning toward my 'untrustworthy as a long term platform' basket. Though they may have great tools, I believe history shows us that open source is the real way to resolve these very reasonable types of architectural concerns.
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#55Will new instances created now not have to be rebooted? A lot of people on EC2 can probably re-create their vms now instead of having to wait for the reboot, no?
At the time of writing, no. See the linked discussion. The real issue here, as the RightScale guys point out, is that there is no way to reliably provision a patched instance. Although the comment in the discussion forum by the EC2 fleet manager says they are working on a tool that will let us know whether an instance is fixed or not.
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#56Earlier quoted context omitted.
And having dealt with ec2 support, I would have expected amazon to half-ass this just the way they did. Another reason to consider owned & operated hardware. Or at least something like monthly rentals.
How would owned & operated hardware or monthly rentals be better?
You might not be vulnerable at all (if it is a Xen problem), or you might not have to patch so urgently.
More to the point, by operating your own systems, the maintenance window is set by your staff and not Amazon.
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#57On AWS you should be prepared for an instance to disappear at any time, for any reason. Why is a scheduled reboot such a big deal?
That's true, but how many services get this right in practice? It's not easy to ensure all instances will perform well in case of failover. A lot of apps are running on a single instance and depend on Amazon's fairly good track record.
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#58A couple of points:
Autoscaling is your friend. If you're not leveraging it (multiple availability zones), you're doing it wrong. Even single instances can be launched in autoscaling groups with a desired capacity of 1 to ensure that if it falls over, a new one is spun up.
Point 2: AWS is likely trying to rotate capacity for updates, which means they need to evict instances. That are running on doms that they need to update/deprecate/etc. The longer your instances are running (or the more specialized the type of instance is), the more likely you'll see an eviction notice. It should be part of a good practice to launch new instances often as new AMIs become available, or as private AMIs are updated for security patches, etc. - at least monthly! Autoscaling and solid config management simplifies this practice greatly.
Good Luck!
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#59This feels like a critical security or stability update.
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#60A couple of points: Autoscaling is your friend. If you're not leveraging it (multiple availability zones), you're doing it wrong. Even single instances can be launched in autoscaling groups with a desired capacity of 1 to ensure that if it falls over, a new one is spun up. Point 2: AWS is likely trying to rotate capacity for updates, which means they need to evict instances. That are running on doms that they need to…
They are not rotating capacity for updates. They are patching a Xen security issue that will be announced on Oct 1. That is why they are rebooting machines and not forcing moves off of those machines. Otherwise, I agree with the advice.