This feels like a critical security or stability update.
XSA-108 perhaps? http://xenbits.xen.org/xsa/
AWS is certainly large enough to have access to an embargoed update.
AWS issues unavoidable reboot schedules with short notice on many EC2 instances
71–80 of 160 posts
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#72Earlier quoted context omitted.
How would owned & operated hardware or monthly rentals be better?
You might not be vulnerable at all (if it is a Xen problem), or you might not have to patch so urgently. More to the point, by operating your own systems, the maintenance window is set by your staff and not Amazon.
I'm not aware of Xen being particularly susceptible to security vulnerabilities. Any software you have on your box could end up being compromised. I'm not sure why it's less urgent if you have your own hardware. And of course the other downside is that while you set maintenance windows, you also have to do all the maintenance. Not that owning your own hardware isn't something worth consider, just not sure that's the most compelling argument.
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#73Earlier quoted context omitted.
How would owned & operated hardware or monthly rentals be better?
You might not be vulnerable at all (if it is a Xen problem), or you might not have to patch so urgently. More to the point, by operating your own systems, the maintenance window is set by your staff and not Amazon.
Apples to oranges if you're comparing bare-metal to xen/ec2.
If you're just worried about being able to manage your own window you won't have access to a embargoed xen issue before it becomes public. It's the first time they forced a 2 day scheduled event on me, I'll error on the side of caution and take my reboot.
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#74Earlier quoted context omitted.
> I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option. I would expect Amazon to mark unpatched hosts as bad and not permit new instances to be deployed to them, similar to queue draining. Not cool Amazon.
There is not enough capacity. I think Amazon did not make this decision lightly.
You must construct additional pylons.
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#75There's a possibility that this issue might be related to HVM. All of our AWS systems are on older, non-hvm instance types, and none have been rebooted, and there are no maintenance events listed. Friends who are using newer instances (which are all hvm) are reporting the reboot issues. Anecdata, but a possible explanation.
We have a lot of non-HVM scheduled for reboot.
One of them was our VPN server, scheduled for reboot during our tech demo.
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#76this might be the xen vulnerability that is as yet undisclosed to the public.
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#77Earlier quoted context omitted.
Each Availability Zone is being rebooted on a different day. Best practices dictate HA clusters with >=1 instance in each AZ. So, in theory well-designed EC2 systems can withstand this without interruption. But yeah, this still sucks.
One thing I'd add: best practices (IMO) dictate HA clusters as you describe, but you get a big boost to survivability by deciding on only using instance stores. Network issues have screwed EBS in the past; EBS is technically neat but very network-sensitive and it's possible to "lose" part of your EBS volume because part of the network goes away (and then your instance faceplants). Instance stores are your friend, and…
You use instance stores for persistent data. That instance disappears. Where are you restoring that data from? Either your backups are stale, or you were replicating the data or its underlying filesystem, which means you're still reliant on the network.
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#78I guess if you already put your machines in different availability zones, they will not reboot in the same maintenance window.
Actually AWS scheduled maintenance is the best you can get in the market already, because most of the time they allow you to reboot yourself in order to land an updated host (This time is different might be due to some critical security issues). Other providers like Azure / Google, you have no choice.
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#79I guess if you already put your machines in different availability zones, they will not reboot in the same maintenance window. Actually AWS scheduled maintenance is the best you can get in the market already, because most of the time they allow you to reboot yourself in order to land an updated host (This time is different might be due to some critical security issues). Other providers like Azure / Google, you have n…
Google Compute Engine offers live migration around maintenance events: https://cloud.google.com/compute/docs/instances#onhostmainte...
(note: I work on GCE, more or less)
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#80Earlier quoted context omitted.
You might not be vulnerable at all (if it is a Xen problem), or you might not have to patch so urgently. More to the point, by operating your own systems, the maintenance window is set by your staff and not Amazon.
I'm not aware of Xen being particularly susceptible to security vulnerabilities. Any software you have on your box could end up being compromised. I'm not sure why it's less urgent if you have your own hardware. And of course the other downside is that while you set maintenance windows, you also have to do all the maintenance. Not that owning your own hardware isn't something worth consider, just not sure that's the…
There are many pros and cons to EC2 vs managed hosting vs operating it yourself.
I meant that in this particular case, you are likely to have fewer headaches from this issue in your own datacenter because:
1. You probably don't run Xen in the first place.
2. If you do, this vulnerability may not be critical to you. I'm guessing that it is exploitable via the other instances on a host. If you don't share hardware with other companies, then a co-tenancy exploit may not be a huge deal.
3. And if the vulnerability is critical to you, you may be able to better schedule the maintenance for your own particular business needs. Fix half your hosts, fail over, fix the other half; schedule it during your particular low traffic period.
The scope of the issue is also smaller, you just have to solve the problem for your use cases rather than the massive undertaking this is for Amazon.
I don't think this one incident means EC2 is bad and real hardware is good, forever and ever. There are tradeoffs to everything.