Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS issues unavoidable reboot schedules with short notice on many EC2 instances

forums.aws.amazon.com

Post reply on HN
Print view View original

31–40 of 160 posts

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#31

Post by iaw »

rurounijones wrote: post #30

Pretty crappy that you cannot immediately check if an instance has restarted on a patched host or not. I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option. Having to wait possibly hours for an email to see if your vm migrated to patched host or not is a terrible solution.

IMO, The problem is that the manager of the cloud product isn't the person that is ordering the reboot. The manager of the cloud product considers communication with customers to mitigate their issues a primary concern. Person ordering the reboot is more 'technically' focused and representative of why engineers have a hard time as managers (eg ignoring the potential ill will this type of action can create).

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#32

Post by toomuchtodo »

rurounijones wrote: post #30

Pretty crappy that you cannot immediately check if an instance has restarted on a patched host or not. I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option. Having to wait possibly hours for an email to see if your vm migrated to patched host or not is a terrible solution.

> I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option.

I would expect Amazon to mark unpatched hosts as bad and not permit new instances to be deployed to them, similar to queue draining.

Not cool Amazon.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#33

Post by adrr »

vetrom wrote: post #3
robszumski wrote: post #2

I'm assuming this is due to fallout from the recent bash security issue?

Swapping bash should not require reboots. The pending XSA-108 ( http://xenbits.xen.org/xsa/ ) however....

It has to be that. Can't think of another reason why you need to mass reboot the servers the instances are running on and serious enough they are patching through the weekend. Wonder if its being exploited in the wild. Also none of our C3.2Xlarges are getting rebooted, those use SR-IOV which can virtualize PCIE devices at the hardware level so it probably a network boundary exploit allow you to pull other instance traffic.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#34

Post by rurounijones »

toomuchtodo wrote: post #32
rurounijones wrote: post #30

Pretty crappy that you cannot immediately check if an instance has restarted on a patched host or not. I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option. Having to wait possibly hours for an email to see if your vm migrated to patched host or not is a terrible solution.

> I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option. I would expect Amazon to mark unpatched hosts as bad and not permit new instances to be deployed to them, similar to queue draining. Not cool Amazon.

Good point, I wonder if they didn't because that would have crippled capacity due to people rebooting, using up the patched hosts and leaving others without the ability to spin up VMs as it sounds like a significant % of hosts are bad at this point..

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#35

Post by rands3311 »

toomuchtodo wrote: post #32
rurounijones wrote: post #30

Pretty crappy that you cannot immediately check if an instance has restarted on a patched host or not. I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option. Having to wait possibly hours for an email to see if your vm migrated to patched host or not is a terrible solution.

> I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option. I would expect Amazon to mark unpatched hosts as bad and not permit new instances to be deployed to them, similar to queue draining. Not cool Amazon.

Ideally, that would be great. I doubt that would be an option though due to capacity. Would you rather stop/start your instance and risk a capacity error or have your impacted instances rebooted in 48 hours?

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#36

Post by toomuchtodo »

rands3311 wrote: post #35
toomuchtodo wrote: post #32

Earlier quoted context omitted.

> I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option. I would expect Amazon to mark unpatched hosts as bad and not permit new instances to be deployed to them, similar to queue draining. Not cool Amazon.

Ideally, that would be great. I doubt that would be an option though due to capacity. Would you rather stop/start your instance and risk a capacity error or have your impacted instances rebooted in 48 hours?

> Would you rather stop/start your instance and risk a capacity error or have your impacted instances rebooted in 48 hours?

Is there that little slack in Amazon's compute capacity? I would hope not! If there isn't capacity to start my instance back up, I would hope that hitting Stop would generate a dialog to the effect of "Hey there, you won't be able to start this instance back up if you stop it right now."

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#37

Post by x0x0 »

rurounijones wrote: post #30

Pretty crappy that you cannot immediately check if an instance has restarted on a patched host or not. I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option. Having to wait possibly hours for an email to see if your vm migrated to patched host or not is a terrible solution.

And having dealt with ec2 support, I would have expected amazon to half-ass this just the way they did.

Another reason to consider owned & operated hardware. Or at least something like monthly rentals.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#38

Post by rands3311 »

toomuchtodo wrote: post #36
rands3311 wrote: post #35

Earlier quoted context omitted.

Ideally, that would be great. I doubt that would be an option though due to capacity. Would you rather stop/start your instance and risk a capacity error or have your impacted instances rebooted in 48 hours?

> Would you rather stop/start your instance and risk a capacity error or have your impacted instances rebooted in 48 hours? Is there that little slack in Amazon's compute capacity? I would hope not! If there isn't capacity to start my instance back up, I would hope that hitting Stop would generate a dialog to the effect of "Hey there, you won't be able to start this instance back up if you stop it right now."

There very well could be. If this issue is related to only handful of instance types then there goes all that extra capacity.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#40

Post by contingencies »

While people may be painting Amazon in a bad light here, the business-level risk of wholly committing to a single infrastructure provider (cloud or otherwise, across multiple 'availability zones' or data centers or countries or continents, or otherwise) is real. There is a clear need for many service authors to work with disparate infrastructure in a cloud provider and platform abstracted manner, and arguably no solid tools for doing it right now.

What will the future look like? I believe that a standard, git-like command line tool for infrastructure, sort of a 'brandless ec2' or 'P-abstracted IaaS' version of heroku, will replace all of the current-era providers with a free market for infrastructure based on transparency, real uptime and performance analysis and incident observations by multiple third parties with cryptographic reputation management. Two angles converging on that at http://stani.sh/walter/pfcts and http://ifex-project.org/

Post reply on HN