Live data from Hacker News

AWS issues unavoidable reboot schedules with short notice on many EC2 instances

forums.aws.amazon.com

21–30 of 160 posts

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#21
post #17

So far I'm 3 for 3 on stopped/started machines that come back no longer having the scheduled reboot event - maybe the problem isn't as annoying as we're thinking? (Edited because I said "reboot" and meant "stop/start")

The post claims that it could take several hours before their scans identify hosts that were claimed during restarts.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#22

I love how the amazon support person is absolutely not answering the very precise question.

Heh, I thought the answer was very clear: no.

I didn't read "no" in the answer, just circonvolutions around: "it's no but we're mangling it, plus we're not changing our course after your feedback". A clear answer would be: "fuck you".

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#23
post #18
post #12

Earlier quoted context omitted.

It's not the results of the script that I'm referring to, it's the contents of the script. If I hand you code that can look at your system and determine something about it, you can look at what the code is doing and identify what it is looking at, which tells you where the vulnerability is. It becomes moderately harder if it's compiled code, but still not very difficult.

So make the code query an opaque EC2 api, instead of testing the machine. You could still find a machine that is vulnerable and one that isn't and attempt to find out what the difference is, but that's a much harder task.

True, and if that's the case, my concern is resolved.

That said, if they were just checking software version or querying an Amazon API endpoint, I'd expect them to give out a tool or URL you could use that would give you the state for all your systems at once, rather than a script that you'd run on the machine itself.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#24
post #17

So far I'm 3 for 3 on stopped/started machines that come back no longer having the scheduled reboot event - maybe the problem isn't as annoying as we're thinking? (Edited because I said "reboot" and meant "stop/start")

A reboot won't address it at all (it also keeps you on the same host).. no point in doing that. From the e-mail notification: "You will not be able to stop/start or re-launch instances in order to avoid this maintenance update."

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#25
post #17

So far I'm 3 for 3 on stopped/started machines that come back no longer having the scheduled reboot event - maybe the problem isn't as annoying as we're thinking? (Edited because I said "reboot" and meant "stop/start")

A reboot won't address it at all (it also keeps you on the same host).. no point in doing that. From the e-mail notification: "You will not be able to stop/start or re-launch instances in order to avoid this maintenance update."

Sorry, I should have phrased better. I issued a "stop and start". Per the AWS thread:

> While executing a stop/start is absolutely fine to do, there is not a guarantee that you will land on an updated host. We are periodically polling for new instances and those impacted will receive new maintenance notifications accordingly.

Which I take to mean there's a chance you could land on an updated host. I'll update if any of those machines come back to the Events view.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#26
This isn't a direct answer, but for others who are coming to this page and need help on where to start, here is some info on what to do now from RightScale's CTO, it also has a little comparison to what happened in Dec 2011 reboot too (just an interesting side note): http://www.rightscale.com/blog/rightscale-news/aws-reboot-su...

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#27
There's a possibility that this issue might be related to HVM. All of our AWS systems are on older, non-hvm instance types, and none have been rebooted, and there are no maintenance events listed. Friends who are using newer instances (which are all hvm) are reporting the reboot issues.

Anecdata, but a possible explanation.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#28
post #25

Earlier quoted context omitted.

A reboot won't address it at all (it also keeps you on the same host).. no point in doing that. From the e-mail notification: "You will not be able to stop/start or re-launch instances in order to avoid this maintenance update."

Sorry, I should have phrased better. I issued a "stop and start". Per the AWS thread: > While executing a stop/start is absolutely fine to do, there is not a guarantee that you will land on an updated host. We are periodically polling for new instances and those impacted will receive new maintenance notifications accordingly. Which I take to mean there's a chance you could land on an updated host. I'll update if any…

[deleted]

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#30
Pretty crappy that you cannot immediately check if an instance has restarted on a patched host or not.

I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option.

Having to wait possibly hours for an email to see if your vm migrated to patched host or not is a terrible solution.

Post reply on HN