Live data from Hacker News

EC2 Maintenance Update II

aws.amazon.com

21–30 of 60 posts

Re: EC2 Maintenance Update II

#21

We noticed that all of our instances -- in 3 different availability zones in US East -- all cycled at the same time. That was pretty disappointing. Kind of defeated the purpose of having things in different AZs.

> you may also want to take this opportunity to re-examine your AWS architecture to look for possible ways to make it even more fault-tolerant Basically they won't allow blame to be placed on them for anything they do. According to AWS, it's your fault this happened.

Amazon had to perform the maintenance due to XSA 108, and the timetable they had to meet was set by the Xen project.

They set up the maintenance to have as little impact as possible by splitting availability zones onto separate days so that people utilizing multiple zones for high-availability would not lose multiple zones at once.

Afterwards, they gave a detailed breakdown, linking to the vulnerability and explaining both why they had to perform maintenance and why they could not share more details upfront.

They also provided information on how to improve the fault-tolerance of your systems so that future issues like this won't stop your workflow.

Amazon isn't placing "fault" on users; they did a pretty stellar job of handling a nasty vulnerability and treating their users as well as possible.

Re: EC2 Maintenance Update II

#22
post #2

So this had nothing to do with Shellshock then?

Shellshock, being a bash bug (and affecting systems that use bash for shell commands, etc), is OS and/or application level, and the responsibility of the individual customers to handle (as they are responsible for administration of the instances they run). The Xen bug affects how the hardware is virtualized, and this Amazon's responsibility.

Re: EC2 Maintenance Update II

#24

@jeffbarr I just wanted to say thanks for not only posting this, but also sticking around in the comments section. Makes you + Amazon seem way more human :-) Also, in case you have any "cloud servers" you want to decommission: https://news.ycombinator.com/item?id=8373394

I am always happy to help, time and circumstances allowing.

Before joining Amazon I earned my living by consulting for startups. I could always tell when they were about to run out of money when they would offer to pay me in servers. This was always the cue to find my next gig.

Re: EC2 Maintenance Update II

#25
post #23
post #6

This is a better announcement than Rackspace's: no spin, direct, to the point.

"An Apology": https://community.rackspace.com/general/f/34/t/4341

This sentence, early in that post, is where my spin shields went up:

Now that this issue has been fully remediated, without any reports of compromised data among our customers, I’d like to explain what happened, and why. [em mine]

Re: EC2 Maintenance Update II

#27
post #20

Earlier quoted context omitted.

You can use the DescribeInstanceStatus API to get hold of scheduled events for an instance. http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ApiRef...

We use this API to post notices into a Slack channel. It's been very effective.

Very cool, mind sharing that script ?

Re: EC2 Maintenance Update II

#28
post #8

Earlier quoted context omitted.

What kinds of alerts would work for you? Let me know and I will pass them along to the team.

I would like to be able to have alerts go right into SNS.

+1 on this.

We got the update on the EC2 reboots, but totally missed the RDS reboot information and as such suffered some downtime through the reboots on Friday (that we could have avoided had we known about it). It would be nice if the RDS console supported the "Scheduled Events" section that the EC2 console had.

Re: EC2 Maintenance Update II

#30

This explains some weird EC2 stuff we saw yesterday. A good reminder to check up that a box starts up as expected (mounts disks, starts services, etc). It's easy to get lazy about that sort of thing when you're moving fast.

Same here, the VM rebooted and restarted all services correctly but the connectivity was failing intermittently. Not fun.

Concluded we better just start a new VM... and it worked.

Next time there is such scheduled event, I guess being proactive and creating a new VM beforehand is the better solution.

Post reply on HN