Live data from Hacker News

EC2 Maintenance Update II

aws.amazon.com

11–20 of 60 posts

Re: EC2 Maintenance Update II

#11

We noticed that all of our instances -- in 3 different availability zones in US East -- all cycled at the same time. That was pretty disappointing. Kind of defeated the purpose of having things in different AZs.

> you may also want to take this opportunity to re-examine your AWS architecture to look for possible ways to make it even more fault-tolerant

Basically they won't allow blame to be placed on them for anything they do. According to AWS, it's your fault this happened.

Re: EC2 Maintenance Update II

#12

We noticed that all of our instances -- in 3 different availability zones in US East -- all cycled at the same time. That was pretty disappointing. Kind of defeated the purpose of having things in different AZs.

Hm... interesting, I had the opposite experience. It was the "aha" moment for me when instances on us-west-1a and us-west-1b were rebooted at different times, which let us avoid any downtime.

Re: EC2 Maintenance Update II

#13

We noticed that all of our instances -- in 3 different availability zones in US East -- all cycled at the same time. That was pretty disappointing. Kind of defeated the purpose of having things in different AZs.

We did a zone-by-zone reboot. If you want to send me the instance ids I will ask our team to see what happened. You can find my email in my profile.

Re: EC2 Maintenance Update II

#14
post #8

Earlier quoted context omitted.

What kinds of alerts would work for you? Let me know and I will pass them along to the team.

I would like to be able to have alerts go right into SNS.

More generally, it would be cool if every account automatically got two free read-only SQS queues: one for events and one for upcoming events. Every known event (startup, terminate, permission change, network error, etc) could be published to the queue.

Not high-priority for us but the parent comment sparked the idea (I'm not in devops so maybe this exists via a different mechanism.)

Re: EC2 Maintenance Update II

#15

We noticed that all of our instances -- in 3 different availability zones in US East -- all cycled at the same time. That was pretty disappointing. Kind of defeated the purpose of having things in different AZs.

We did a zone-by-zone reboot. If you want to send me the instance ids I will ask our team to see what happened. You can find my email in my profile.

How much delay is between zones? Some peoples services don't come up instantly. Perhaps when zone 3 when down, the users services in zone 1 hadn't finished coming back up.

Re: EC2 Maintenance Update II

#17

"Pay attention to your Inbox and to the alerts on the AWS Management Console." Especially with incidents like these (and other cases where instances are scheduled to be taken down), it really annoys me that AWS doesn't offer any push alerts besides emailing the account owner.

You can use the DescribeInstanceStatus API to get hold of scheduled events for an instance.

http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ApiRef...

Re: EC2 Maintenance Update II

#18
post #14

Earlier quoted context omitted.

I would like to be able to have alerts go right into SNS.

More generally, it would be cool if every account automatically got two free read-only SQS queues: one for events and one for upcoming events. Every known event (startup, terminate, permission change, network error, etc) could be published to the queue. Not high-priority for us but the parent comment sparked the idea (I'm not in devops so maybe this exists via a different mechanism.)

These are both great ideas. I will share them with the team today. Keep them coming!

Re: EC2 Maintenance Update II

#19

Earlier quoted context omitted.

We did a zone-by-zone reboot. If you want to send me the instance ids I will ask our team to see what happened. You can find my email in my profile.

How much delay is between zones? Some peoples services don't come up instantly. Perhaps when zone 3 when down, the users services in zone 1 hadn't finished coming back up.

They did a different zone every day, so ~24 hours.

Re: EC2 Maintenance Update II

#20

"Pay attention to your Inbox and to the alerts on the AWS Management Console." Especially with incidents like these (and other cases where instances are scheduled to be taken down), it really annoys me that AWS doesn't offer any push alerts besides emailing the account owner.

You can use the DescribeInstanceStatus API to get hold of scheduled events for an instance. http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ApiRef...

We use this API to post notices into a Slack channel. It's been very effective.
Post reply on HN