Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS issues unavoidable reboot schedules with short notice on many EC2 instances

forums.aws.amazon.com

Post reply on HN
Print view View original

61–70 of 160 posts

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#61

Post by fletchowns »

reedlaw wrote: post #57
fletchowns wrote: post #42

On AWS you should be prepared for an instance to disappear at any time, for any reason. Why is a scheduled reboot such a big deal?

That's true, but how many services get this right in practice? It's not easy to ensure all instances will perform well in case of failover. A lot of apps are running on a single instance and depend on Amazon's fairly good track record.

Obviously these individuals have never visited the EC2 forums: https://forums.aws.amazon.com/search.jspa?objID=f30&q=unresp...

If you are relying on a single instance and depending on Amazon's track record, you're doing it wrong.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#62

Post by eropple »

all_usernames wrote: post #46
omni wrote: post #44

Earlier quoted context omitted.

It's a big deal because about half of my 100 instances are all going down at roughly the same time. Distribution and replication save you if you have 3 boxes and 1 dies. If all 3 die at the same time, you're still screwed.

Each Availability Zone is being rebooted on a different day. Best practices dictate HA clusters with >=1 instance in each AZ. So, in theory well-designed EC2 systems can withstand this without interruption. But yeah, this still sucks.

One thing I'd add: best practices (IMO) dictate HA clusters as you describe, but you get a big boost to survivability by deciding on only using instance stores. Network issues have screwed EBS in the past; EBS is technically neat but very network-sensitive and it's possible to "lose" part of your EBS volume because part of the network goes away (and then your instance faceplants). Instance stores are your friend, and acutely knowing they can disappear in an eyeblink will make you design a better system. One that can survive you having your instances forcibly retired by AWS. :-)

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#63

Post by [deleted] »

spullara wrote: post #60
gpjonesii wrote: post #58

A couple of points: Autoscaling is your friend. If you're not leveraging it (multiple availability zones), you're doing it wrong. Even single instances can be launched in autoscaling groups with a desired capacity of 1 to ensure that if it falls over, a new one is spun up. Point 2: AWS is likely trying to rotate capacity for updates, which means they need to evict instances. That are running on doms that they need to…

They are not rotating capacity for updates. They are patching a Xen security issue that will be announced on Oct 1. That is why they are rebooting machines and not forcing moves off of those machines. Otherwise, I agree with the advice.

[deleted]

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#64

Post by Xorlev »

gpjonesii wrote: post #58

A couple of points: Autoscaling is your friend. If you're not leveraging it (multiple availability zones), you're doing it wrong. Even single instances can be launched in autoscaling groups with a desired capacity of 1 to ensure that if it falls over, a new one is spun up. Point 2: AWS is likely trying to rotate capacity for updates, which means they need to evict instances. That are running on doms that they need to…

This helps services, but as some point you have to run a database layer too ;) Cassandra helps, but it's not the whole story during large-scale close together reboots like these.

It ends up being a decent bit of manual operator time spent when security patches force AWS to reboot. You have to be pretty careful about any service that can't lose members as quickly due to bootstrap times or technology limitations, e.g. RDBMSes.

For things running in an ASG, it's trivial to let it die or just kill it.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#65

Post by eropple »

contingencies wrote: post #54
btown wrote: post #47

Earlier quoted context omitted.

Is Rightscale's multi-cloud offering a step in the direction you're advocating? http://assets.rightscale.com/uploads/pdfs/RightScale-Technic...

Kind of. At a glance, it's new, commercial and they gloss over the complexities... therefore I'm skeptical it really works as well as they say it does, and is leaning toward my 'untrustworthy as a long term platform' basket. Though they may have great tools, I believe history shows us that open source is the real way to resolve these very reasonable types of architectural concerns.

RightScale's single-cloud offering doesn't seem that great, I'd be really worried about them having multi-cloud support.

I have some ideas around a project for "cloud abstraction" - kind of PaaS-as-a-service (we have to go deeeeeper) - but only some early thoughts right now.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#66

Post by monjinan »

I don't think a lot of people are really understanding how much of a larger issue this would be for us if AWS didn't patch a major security issue before it was made public.

The company has treated me very well over the years, from AWS to retail. My stuff arrives on time, if it doesn't I get reimbursed, most of the time with an extra few bucks for my trouble. The AWS platform is more mature and feature rich than anyone else and keeps getting better.

These reboots are going to save a lot of peoples butts, they wouldn't give a 48 hour notice if they had a better option.

That being said, I'm very curious to see what info is released next week.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#67

Post by Corrado »

rands3311 wrote: post #14
Corrado wrote: post #7

They say that its fine to reboot but that there is no guarantee that you will land on an updated host. However, AWS does provide a script to run on Windows machines which should tell you if that particular machine has the issue. I took a quick look at the script and deduced that it is indeed a Xen issue.

This is not related to the scheduled reboot maintenance. The script that you are referring to has to do with a separate Windows networking driver and Xen.

Indeed, it appears as if you are correct [1]. This issue is with Windows Server 2012 R2 network drivers as is from Sep. 17, 2014. If you're interested in the script you can download it from that page.

[1] http://aws.amazon.com/windows/2012r2-network-drivers/

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#68

Post by monjinan »

fletchowns wrote: post #61
reedlaw wrote: post #57

Earlier quoted context omitted.

That's true, but how many services get this right in practice? It's not easy to ensure all instances will perform well in case of failover. A lot of apps are running on a single instance and depend on Amazon's fairly good track record.

Obviously these individuals have never visited the EC2 forums: https://forums.aws.amazon.com/search.jspa?objID=f30&q=unresp... If you are relying on a single instance and depending on Amazon's track record, you're doing it wrong.

Do you have any actual sources of their average instance uptime over the year?

I see an SLA of 99.95%(~4hrs/year) on the site but cloudharmony.com/status has AWS at the top. Googling average ec2 uptime has people posting instances running for years with nothing.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#69

Post by dvidsilva »

monjinan wrote: post #66

I don't think a lot of people are really understanding how much of a larger issue this would be for us if AWS didn't patch a major security issue before it was made public. The company has treated me very well over the years, from AWS to retail. My stuff arrives on time, if it doesn't I get reimbursed, most of the time with an extra few bucks for my trouble. The AWS platform is more mature and feature rich than anyon…

yeah, we had a few minutes downtime but nothing too bad; definitely rather have some few time off than a security threat.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#70

Post by pjl »

vacri wrote: post #27

There's a possibility that this issue might be related to HVM. All of our AWS systems are on older, non-hvm instance types, and none have been rebooted, and there are no maintenance events listed. Friends who are using newer instances (which are all hvm) are reporting the reboot issues. Anecdata, but a possible explanation.

I don't think the issue is related to HVM. I have instances scheduled for system reboot that are not HVM.
Post reply on HN