Live data from Hacker News

AWS issues unavoidable reboot schedules with short notice on many EC2 instances

forums.aws.amazon.com

91–100 of 160 posts

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#91
post #81

These instance types are not affected: T1, T2, M2, R3, and HS1 instance types are not affected. http://www.rightscale.com/blog/rightscale-news/aws-reboot-su...

I'd pour one out for the people with the gigantic I2's that they'll have to reprovision, but I'm pretty sure they can afford their own, they don't need me to pour one out for them.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#92
post #88
post #83

CVE-2014-6271 was posted on the AWS Security Bulletin earlier today: http://aws.amazon.com/security/security-bulletins/CVE_2014_6... My guess is that they cannot rely on all users to perform these updates.

can someone explain the downvotes?

Already confirmed by AWS its not related. Confirmed in this thread its not related. That issue is an OS level problem, AWS won't touch your OS.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#94

A couple of points: Autoscaling is your friend. If you're not leveraging it (multiple availability zones), you're doing it wrong. Even single instances can be launched in autoscaling groups with a desired capacity of 1 to ensure that if it falls over, a new one is spun up. Point 2: AWS is likely trying to rotate capacity for updates, which means they need to evict instances. That are running on doms that they need to…

They are not rotating capacity for updates. They are patching a Xen security issue that will be announced on Oct 1. That is why they are rebooting machines and not forcing moves off of those machines. Otherwise, I agree with the advice.

Could you please confirm or provide evidence for such speculation?

I don't see anything about this on google.

I heard it is because they are having power issues within their datacenter.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#95

Earlier quoted context omitted.

They are not rotating capacity for updates. They are patching a Xen security issue that will be announced on Oct 1. That is why they are rebooting machines and not forcing moves off of those machines. Otherwise, I agree with the advice.

Could you please confirm or provide evidence for such speculation? I don't see anything about this on google. I heard it is because they are having power issues within their datacenter.

May be this: http://xenbits.xen.org/xsa/

XSA-108 2014-10-01 12:00 (Prereleased, but embargoed)

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#96
post #95

Earlier quoted context omitted.

Could you please confirm or provide evidence for such speculation? I don't see anything about this on google. I heard it is because they are having power issues within their datacenter.

May be this: http://xenbits.xen.org/xsa/ XSA-108 2014-10-01 12:00 (Prereleased, but embargoed)

Cool speculation!

Anyone with anything concrete?

I still heard it is power issues.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#97

Earlier quoted context omitted.

Ideally, that would be great. I doubt that would be an option though due to capacity. Would you rather stop/start your instance and risk a capacity error or have your impacted instances rebooted in 48 hours?

> Would you rather stop/start your instance and risk a capacity error or have your impacted instances rebooted in 48 hours? Is there that little slack in Amazon's compute capacity? I would hope not! If there isn't capacity to start my instance back up, I would hope that hitting Stop would generate a dialog to the effect of "Hey there, you won't be able to start this instance back up if you stop it right now."

> Is there that little slack in Amazon's compute capacity?

I'm sure there's plenty of "slack" under most circumstances. However this affects the majority of users so all of them setting up new instances at the same time would likely be impossible.

For the scale of this reboot, they'd have to maintain 30-50% extra capacity which would likely be financially impossible at those rates.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#98

I don't think a lot of people are really understanding how much of a larger issue this would be for us if AWS didn't patch a major security issue before it was made public. The company has treated me very well over the years, from AWS to retail. My stuff arrives on time, if it doesn't I get reimbursed, most of the time with an extra few bucks for my trouble. The AWS platform is more mature and feature rich than anyon…

You get reimbursed?!?! We've been pleading for several months now for a service credit or at least an acknowledgment that they screwed up. Discovered an arcane issue with ARPing to elasticache from within a vpc. Cost us ~$8000 in instances we left running at their request to diagnose, and about the same in man time from our side. Took them 6 weeks to diagnose, too - bloody pathetic.

We spend $30k a month with AWS and they treat us like crap. The tech is ok, but their communications and customer service are mindblowingly atrocious.

If I can give one bit of advice - don't pay for either their support or an account manager. They don't provide either, and their staff turnover is severe, so your account manager will be a noob to their business every few weeks.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#99

A couple of points: Autoscaling is your friend. If you're not leveraging it (multiple availability zones), you're doing it wrong. Even single instances can be launched in autoscaling groups with a desired capacity of 1 to ensure that if it falls over, a new one is spun up. Point 2: AWS is likely trying to rotate capacity for updates, which means they need to evict instances. That are running on doms that they need to…

Yes, isn't it wonderful that you have to completely remake your autoscaling groups every time you add/remove/modify an ELB. Thank Cthulhu (or opscode...) for chef.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#100

Does anyone think it might be related to the recent bash bug? It even affects dhcp clients which is a case with aws. https://securityblog.redhat.com/2014/09/24/bash-specially-cr...

> Does anyone think it might be related to the recent bash bug

That would be...pathetic, at best. There's a Xen bug that's being embargoed 'till 2014-10-01. I think that's more likely.

Post reply on HN