Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS issues unavoidable reboot schedules with short notice on many EC2 instances

forums.aws.amazon.com

Post reply on HN
Print view View original

131–140 of 160 posts

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#131

Post by seanp2k2 »

toomuchtodo wrote: post #36
rands3311 wrote: post #35

Earlier quoted context omitted.

Ideally, that would be great. I doubt that would be an option though due to capacity. Would you rather stop/start your instance and risk a capacity error or have your impacted instances rebooted in 48 hours?

> Would you rather stop/start your instance and risk a capacity error or have your impacted instances rebooted in 48 hours? Is there that little slack in Amazon's compute capacity? I would hope not! If there isn't capacity to start my instance back up, I would hope that hitting Stop would generate a dialog to the effect of "Hey there, you won't be able to start this instance back up if you stop it right now."

Any slack capacity is waste, and Amazon sure hates that. Consider where the supply of the spot market comes from.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#132

Post by count »

coolbeans01 wrote: post #94
spullara wrote: post #60

Earlier quoted context omitted.

They are not rotating capacity for updates. They are patching a Xen security issue that will be announced on Oct 1. That is why they are rebooting machines and not forcing moves off of those machines. Otherwise, I agree with the advice.

Could you please confirm or provide evidence for such speculation? I don't see anything about this on google. I heard it is because they are having power issues within their datacenter.

within their datacenter? Wtf? Do you have any idea how big AWS is?

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#133

Post by angrybits »

allegory wrote: post #103

This is one of the reasons why we fished out for 3x full racks at different DCs and bought our own kit. It's always our schedule.

Everything else is yours too, like getting an ROI out of the capital your spending. It's nice you have that kind of money laying around to spend on iron instead of advertising (as an example), but not everyone does.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#134

Post by james-at-aws »

madaxe_again wrote: post #98
monjinan wrote: post #66

I don't think a lot of people are really understanding how much of a larger issue this would be for us if AWS didn't patch a major security issue before it was made public. The company has treated me very well over the years, from AWS to retail. My stuff arrives on time, if it doesn't I get reimbursed, most of the time with an extra few bucks for my trouble. The AWS platform is more mature and feature rich than anyon…

You get reimbursed?!?! We've been pleading for several months now for a service credit or at least an acknowledgment that they screwed up. Discovered an arcane issue with ARPing to elasticache from within a vpc. Cost us ~$8000 in instances we left running at their request to diagnose, and about the same in man time from our side. Took them 6 weeks to diagnose, too - bloody pathetic. We spend $30k a month with AWS and…

Hi madaxe_again,

I work at AWS (npinguy does not to the best of my knowledge). If you've spent $8,000 on maintaining instances you wouldn't have running otherwise to reproduce a bug on our behest, we don't want you to have to pay for that. I'd like to look into getting a refund for you. Is there an email address I can reach you at to get more details on the situation?

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#135

Post by tehmasp »

mdellabitta wrote: post #117
coolbeans01 wrote: post #94

Earlier quoted context omitted.

Could you please confirm or provide evidence for such speculation? I don't see anything about this on google. I heard it is because they are having power issues within their datacenter.

Restarting an instance (not stop-start) doesn't change the hardware you're on, so I don't think this has a physical explanation.

all of our instances scheduled for maint. are indeed system-reboot event types; it is indeed a stop/start situation.

stop/start can possibly put you on another physical - but it all depends on how aws has setup the hypervisors and their instance schedulers.

this, the maint from aws, appears to be security related - but that doesn't mean that aws is not getting folks off of old hardware if they have that desire.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#136

Post by eropple »

mdellabitta wrote: post #118
eropple wrote: post #91

Earlier quoted context omitted.

I'd pour one out for the people with the gigantic I2's that they'll have to reprovision, but I'm pretty sure they can afford their own, they don't need me to pour one out for them.

Are you kidding? I have to run i2s. I can't afford anything else! :)

Well, not once you've bought the I2's, no...

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#137

Post by mdellabitta »

tehmasp wrote: post #135
mdellabitta wrote: post #117

Earlier quoted context omitted.

Restarting an instance (not stop-start) doesn't change the hardware you're on, so I don't think this has a physical explanation.

all of our instances scheduled for maint. are indeed system-reboot event types; it is indeed a stop/start situation. stop/start can possibly put you on another physical - but it all depends on how aws has setup the hypervisors and their instance schedulers. this, the maint from aws, appears to be security related - but that doesn't mean that aws is not getting folks off of old hardware if they have that desire.

I think it's more complicated than that. Because Amazon is claiming that if you let them handle it, you keep your instance data. That's not a stop-start (at least what we ordinary users can do).

It's more like a system restart with a little downtime managed by them.

You can try a stop-start yourself, but it's not guaranteed to help. And a restart yourself doesn't do anything.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#138

Post by james-at-aws »

xtrumanx wrote: post #111
npinguy wrote: post #104

Earlier quoted context omitted.

I apologize for your poor experience with Amazon, and have no doubt that many mistakes were made with your account. But you should recognize that your experience is the exception not the rule. The vast majority of customers have nothing but good things to say, and Amazon works tirelessly to maintain that reputation.

Not sure if you actually work for Amazon, but your apology would probably be worth something if you actually decided to take steps to rebuild Amazon's reputation with madaxe_again rather than telling them they're the exception not the rule.

Hey xtrumanx,

I do work at AWS - npinguy does not to the best of my knowledge - and we're looking into a refund for madaxe_again. I'm hoping for a reply from him so we can get more details as to which company this is for, as we'd need that to get the refund process going.

At AWS, we work to keep prices as low as possible and we definitely don't like seeing customers spend money where they wouldn't normally - for example, in reproducing a bug at our request. We're totally happy to look into getting some a refund, and if there's any frustrations with AWS that you or anyone else has, please don't hesitate to let us know.

The current security issue that's causing the need for rebooting EC2 instances isn't something anyone wants, and we do not take steps like this lightly. We work to make sure that the EC2 platform is stable and secure, and in the event of a bug that causes security issues that have the potential to affect our customer's infrastructure that runs on AWS, we want to make sure that the impact is as minimal as possible. In this case, we did not have any other options unfortunately, and we are working with our customers to try and keep the impact as minimal as possible. If you're an AWS customer and you're having issues caused by this (or anything else), please contact us so we can assist you.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#139

Post by eropple »

kennu wrote: post #107
gpjonesii wrote: post #58

A couple of points: Autoscaling is your friend. If you're not leveraging it (multiple availability zones), you're doing it wrong. Even single instances can be launched in autoscaling groups with a desired capacity of 1 to ensure that if it falls over, a new one is spun up. Point 2: AWS is likely trying to rotate capacity for updates, which means they need to evict instances. That are running on doms that they need to…

Autoscaling is your friend, but you can also use "auto-healing" if your stack is built on Amazon's AWS OpsWorks and you just want to keep a single instance alive. It will automatically spawn a replacement instance and reattach and mount any EBS volumes.

My understanding is that that doesn't work correctly in the case of AZ failure; the EBS data isn't duped to another AZ and so your instance will fail to come up. So it's not really a solution.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#140

Post by morpher »

tszming wrote: post #87
jsolson wrote: post #79

Earlier quoted context omitted.

Google Compute Engine offers live migration around maintenance events: https://cloud.google.com/compute/docs/instances#onhostmainte... (note: I work on GCE, more or less)

Yes you are right, but GCE does not offer the flexibility that allow user to reboot themselves in their own convenient time. Note: I am not saying the 'live migration' in GCE does not work, I am just saying I've more confident (well...peace of mind, IMO) to shutdown the my own database manually (which is automated and tested), ensure all data are flushed to disk, clients are disconnected gracefully and slave has prom…

You can configure your instances to automatically shutdown and restart instead of migrating: https://cloud.google.com/compute/docs/instances#onhostmainte...

Alternatively, you have a short period (60s) before the migration occurs during which you can lame a service: https://cloud.google.com/compute/docs/metadata#maintenanceev...

However, I'll grant that both of these lack the "in your own convenient time" portion of your desire.

Post reply on HN