AWS issues unavoidable reboot schedules with short notice on many EC2 instances

Post by vacri » Fri, Sep 26, 2014, 12:01 AM UTC

tszming wrote: post #78

I guess if you already put your machines in different availability zones, they will not reboot in the same maintenance window. Actually AWS scheduled maintenance is the best you can get in the market already, because most of the time they allow you to reboot yourself in order to land an updated host (This time is different might be due to some critical security issues). Other providers like Azure / Google, you have n…

Post by theshrike79 » Fri, Sep 26, 2014, 5:48 AM UTC

nicobn wrote: post #109

madaxe_again wrote: post #98

Earlier quoted context omitted.

You get reimbursed?!?! We've been pleading for several months now for a service credit or at least an acknowledgment that they screwed up. Discovered an arcane issue with ARPing to elasticache from within a vpc. Cost us ~$8000 in instances we left running at their request to diagnose, and about the same in man time from our side. Took them 6 weeks to diagnose, too - bloody pathetic. We spend $30k a month with AWS and…

When a script mistakingly provisioned a ridiculous amount of read/write capacity on hundreds of DynamoDB tables, we were presented with a bill in the thousands of dollars. We got reimbursed after talking with our account manager for 15 minutes. Maybe my experience is the exception, but my account manager has always been extremely helpful at resolving whatever issues we had.

Post by madaxe_again » Fri, Sep 26, 2014, 7:38 AM UTC

james-at-aws wrote: post #134

madaxe_again wrote: post #98

Earlier quoted context omitted.

You get reimbursed?!?! We've been pleading for several months now for a service credit or at least an acknowledgment that they screwed up. Discovered an arcane issue with ARPing to elasticache from within a vpc. Cost us ~$8000 in instances we left running at their request to diagnose, and about the same in man time from our side. Took them 6 weeks to diagnose, too - bloody pathetic. We spend $30k a month with AWS and…

Hi madaxe_again, I work at AWS (npinguy does not to the best of my knowledge). If you've spent $8,000 on maintaining instances you wouldn't have running otherwise to reproduce a bug on our behest, we don't want you to have to pay for that. I'd like to look into getting a refund for you. Is there an email address I can reach you at to get more details on the situation?

Post by xtrumanx » Fri, Sep 26, 2014, 5:10 PM UTC

james-at-aws wrote: post #138

xtrumanx wrote: post #111

Earlier quoted context omitted.

Not sure if you actually work for Amazon, but your apology would probably be worth something if you actually decided to take steps to rebuild Amazon's reputation with madaxe_again rather than telling them they're the exception not the rule.

Hey xtrumanx, I do work at AWS - npinguy does not to the best of my knowledge - and we're looking into a refund for madaxe_again. I'm hoping for a reply from him so we can get more details as to which company this is for, as we'd need that to get the refund process going. At AWS, we work to keep prices as low as possible and we definitely don't like seeing customers spend money where they wouldn't normally - for exam…

Post by eropple » Mon, Sep 29, 2014, 5:25 PM UTC

kennu wrote: post #154

eropple wrote: post #139

Earlier quoted context omitted.

My understanding is that that doesn't work correctly in the case of AZ failure; the EBS data isn't duped to another AZ and so your instance will fail to come up. So it's not really a solution.

Quite possible. But it depends on what level of disaster you want to protect yourself from. Single EC2 instance termination is much more common than an entire availability zone going down. I'd say OpsWorks auto-healing is better than just running a standalone EC2 instance, and configuring a full auto-scaling setup with your own custom AMIs and boot scripts is even better, but also much more work.