AWS issues unavoidable reboot schedules with short notice on many EC2 instances

Post by npinguy » Thu, Sep 25, 2014, 5:53 PM UTC

james-at-aws wrote: post #134

madaxe_again wrote: post #98

Earlier quoted context omitted.

You get reimbursed?!?! We've been pleading for several months now for a service credit or at least an acknowledgment that they screwed up. Discovered an arcane issue with ARPing to elasticache from within a vpc. Cost us ~$8000 in instances we left running at their request to diagnose, and about the same in man time from our side. Took them 6 weeks to diagnose, too - bloody pathetic. We spend $30k a month with AWS and…

Hi madaxe_again, I work at AWS (npinguy does not to the best of my knowledge). If you've spent $8,000 on maintaining instances you wouldn't have running otherwise to reproduce a bug on our behest, we don't want you to have to pay for that. I'd like to look into getting a refund for you. Is there an email address I can reach you at to get more details on the situation?

Post by toomuchtodo » Thu, Sep 25, 2014, 6:05 PM UTC

rodgerd wrote: post #142

toomuchtodo wrote: post #36

Earlier quoted context omitted.

> Would you rather stop/start your instance and risk a capacity error or have your impacted instances rebooted in 48 hours? Is there that little slack in Amazon's compute capacity? I would hope not! If there isn't capacity to start my instance back up, I would hope that hitting Stop would generate a dialog to the effect of "Hey there, you won't be able to start this instance back up if you stop it right now."

> Is there that little slack in Amazon's compute capacity? Amazon's ability to deliver sharp prices does not come from leaving unused tin lying around in datacentres.

Post by pas256 » Thu, Sep 25, 2014, 7:23 PM UTC

toomuchtodo wrote: post #32

rurounijones wrote: post #30

Pretty crappy that you cannot immediately check if an instance has restarted on a patched host or not. I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option. Having to wait possibly hours for an email to see if your vm migrated to patched host or not is a terrible solution.

> I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option. I would expect Amazon to mark unpatched hosts as bad and not permit new instances to be deployed to them, similar to queue draining. Not cool Amazon.

Post by toomuchtodo » Thu, Sep 25, 2014, 7:29 PM UTC

pas256 wrote: post #145

toomuchtodo wrote: post #32

Earlier quoted context omitted.

> I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option. I would expect Amazon to mark unpatched hosts as bad and not permit new instances to be deployed to them, similar to queue draining. Not cool Amazon.

Stop relying on long running instances. Design your infrastructure for failure. This is not Amazon's responsibility.

Post by jlouvel » Thu, Sep 25, 2014, 8:31 PM UTC

nicobn wrote: post #109

madaxe_again wrote: post #98

Earlier quoted context omitted.

You get reimbursed?!?! We've been pleading for several months now for a service credit or at least an acknowledgment that they screwed up. Discovered an arcane issue with ARPing to elasticache from within a vpc. Cost us ~$8000 in instances we left running at their request to diagnose, and about the same in man time from our side. Took them 6 weeks to diagnose, too - bloody pathetic. We spend $30k a month with AWS and…

When a script mistakingly provisioned a ridiculous amount of read/write capacity on hundreds of DynamoDB tables, we were presented with a bill in the thousands of dollars. We got reimbursed after talking with our account manager for 15 minutes. Maybe my experience is the exception, but my account manager has always been extremely helpful at resolving whatever issues we had.