Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS issues unavoidable reboot schedules with short notice on many EC2 instances

forums.aws.amazon.com

Post reply on HN
Print view View original

111–120 of 160 posts

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#111

Post by xtrumanx »

npinguy wrote: post #104
madaxe_again wrote: post #98

Earlier quoted context omitted.

You get reimbursed?!?! We've been pleading for several months now for a service credit or at least an acknowledgment that they screwed up. Discovered an arcane issue with ARPing to elasticache from within a vpc. Cost us ~$8000 in instances we left running at their request to diagnose, and about the same in man time from our side. Took them 6 weeks to diagnose, too - bloody pathetic. We spend $30k a month with AWS and…

I apologize for your poor experience with Amazon, and have no doubt that many mistakes were made with your account. But you should recognize that your experience is the exception not the rule. The vast majority of customers have nothing but good things to say, and Amazon works tirelessly to maintain that reputation.

Not sure if you actually work for Amazon, but your apology would probably be worth something if you actually decided to take steps to rebuild Amazon's reputation with madaxe_again rather than telling them they're the exception not the rule.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#112

Post by RossM »

I have an instance listed as scheduled for reboot in 2 days but haven't received an email. Without this post I'd have been caught out on Saturday wondering why I'm getting a ton of message queue connection errors.

Downtime I can live with, but unreported planned downtime? Not impressed.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#114

Post by fennecfoxen »

npinguy wrote: post #104
madaxe_again wrote: post #98

Earlier quoted context omitted.

You get reimbursed?!?! We've been pleading for several months now for a service credit or at least an acknowledgment that they screwed up. Discovered an arcane issue with ARPing to elasticache from within a vpc. Cost us ~$8000 in instances we left running at their request to diagnose, and about the same in man time from our side. Took them 6 weeks to diagnose, too - bloody pathetic. We spend $30k a month with AWS and…

I apologize for your poor experience with Amazon, and have no doubt that many mistakes were made with your account. But you should recognize that your experience is the exception not the rule. The vast majority of customers have nothing but good things to say, and Amazon works tirelessly to maintain that reputation.

> But you should recognize that your experience is the exception not the rule.

Don't tell people that. They don't believe you.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#115

Post by _asummers »

mustafab wrote: post #84

Does anyone think it might be related to the recent bash bug? It even affects dhcp clients which is a case with aws. https://securityblog.redhat.com/2014/09/24/bash-specially-cr...

The bash bug does not require a system restart. In fact, it can be updated without shutting down the running bash processes, as the vulnerability only affects new shells.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#117

Post by mdellabitta »

coolbeans01 wrote: post #94
spullara wrote: post #60

Earlier quoted context omitted.

They are not rotating capacity for updates. They are patching a Xen security issue that will be announced on Oct 1. That is why they are rebooting machines and not forcing moves off of those machines. Otherwise, I agree with the advice.

Could you please confirm or provide evidence for such speculation? I don't see anything about this on google. I heard it is because they are having power issues within their datacenter.

Restarting an instance (not stop-start) doesn't change the hardware you're on, so I don't think this has a physical explanation.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#118

Post by mdellabitta »

eropple wrote: post #91
sounds wrote: post #81

These instance types are not affected: T1, T2, M2, R3, and HS1 instance types are not affected. http://www.rightscale.com/blog/rightscale-news/aws-reboot-su...

I'd pour one out for the people with the gigantic I2's that they'll have to reprovision, but I'm pretty sure they can afford their own, they don't need me to pour one out for them.

Are you kidding? I have to run i2s. I can't afford anything else! :)

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#120

Post by ceejayoz »

allegory wrote: post #103

This is one of the reasons why we fished out for 3x full racks at different DCs and bought our own kit. It's always our schedule.

Yes, but Amazon's schedule involves them having advance, non-public knowledge of major security flaws. In some ways, you've got a false sense of security.
Post reply on HN