Live data from Hacker News

AWS issues unavoidable reboot schedules with short notice on many EC2 instances

forums.aws.amazon.com

121–130 of 160 posts

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#121

This is one of the reasons why we fished out for 3x full racks at different DCs and bought our own kit. It's always our schedule.

Yes, but Amazon's schedule involves them having advance, non-public knowledge of major security flaws. In some ways, you've got a false sense of security.

Well the bash vulnerability and then immediate disclose of another vulnerability makes that statement pretty moot.

We also have deeply layered non-hetereogenous kit so you have to punch hard to get through it all to something useful.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#122
post #59

This feels like a critical security or stability update.

XSA-108 perhaps? http://xenbits.xen.org/xsa/ AWS is certainly large enough to have access to an embargoed update.

The list of people receiving pre-disclosure access is public:

http://www.xenproject.org/security-policy.html

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#123

I don't think a lot of people are really understanding how much of a larger issue this would be for us if AWS didn't patch a major security issue before it was made public. The company has treated me very well over the years, from AWS to retail. My stuff arrives on time, if it doesn't I get reimbursed, most of the time with an extra few bucks for my trouble. The AWS platform is more mature and feature rich than anyon…

You get reimbursed?!?! We've been pleading for several months now for a service credit or at least an acknowledgment that they screwed up. Discovered an arcane issue with ARPing to elasticache from within a vpc. Cost us ~$8000 in instances we left running at their request to diagnose, and about the same in man time from our side. Took them 6 weeks to diagnose, too - bloody pathetic. We spend $30k a month with AWS and…

Never said I got reimbursed for AWS. As far as I know the SLA is on their infrastructure, was your problem tied to a specific OS?

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#124
post #112

I have an instance listed as scheduled for reboot in 2 days but haven't received an email. Without this post I'd have been caught out on Saturday wondering why I'm getting a ton of message queue connection errors. Downtime I can live with, but unreported planned downtime? Not impressed.

Check your spam folder. A few of their mail servers occasionally have blacklisted IPs.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#125

Earlier quoted context omitted.

You get reimbursed?!?! We've been pleading for several months now for a service credit or at least an acknowledgment that they screwed up. Discovered an arcane issue with ARPing to elasticache from within a vpc. Cost us ~$8000 in instances we left running at their request to diagnose, and about the same in man time from our side. Took them 6 weeks to diagnose, too - bloody pathetic. We spend $30k a month with AWS and…

I apologize for your poor experience with Amazon, and have no doubt that many mistakes were made with your account. But you should recognize that your experience is the exception not the rule. The vast majority of customers have nothing but good things to say, and Amazon works tirelessly to maintain that reputation.

Why are you apologizing unless you work for Amazon?

If you do work for Amazon, I would suggest this approach is not very good customer relations, it's only going to make people madder.

If you don't work for Amazon, and are some kind of Amazon fanboy trying to do free volunteer customer relations for Amazon... you're not helping.

("Oh yeah, we totally fucked up your account, but I insist you recognize that we're awesome anyway, even though I'm not doing anything to make good on our mistakes. Why do you have to recognize that? Cause most people have 'nothing but good things to say' about us, and besides we work really hard, and you can know both of those things are true because I say so.")

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#126
post #89

Earlier quoted context omitted.

Why would you be restoring data? The other instances in your high-availability datastore should have sufficient redundancy to keep you alive until a replacement can be spun up and brought back up to speed. If you aren't using a high-availability datastore, I would suggest that you have not sufficiently sussed out how AWS works and probably shouldn't be using it until you do.

I'd be interested to know more about this as I've been curious for a while about how people do this stuff. No matter how many instances you have, surely you'll still be hosed if they all go down at the same time? Or if there's rolling downtime taking out instances faster than you can bring the restarted instances up to speed? So if you're replicated across three availability zones you're not truly prepared for any in…

>So if you're replicated across three availability zones you're not truly prepared for any instance to go down at any time - you're only prepared for two thirds of your instances to go down at a time?

Correct.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#127
post #79
post #78

I guess if you already put your machines in different availability zones, they will not reboot in the same maintenance window. Actually AWS scheduled maintenance is the best you can get in the market already, because most of the time they allow you to reboot yourself in order to land an updated host (This time is different might be due to some critical security issues). Other providers like Azure / Google, you have n…

Google Compute Engine offers live migration around maintenance events: https://cloud.google.com/compute/docs/instances#onhostmainte... (note: I work on GCE, more or less)

VMWare has this feature too, they call it vMotion. I haven't found anyone who could explain the resilience of vMotion in production use cases though.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#128

I don't think a lot of people are really understanding how much of a larger issue this would be for us if AWS didn't patch a major security issue before it was made public. The company has treated me very well over the years, from AWS to retail. My stuff arrives on time, if it doesn't I get reimbursed, most of the time with an extra few bucks for my trouble. The AWS platform is more mature and feature rich than anyon…

You get reimbursed?!?! We've been pleading for several months now for a service credit or at least an acknowledgment that they screwed up. Discovered an arcane issue with ARPing to elasticache from within a vpc. Cost us ~$8000 in instances we left running at their request to diagnose, and about the same in man time from our side. Took them 6 weeks to diagnose, too - bloody pathetic. We spend $30k a month with AWS and…

I've been reimbursed after an issue that led to an unknown malicious attacker spinning up many maximally-sized EC2 instances for a day. We just had to ask support and comply with cleaning up our instances (which we had done anyway...).

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#129
post #95

Earlier quoted context omitted.

May be this: http://xenbits.xen.org/xsa/ XSA-108 2014-10-01 12:00 (Prereleased, but embargoed)

Cool speculation! Anyone with anything concrete? I still heard it is power issues.

I'd just like to point out that you asked for concrete, then speculated based on something you heard.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#130
post #89

Earlier quoted context omitted.

Why would you be restoring data? The other instances in your high-availability datastore should have sufficient redundancy to keep you alive until a replacement can be spun up and brought back up to speed. If you aren't using a high-availability datastore, I would suggest that you have not sufficiently sussed out how AWS works and probably shouldn't be using it until you do.

I'd be interested to know more about this as I've been curious for a while about how people do this stuff. No matter how many instances you have, surely you'll still be hosed if they all go down at the same time? Or if there's rolling downtime taking out instances faster than you can bring the restarted instances up to speed? So if you're replicated across three availability zones you're not truly prepared for any in…

There are lots of ways to set it up. I should note first that most interesting datastores you'll run in the cloud will end up needing instance stores for performance reasons anyway--you want sequential read perf, you know?--and so this is really just extending it to other nodes that, if you're writing twelve-factor apps, should pop back up without a hitch anyway. (If you're not writing twelve-factor apps...why not?)

Straight failover, with 1:1 mirroring on all nodes? You're massively degraded, unless you've significantly overprovisioned in the happy case, but you have all your stuff. Amazon will (once it unscrews itself from the thrash) start spinning up replacement machines in healthy AZs to replace the dead machines, and if you've done it right they can come up and rejoin the cluster, getting synced back up. (Building that part, auto-scaling groups and replacing dead instances, is probably the hardest part of this whole thing, even with a provisioner like Chef or Puppet.) If you're using a quorum for leader election or you're replicating shard data, being in three AZs actually only protects you from a single AZ interruption. Amazon has lost (or partially lost, I wasn't doing AWS at the time so I'm a little fuzzy) two AZs simultaneously before, and so if you're that sensitive to the failure case you want five AZs (quorum/sharding of 3, so you can lose two). I generally go with three, because in my estimation the likelihood of two AZs going down is low enough that I'm willing to roll the dice, but reasonable people can totally differ there.

If Amazon goes down, yes, you're hosed, and you need to restore from your last S3 backup. But while that is possible, I consider that to be the least likely case (though you should have DR procedures for bringing it back, and you should test them). You have to figure out your acceptable level of risk; for mine, "total failure" is a low enough likelihood, and the rest of the Internet likely to be so completely boned that I should have time to come back online.

Thing is, EBS is not a panacea for any of this; I am pretty sure that a fast rolling bounce would leave most people not named Netflix dead on the ground and not really any better off for recovery than somebody who has to restore a backup.

Post reply on HN