Earlier quoted context omitted.
May be this: http://xenbits.xen.org/xsa/ XSA-108 2014-10-01 12:00 (Prereleased, but embargoed)
Cool speculation! Anyone with anything concrete? I still heard it is power issues.
AWS issues unavoidable reboot schedules with short notice on many EC2 instances
101–110 of 160 posts
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#102Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#103Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#104I don't think a lot of people are really understanding how much of a larger issue this would be for us if AWS didn't patch a major security issue before it was made public. The company has treated me very well over the years, from AWS to retail. My stuff arrives on time, if it doesn't I get reimbursed, most of the time with an extra few bucks for my trouble. The AWS platform is more mature and feature rich than anyon…
You get reimbursed?!?! We've been pleading for several months now for a service credit or at least an acknowledgment that they screwed up. Discovered an arcane issue with ARPing to elasticache from within a vpc. Cost us ~$8000 in instances we left running at their request to diagnose, and about the same in man time from our side. Took them 6 weeks to diagnose, too - bloody pathetic. We spend $30k a month with AWS and…
But you should recognize that your experience is the exception not the rule. The vast majority of customers have nothing but good things to say, and Amazon works tirelessly to maintain that reputation.
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#105Earlier quoted context omitted.
You use instance stores for persistent data. That instance disappears. Where are you restoring that data from? Either your backups are stale, or you were replicating the data or its underlying filesystem, which means you're still reliant on the network.
Why would you be restoring data? The other instances in your high-availability datastore should have sufficient redundancy to keep you alive until a replacement can be spun up and brought back up to speed. If you aren't using a high-availability datastore, I would suggest that you have not sufficiently sussed out how AWS works and probably shouldn't be using it until you do.
No matter how many instances you have, surely you'll still be hosed if they all go down at the same time? Or if there's rolling downtime taking out instances faster than you can bring the restarted instances up to speed?
So if you're replicated across three availability zones you're not truly prepared for any instance to go down at any time - you're only prepared for two thirds of your instances to go down at a time?
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#106Earlier quoted context omitted.
You get reimbursed?!?! We've been pleading for several months now for a service credit or at least an acknowledgment that they screwed up. Discovered an arcane issue with ARPing to elasticache from within a vpc. Cost us ~$8000 in instances we left running at their request to diagnose, and about the same in man time from our side. Took them 6 weeks to diagnose, too - bloody pathetic. We spend $30k a month with AWS and…
I apologize for your poor experience with Amazon, and have no doubt that many mistakes were made with your account. But you should recognize that your experience is the exception not the rule. The vast majority of customers have nothing but good things to say, and Amazon works tirelessly to maintain that reputation.
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#107A couple of points: Autoscaling is your friend. If you're not leveraging it (multiple availability zones), you're doing it wrong. Even single instances can be launched in autoscaling groups with a desired capacity of 1 to ensure that if it falls over, a new one is spun up. Point 2: AWS is likely trying to rotate capacity for updates, which means they need to evict instances. That are running on doms that they need to…
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#108While people may be painting Amazon in a bad light here, the business-level risk of wholly committing to a single infrastructure provider (cloud or otherwise, across multiple 'availability zones' or data centers or countries or continents, or otherwise) is real. There is a clear need for many service authors to work with disparate infrastructure in a cloud provider and platform abstracted manner, and arguably no soli…
I believe Rackspace are one of those involved in its development.
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#109I don't think a lot of people are really understanding how much of a larger issue this would be for us if AWS didn't patch a major security issue before it was made public. The company has treated me very well over the years, from AWS to retail. My stuff arrives on time, if it doesn't I get reimbursed, most of the time with an extra few bucks for my trouble. The AWS platform is more mature and feature rich than anyon…
You get reimbursed?!?! We've been pleading for several months now for a service credit or at least an acknowledgment that they screwed up. Discovered an arcane issue with ARPing to elasticache from within a vpc. Cost us ~$8000 in instances we left running at their request to diagnose, and about the same in man time from our side. Took them 6 weeks to diagnose, too - bloody pathetic. We spend $30k a month with AWS and…
Maybe my experience is the exception, but my account manager has always been extremely helpful at resolving whatever issues we had.
Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances
#110Earlier quoted context omitted.
Google Compute Engine offers live migration around maintenance events: https://cloud.google.com/compute/docs/instances#onhostmainte... (note: I work on GCE, more or less)
Yes you are right, but GCE does not offer the flexibility that allow user to reboot themselves in their own convenient time. Note: I am not saying the 'live migration' in GCE does not work, I am just saying I've more confident (well...peace of mind, IMO) to shutdown the my own database manually (which is automated and tested), ensure all data are flushed to disk, clients are disconnected gracefully and slave has prom…
There's a story from a partner who was testing it, who at the end of the day said "when are you going to live migrate us?", only to be told that Google had moved them six times that day and they hadn't even noticed.