Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS issues unavoidable reboot schedules with short notice on many EC2 instances

forums.aws.amazon.com

Post reply on HN
Print view View original

81–90 of 160 posts

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#82

Post by fletchowns »

monjinan wrote: post #68
fletchowns wrote: post #61

Earlier quoted context omitted.

Obviously these individuals have never visited the EC2 forums: https://forums.aws.amazon.com/search.jspa?objID=f30&q=unresp... If you are relying on a single instance and depending on Amazon's track record, you're doing it wrong.

Do you have any actual sources of their average instance uptime over the year? I see an SLA of 99.95%(~4hrs/year) on the site but cloudharmony.com/status has AWS at the top. Googling average ec2 uptime has people posting instances running for years with nothing.

I don't doubt that ec2 instances probably have good uptime on average. But instances can mysteriously disappear, that is a fact, and if you are on AWS you should plan for it. They give you all the tools you need for such planning.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#85

Post by fleitz »

monjinan wrote: post #66

I don't think a lot of people are really understanding how much of a larger issue this would be for us if AWS didn't patch a major security issue before it was made public. The company has treated me very well over the years, from AWS to retail. My stuff arrives on time, if it doesn't I get reimbursed, most of the time with an extra few bucks for my trouble. The AWS platform is more mature and feature rich than anyon…

Yup, fundamentally it's not following best practices vs. a few minutes of downtime. It's a far better call than not forcing the update.

Of course all the people who built shoddy cloud presences are going to be screaming blue murder that their 'oh so successful' but 'couldn't be fucked to run in multiple AZs' businesses are going to be down for a few minutes.

It reminds me of doing tech support during the dotBomb, every daytrader would phone about how they were losing thousand per second while their internet was down, I'd ask them to switch to their backup connection, and then explain that they had the cheapest residential connection available that's inappropriate for day trading and that if they are making thousands per second they should have a backup connection in place.

Then I'd upsell them on a business package and have a guy there in 24 hours.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#86

Post by reedlaw »

fletchowns wrote: post #61
reedlaw wrote: post #57

Earlier quoted context omitted.

That's true, but how many services get this right in practice? It's not easy to ensure all instances will perform well in case of failover. A lot of apps are running on a single instance and depend on Amazon's fairly good track record.

Obviously these individuals have never visited the EC2 forums: https://forums.aws.amazon.com/search.jspa?objID=f30&q=unresp... If you are relying on a single instance and depending on Amazon's track record, you're doing it wrong.

Yes, clearly it's wrong, and I've had instances disappear myself. But it happens rarely enough, and some don't want the overhead of running another instance or ensuring an application has zero downtime. There is a large spectrum between having zero downtime and running a single-instance app on EC2.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#87

Post by tszming »

jsolson wrote: post #79
tszming wrote: post #78

I guess if you already put your machines in different availability zones, they will not reboot in the same maintenance window. Actually AWS scheduled maintenance is the best you can get in the market already, because most of the time they allow you to reboot yourself in order to land an updated host (This time is different might be due to some critical security issues). Other providers like Azure / Google, you have n…

Google Compute Engine offers live migration around maintenance events: https://cloud.google.com/compute/docs/instances#onhostmainte... (note: I work on GCE, more or less)

Yes you are right, but GCE does not offer the flexibility that allow user to reboot themselves in their own convenient time.

Note: I am not saying the 'live migration' in GCE does not work, I am just saying I've more confident (well...peace of mind, IMO) to shutdown the my own database manually (which is automated and tested), ensure all data are flushed to disk, clients are disconnected gracefully and slave has promoted to master.. and things like that, rather than some black magic like 'live migration'

Of course, I am not saying 'live migration' is wrong, but it is just my preference.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#88

Post by fjordan »

fjordan wrote: post #83

CVE-2014-6271 was posted on the AWS Security Bulletin earlier today: http://aws.amazon.com/security/security-bulletins/CVE_2014_6... My guess is that they cannot rely on all users to perform these updates.

can someone explain the downvotes?

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#89

Post by eropple »

toomuchtodo wrote: post #77
eropple wrote: post #62

Earlier quoted context omitted.

One thing I'd add: best practices (IMO) dictate HA clusters as you describe, but you get a big boost to survivability by deciding on only using instance stores. Network issues have screwed EBS in the past; EBS is technically neat but very network-sensitive and it's possible to "lose" part of your EBS volume because part of the network goes away (and then your instance faceplants). Instance stores are your friend, and…

You use instance stores for persistent data. That instance disappears. Where are you restoring that data from? Either your backups are stale, or you were replicating the data or its underlying filesystem, which means you're still reliant on the network.

Why would you be restoring data? The other instances in your high-availability datastore should have sufficient redundancy to keep you alive until a replacement can be spun up and brought back up to speed.

If you aren't using a high-availability datastore, I would suggest that you have not sufficiently sussed out how AWS works and probably shouldn't be using it until you do.

Re: AWS issues unavoidable reboot schedules with short notice on many EC2 instances

#90

Post by eropple »

mustafab wrote: post #84

Does anyone think it might be related to the recent bash bug? It even affects dhcp clients which is a case with aws. https://securityblog.redhat.com/2014/09/24/bash-specially-cr...

It is unrelated. This is a Xen patch.
Post reply on HN