Hardware Failure on EC2

#1

Post by fctorial » Sat, Apr 17, 2021, 5:24 AM UTC

Hardware Failure on EC2
forums.aws.amazon.com

Re: Hardware Failure on EC2

#2

Post by znpy » Sat, Apr 17, 2021, 12:02 PM UTC

This is not a news really, if anything a chance to express a consideration about "the cloud" in general.

Hardware is fairly reliable nowadays (whether you're hosting your own or using the cloud provider's) but failure still happens.

The bonus that you get by using a cloud provider is that you get API machinery and tooling to automatically handling most failures.

The malus is that depending on your level of paranoia (eg: architecting for multi-region) redundancy will get very expensive very soon.

But in the end... meh.

Despite what "evangelists" or "detractors" will say, there's no free lunch.

Re: Hardware Failure on EC2

#3

Post by redis_mlc » Sat, Apr 17, 2021, 12:32 PM UTC

> We just recently received notification of a hardware failure affecting some of our EC2 instances. I was under the impression that by using EC2, hardware failures were handled behind the scenes and would not affect a running instance. Obviously, I was wrong and I need to go back and review our assumptions.

So, to confirm, there are individual EC2 hardware failure scenarios that we have to plan for. Is this a correct statement?

Newb.

Re: Hardware Failure on EC2

#4

Post by bdcravens » Sat, Apr 17, 2021, 12:39 PM UTC

needs (2012)

Re: Hardware Failure on EC2

#5

Post by phamilton » Sat, Apr 17, 2021, 2:43 PM UTC

We have switched to using spot instances for our production traffic. If you diversify across instance types and AZs, there's always plenty of capacity.

The result is, as one engineer on our team says, you basically run chaos monkey but amazon pays you to do so.

Re: Hardware Failure on EC2

#6

Post by cellover » Sat, Apr 17, 2021, 3:17 PM UTC

Totaly off topic but this made my day: "Elastic lad balancing"

Re: Hardware Failure on EC2

#7

Post by doanerock » Sat, Apr 17, 2021, 5:13 PM UTC

When there is a hardware failure, do you have to rebuild or does the original vm come back in minutes/hours/days?

Re: Hardware Failure on EC2

#8

Post by bdcravens » Sat, Apr 17, 2021, 5:31 PM UTC

doanerock wrote: post #7

When there is a hardware failure, do you have to rebuild or does the original vm come back in minutes/hours/days?

I believe the instance goes away. If you've persisted data to EBS, you can attach the volume to a different instance. Anything outside of EBS is lost. However, if you created an AMI (AWS Machine Image), you can launch a new instance from that. (but it's a snapshot, not a live image, so it'll be only as up to date as it was when saved)

Re: Hardware Failure on EC2

#9

Post by bdcravens » Sat, Apr 17, 2021, 5:34 PM UTC

phamilton wrote: post #5

We have switched to using spot instances for our production traffic. If you diversify across instance types and AZs, there's always plenty of capacity. The result is, as one engineer on our team says, you basically run chaos monkey but amazon pays you to do so.

I've run a ton of background processing through spot (generally a couple of hundred instances at a time) for many years, and I think I've only lost all capacity once (and that was before fleet)

Re: Hardware Failure on EC2

#10

Post by mhh__ » Sat, Apr 17, 2021, 7:09 PM UTC

I wish IBM had a few competitors because I think the industry would be interesting if the mainframe approach to reliability was also available, but I think you'd be an idiot to even dip your finger in their kool-aid as of now.