Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hardware Failure on EC2

forums.aws.amazon.com

Post reply on HN
Print view View original

11–20 of 21 posts

Re: Hardware Failure on EC2

#11

Post by ncsurfus »

bdcravens wrote: post #8
doanerock wrote: post #7

When there is a hardware failure, do you have to rebuild or does the original vm come back in minutes/hours/days?

I believe the instance goes away. If you've persisted data to EBS, you can attach the volume to a different instance. Anything outside of EBS is lost. However, if you created an AMI (AWS Machine Image), you can launch a new instance from that. (but it's a snapshot, not a live image, so it'll be only as up to date as it was when saved)

The instance isn’t terminated (gone). It’s either shutdown or rebooted depending on the type of hardware failure.

Re: Hardware Failure on EC2

#12

Post by ihumanable »

Having built out big infra on AWS and now on GCP, GCP's live migrations really change the way I think about the cloud.

GCP has hardware failures, and if there's a bad enough one it impacts your instance, but you can have instances that live forever and they just move it from physical machine to physical machine and it mostly works.

I was very skeptical about this, having spent 5-6 years on AWS and got used to the idea that AWS might just blow your instance away (normally with notice, sometimes not though if the hardware hosting it had some catastrophic failure). GCP just says, "Don't worry about it, things will get moved" and the servers in question have long lived connections, they aren't just simple stateless HTTP API servers, and it works 99.9% of the time (that last 0.1% of the time is a real pain to debug).

We still treat our machines as cattle instead of pets, but not having to constantly deal with cattle roaming off the reservation is nice.

Re: Hardware Failure on EC2

#13

Post by bdcravens »

ncsurfus wrote: post #11
bdcravens wrote: post #8

Earlier quoted context omitted.

I believe the instance goes away. If you've persisted data to EBS, you can attach the volume to a different instance. Anything outside of EBS is lost. However, if you created an AMI (AWS Machine Image), you can launch a new instance from that. (but it's a snapshot, not a live image, so it'll be only as up to date as it was when saved)

The instance isn’t terminated (gone). It’s either shutdown or rebooted depending on the type of hardware failure.

It doesn't get transferred to a different physical instance, right?

Re: Hardware Failure on EC2

#14

Post by NikolaeVarius »

bdcravens wrote: post #13
ncsurfus wrote: post #11

Earlier quoted context omitted.

The instance isn’t terminated (gone). It’s either shutdown or rebooted depending on the type of hardware failure.

It doesn't get transferred to a different physical instance, right?

Why would anyone even care

Re: Hardware Failure on EC2

#15

Post by bdcravens »

NikolaeVarius wrote: post #14
bdcravens wrote: post #13

Earlier quoted context omitted.

It doesn't get transferred to a different physical instance, right?

Why would anyone even care

I think many would make assumptions about the state of the instance and their application, which are likely to be wrong.

Re: Hardware Failure on EC2

#16

Post by christrotman »

bdcravens wrote: post #13
ncsurfus wrote: post #11

Earlier quoted context omitted.

The instance isn’t terminated (gone). It’s either shutdown or rebooted depending on the type of hardware failure.

It doesn't get transferred to a different physical instance, right?

I have never seen it migrated to a different instance type, but it would be running on a different physical server.

Re: Hardware Failure on EC2

#17

Post by Salgat »

Is it even possible for a VM in realtime to failover to another server in the case of a hardware failure? I'm talking where the failure is unpredictable and the VM can't be snapshotted before transfer.

Re: Hardware Failure on EC2

#18

Post by lillecarl »

Salgat wrote: post #17

Is it even possible for a VM in realtime to failover to another server in the case of a hardware failure? I'm talking where the failure is unpredictable and the VM can't be snapshotted before transfer.

It's possible but VERY expensive and severely limited.[1] What is possible and usually acceptable though is shared storage. So another host will spin up the machine for you and you'll have encountered an "unexpected reboot". More often than not monitoring will tell you something is wonky with the server in advance, so you can live migrate to another host and lose nothing "nothing", fix or retire the machine and problem solved.

1: https://www.vmware.com/products/vsphere/fault-tolerance.html

Re: Hardware Failure on EC2

#19

Post by brianwawok »

ihumanable wrote: post #12

Having built out big infra on AWS and now on GCP, GCP's live migrations really change the way I think about the cloud. GCP has hardware failures, and if there's a bad enough one it impacts your instance, but you can have instances that live forever and they just move it from physical machine to physical machine and it mostly works. I was very skeptical about this, having spent 5-6 years on AWS and got used to the ide…

This is neat and something I really like about GCP. Surprised AWS hasn’t been able to replicate this yet.

Yes your code should handle an instance dying. But if 98% of the time you can live migrate and not... hey why not.

Re: Hardware Failure on EC2

#20

Post by ncsurfus »

bdcravens wrote: post #15
NikolaeVarius wrote: post #14

Earlier quoted context omitted.

Why would anyone even care

I think many would make assumptions about the state of the instance and their application, which are likely to be wrong.

Some virtualization platforms support live migrations (hyper-v and ESXi), but this isn’t something AWS supports (afaik). Anytime your instance is stopped/started or rebooted then you may end up on a different physical host.
Post reply on HN