Persisting state between AWS EC2 spot instances • Hacker News Board

Re: Persisting state between AWS EC2 spot instances

#41

Post by Johnny555 » Sun, Oct 08, 2017, 5:34 PM UTC

otterley wrote: post #15

amq wrote: post #9

Wouldn't it be simpler to have the smallest possible instance run an NFS server? This would also have an additional bonus of scalability. Edit: or use AWS EFS

EFS is far more expensive than EBS. Price it out; you'll see.

It is 3X more expensive ($0.30/gb vs $0.10/gb for us-east), but it's replicated across AZ's (so is more durable than EBS which is only replicated within an AZ), and you only pay for what you use, you don't need to overprovision the EBS volume to account for peak dataset size.

And since it's shared, you don't need to replicate data across multiple nodes... so if 10 compute nodes needs access to the data set, they can all just read it from the same EFS filesystem, no need to download it 10 times to each compute node.

So EFS can still be very cost effective compared to EBS.

Re: Persisting state between AWS EC2 spot instances

#42

Post by sciurus » Sun, Oct 08, 2017, 5:38 PM UTC

The author goes to great lengths to come up with a way for the software that was running on a terminated spot instance to be relaunched using the same root filesystem on a new spot instance, but they never explain why they need to do exactly this. Maybe they already ran everything in Docker containers on CoreOS, so their solution isn't a big shift, but I strongly suspect they could find a simpler way to save and restore state if they got over this obsession with preserving the root filesystem their software sees.

Re: Persisting state between AWS EC2 spot instances

#43

Post by jen20 » Sun, Oct 08, 2017, 5:47 PM UTC

solatic wrote: post #39

OP is offering some very dangerous advice. Twenty years ago, software was hosted on fragile single-node servers with fragile, physical hard disks. Programmers would read and write files directly from and to the disk, and learn the hard way that this left their systems susceptible to corruption in case things crashed in the middle of a write. So behold! People began to use relational databases which offered ACID guara…

This pattern is a lot safer if you use ZFS. Spot instances don't just disappear though, you get notification and have a chance to perform shutdown actions, except in the case of hardware failure - which is the same with non-spot instances.

Re: Persisting state between AWS EC2 spot instances

#44

Post by olegkikin » Sun, Oct 08, 2017, 5:57 PM UTC

blibble wrote: post #7

olegkikin wrote: post #4

If you don't care about reliability, why not just get a cheap and powerful VPS? Paying $90/month for that machine is madness. I pay $6/month for 6GB RAM, 4 cores, 50GB disk.

where are you getting that for $6?

Hostus.us

The deal was found on LowEndBox, not sure if it's still available, but there are many other ones.

Re: Persisting state between AWS EC2 spot instances

#45

Post by otterley » Sun, Oct 08, 2017, 6:04 PM UTC

solatic wrote: post #39

OP is offering some very dangerous advice. Twenty years ago, software was hosted on fragile single-node servers with fragile, physical hard disks. Programmers would read and write files directly from and to the disk, and learn the hard way that this left their systems susceptible to corruption in case things crashed in the middle of a write. So behold! People began to use relational databases which offered ACID guara…

Spot instances are shut down cleanly via the usual stop semantics (which includes all the shutdown handlers provided your OS supports them). Assuming your database software supports clean shutdowns via SIGTERM, everything should be fine.

Re: Persisting state between AWS EC2 spot instances

#46

Post by solatic » Sun, Oct 08, 2017, 6:06 PM UTC

jen20 wrote: post #43

solatic wrote: post #39

OP is offering some very dangerous advice. Twenty years ago, software was hosted on fragile single-node servers with fragile, physical hard disks. Programmers would read and write files directly from and to the disk, and learn the hard way that this left their systems susceptible to corruption in case things crashed in the middle of a write. So behold! People began to use relational databases which offered ACID guara…

This pattern is a lot safer if you use ZFS. Spot instances don't just disappear though, you get notification and have a chance to perform shutdown actions, except in the case of hardware failure - which is the same with non-spot instances.

- EBS, being block storage, doesn't recognize the filesystem format on top of it, and therefore doesn't recognize if you formatted the block storage as ZFS and therefore will not use ZFS snapshots when using Amazon's native EBS snapshotting. If you wish to use ZFS snapshots, you have to build that on top of what Amazon gives you, along with all the other aspects of ZFS storage, i.e. building a ZFS storage pool from separate EBS volumes. I mean, it would be nice if Amazon had a hosted ZFS solution, but so far, doesn't seem like it.

- Yes, you get a notification, but it's a proprietary notification scheme that your application must be designed to poll for. Why can't Amazon use standard signals like SIGPWR to indicate imminent shutdown?

- Just because it isn't smart for non-spot instances doesn't suddenly make it smart for spot instances ;)

Re: Persisting state between AWS EC2 spot instances

#47

Post by otterley » Sun, Oct 08, 2017, 6:10 PM UTC

Johnny555 wrote: post #41

otterley wrote: post #15

Earlier quoted context omitted.

EFS is far more expensive than EBS. Price it out; you'll see.

It is 3X more expensive ($0.30/gb vs $0.10/gb for us-east), but it's replicated across AZ's (so is more durable than EBS which is only replicated within an AZ), and you only pay for what you use, you don't need to overprovision the EBS volume to account for peak dataset size. And since it's shared, you don't need to replicate data across multiple nodes... so if 10 compute nodes needs access to the data set, they can…

Are you counting the impact on the ENI's available bandwidth and additional instance costs needed for more network throughput? As I understand it, EFS requests are issued through the front end interface, while EBS requests go through the storage backplane interface.

Also, NFS has different behavior with respect to buffer caching that needs to be taken into account. It often does not cache as effectively as block storage does.

Re: Persisting state between AWS EC2 spot instances

#48

Post by solatic » Sun, Oct 08, 2017, 6:13 PM UTC

otterley wrote: post #45

solatic wrote: post #39

OP is offering some very dangerous advice. Twenty years ago, software was hosted on fragile single-node servers with fragile, physical hard disks. Programmers would read and write files directly from and to the disk, and learn the hard way that this left their systems susceptible to corruption in case things crashed in the middle of a write. So behold! People began to use relational databases which offered ACID guara…

Spot instances are shut down cleanly via the usual stop semantics (which includes all the shutdown handlers provided your OS supports them). Assuming your database software supports clean shutdowns via SIGTERM, everything should be fine.

> Assuming your database software

You're assuming that people are saving their state in databases to begin with. If you're saving state to a database in production, typically you're communicating with that database over a network connection, and not running the database on the same machine as your application. Containerizing databases is a whole separate issue.

OP's specific example is saving /var/opt/gitlab to an EBS volume and expecting to be able to move it from one spot instance to another without corruption. That strikes me as insane.

Re: Persisting state between AWS EC2 spot instances

#49

Post by noja » Sun, Oct 08, 2017, 6:15 PM UTC

solatic wrote: post #48

otterley wrote: post #45

Earlier quoted context omitted.

Spot instances are shut down cleanly via the usual stop semantics (which includes all the shutdown handlers provided your OS supports them). Assuming your database software supports clean shutdowns via SIGTERM, everything should be fine.

> Assuming your database software You're assuming that people are saving their state in databases to begin with. If you're saving state to a database in production, typically you're communicating with that database over a network connection, and not running the database on the same machine as your application. Containerizing databases is a whole separate issue. OP's specific example is saving /var/opt/gitlab to an EB…

Why? The gitlab init script to stop it is being run. It's a clean shutdown.

Re: Persisting state between AWS EC2 spot instances

#50

Post by manigandham » Sun, Oct 08, 2017, 6:24 PM UTC

solatic wrote: post #48

otterley wrote: post #45

Earlier quoted context omitted.

Spot instances are shut down cleanly via the usual stop semantics (which includes all the shutdown handlers provided your OS supports them). Assuming your database software supports clean shutdowns via SIGTERM, everything should be fine.

> Assuming your database software You're assuming that people are saving their state in databases to begin with. If you're saving state to a database in production, typically you're communicating with that database over a network connection, and not running the database on the same machine as your application. Containerizing databases is a whole separate issue. OP's specific example is saving /var/opt/gitlab to an EB…

What is so insane about this? It's no different than plugging in a USB drive, modifying some data on it, then disconnecting. Except in this case, the mount/unmount happens outside of the application's lifecycle so it can initialize and shutdown cleanly without worry.