Wouldn't it be simpler to have the smallest possible instance run an NFS server? This would also have an additional bonus of scalability. Edit: or use AWS EFS
Persisting state between AWS EC2 spot instances
21–30 of 80 posts
Re: Persisting state between AWS EC2 spot instances
#22To make this even more streamlined you'd tag the volumes and discover the volumes with `aws ec2 describe-volumes` and filter unattached volumes with the magic tag.
* https://github.com/sevagh/goat (my own) * https://github.com/UKHomeOffice/smilodon
Re: Persisting state between AWS EC2 spot instances
#23Wouldn't it be simpler to have the smallest possible instance run an NFS server? This would also have an additional bonus of scalability. Edit: or use AWS EFS
A positive thing with EFS is that it can be shared across AZ while EBS needs to be snapshotted and then imported to the other AZ.
Re: Persisting state between AWS EC2 spot instances
#24This is leading to rapid progress in clustered/distributed filesystems and it's even built into the Linux kernel now with OrangeFS [1]. There are also commercial companies like Avere [2] who make filers that run on object storage with sophisticated caching to provide a fast networked but durable filesystem.
Kubernetes is also changing the game with container-native storage. This seems to be the most promising model for the future as K8S can take care of orchestrating all the complexities of replicas and stateful containers while storage is just another container-based service using whatever volumes are available to the nodes underneath. Portworx [3] is the great commercial option today with Rook and OpenEBS [4] catching up quickly.
Re: Persisting state between AWS EC2 spot instances
#25Wouldn't it be simpler to have the smallest possible instance run an NFS server? This would also have an additional bonus of scalability. Edit: or use AWS EFS
EFS is far more expensive than EBS. Price it out; you'll see.
Re: Persisting state between AWS EC2 spot instances
#26Earlier quoted context omitted.
Where? I'm using Digital Ocean and it'd be way more expensive for that kind of configuration.
Here’s a list of providers by cost: https://git.io/vps (PS: Don’t use DigitalOcean, they tend to steal your credit if they feel like it. Lost 100 bucks "promotional credit" that way with only a few days notice)
Now I am happy with AWS.
Re: Persisting state between AWS EC2 spot instances
#27ec2-spotter classic uses this, but you can also make a pivoting AMI of your favourite Linux distribution.
One thing to watch out for is how to keep the OS automatic kernel updates working. AMIs are rarely updated and you're going to have a "damn vulnerable linux" if you don't get the updates just after booting a new image.
Re: Persisting state between AWS EC2 spot instances
#281) https://aws.amazon.com/about-aws/whats-new/2017/09/amazon-ec...
Re: Persisting state between AWS EC2 spot instances
#29Earlier quoted context omitted.
Here’s a list of providers by cost: https://git.io/vps (PS: Don’t use DigitalOcean, they tend to steal your credit if they feel like it. Lost 100 bucks "promotional credit" that way with only a few days notice)
Same happened to me. I "lost" my all my credit. It was not promotional, but something I had paid. They informed me on March 31th that I wouldn't be able to use that credit after May 1st. :-( P.S. They had no expiration policy in place when I added the credit. Now I am happy with AWS.
Re: Persisting state between AWS EC2 spot instances
#30Earlier quoted context omitted.
They expired some credits that I haven't used but after asking they just restored them and I could use them. Asking helps.
Asking is not a solution. This is a question of trust. I have to trust that DO will keep my data safe, that, if the US government would be after my data, DO would prevent them from accessing it. I have to trust that DO won’t access my data. How am I supposed to trust my, and my user’s personally identifying data, to a company that just like that revokes credit, without warning, and says "well, if you ask nicely, you…