Live data from Hacker News

Persisting state between AWS EC2 spot instances

peteris.rocks

61–70 of 80 posts

Re: Persisting state between AWS EC2 spot instances

#61
post #58

Earlier quoted context omitted.

Partially completed write of a block, sure. But partially completed write of a file? I can imagine (cough) an application where the application is trying to write some binary blob to disk, doesn't finish before shutdown, and upon reboot, tries to load the binary blob back into memory, fails because the binary blob isn't consistent, doesn't handle the failure well, and refuses to boot. App's fault? Sure. Does the cust…

Then all you're saying over and over is that in your imagination, not using a long running instance is very dangerous because rebooting exposes the fragility of your app. Honestly, it's much safer in that circumstance to have a frequently rebooting instance because it will quickly expose your app's fragility during normal operations instead of that fragility being exposed in a disaster.

> it's much safer in that circumstance to have a frequently rebooting instance

I actually happen to agree with you in principle on this, and it's at the root of my current side project.

But sometimes you just don't have the flexibility to fix or replace the app. Ops engineering, like any other kind of engineering, is about dealing with real-world constraints and making the most of the resources you have. Most apps, on some notion of a fragility spectrum, are far closer to fragile than to antifragile, because fragile is the default, and extensive stress-testing to understand and plan for all failure modes before a production deployment isn't typically feasible. At that point, if you can't fix it, you have to work around it.

Re: Persisting state between AWS EC2 spot instances

#63
It sounds wrong to try to keep the state across two ec2 instances. If you find yourself in that situation, try pushing your state outside the ec2 instance a bit harder. (dynamodb, s3 etc...)

You will get a lot of benefit out of it, but may lose in performance, which is fine in 99% of the cases.

Re: Persisting state between AWS EC2 spot instances

#64
post #61

Earlier quoted context omitted.

Then all you're saying over and over is that in your imagination, not using a long running instance is very dangerous because rebooting exposes the fragility of your app. Honestly, it's much safer in that circumstance to have a frequently rebooting instance because it will quickly expose your app's fragility during normal operations instead of that fragility being exposed in a disaster.

> it's much safer in that circumstance to have a frequently rebooting instance I actually happen to agree with you in principle on this, and it's at the root of my current side project. But sometimes you just don't have the flexibility to fix or replace the app. Ops engineering, like any other kind of engineering, is about dealing with real-world constraints and making the most of the resources you have. Most apps, o…

All you're doing is advocating larger, less frequent failures with people who know less. Robustness isn't just about your software or your ops setup, but also about your people and their knowledge and experience. I cannot see how less frequent, more intense failures with people who know less is preferable, and that anything else is "very dangerous advice"

You will ultimately have many fewer resources available if your strategy is to gloss over failure modes by telling inexperienced engineers to hope they won't happen. It's technical debt and the interest payments are very high.

Re: Persisting state between AWS EC2 spot instances

#65

Spot instances can now "stop" instead of "terminate" when you get priced out, persisting the attached EBS volumes: https://aws.amazon.com/about-aws/whats-new/2017/09/amazon-ec...

This should really be at the top!

EBS is not the root volume.

Re: Persisting state between AWS EC2 spot instances

#66
I don't know why all the comments are saying this is bad idea. For me, one of thing for I use EC2 is deep learning. I just use spot GPU instance, attach overlayroot volume and launch jupyter notebook in it. Other things like google dataflow is not useful to me due to the price and the process of installing packages. I can also think of many other use cases for using some persistence volume for some manual task.

Re: Persisting state between AWS EC2 spot instances

#67

Persistent storage remains a complicated problem. Attaching volumes on the fly with docker volume abstraction works well enough for most cloud workloads, whether on-demand or spot, but it's still easy to run into problems. This is leading to rapid progress in clustered/distributed filesystems and it's even built into the Linux kernel now with OrangeFS [1]. There are also commercial companies like Avere [2] who make f…

Using a clustered/distributed filesystem definitively simplifies persisting the state between EC2 spot instances. It also makes it easier to scale out the work load when you need more instances accessing the same data. To add to your list: there is also ObjectiveFS[1] that integrates well with AWS (uses S3 for storage, works with IAM roles, etc) and EC2 spot instances.

[1]. https://objectivefs.com

Re: Persisting state between AWS EC2 spot instances

#68
So I was pleasantly surprised to discover that for the last several years, spot instances have provided a mechanism that give you 2 minutes notice prior to shutdown:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-inte...

Learn something new everyday. :)

https://aws.amazon.com/blogs/aws/new-ec2-spot-instance-termi...

Re: Persisting state between AWS EC2 spot instances

#69
post #61

Earlier quoted context omitted.

> it's much safer in that circumstance to have a frequently rebooting instance I actually happen to agree with you in principle on this, and it's at the root of my current side project. But sometimes you just don't have the flexibility to fix or replace the app. Ops engineering, like any other kind of engineering, is about dealing with real-world constraints and making the most of the resources you have. Most apps, o…

All you're doing is advocating larger, less frequent failures with people who know less. Robustness isn't just about your software or your ops setup, but also about your people and their knowledge and experience. I cannot see how less frequent, more intense failures with people who know less is preferable, and that anything else is "very dangerous advice" You will ultimately have many fewer resources available if you…

You are both right. But both wrong. If you want better consistency, use either object storage or a database. If you are mutating multiple entities and need consistency, now you need a distributed transaction.

But ALL cloud providers provide warning before an instance is shutdown. There is absolutely no reason, other than a crash for an instance to have a hard shutdown.

Re: Persisting state between AWS EC2 spot instances

#70

Earlier quoted context omitted.

This should really be at the top!

EBS is not the root volume.

from the announcement:

"The EBS root device and attached EBS volumes are saved..."

Some instance types don't support an instance root, but require an EBS root.

Post reply on HN