Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Persisting state between AWS EC2 spot instances

peteris.rocks

Post reply on HN
Print view View original

71–80 of 80 posts

Re: Persisting state between AWS EC2 spot instances

#71

Post by fulafel »

derefr wrote: post #55
colechristensen wrote: post #54

Earlier quoted context omitted.

Your response is fairly ridiculous. A spot instance interruption isn't a system crash, it's a shutdown signal. Storing your important spot instance data on EBS is recommended by AWS. If your application can't handle a normal system shutdown without losing data, your application is at fault, not your system setup. >exactly how state is being saved on that EBS volume Files are written to a filesystem which is cleanly u…

And even if that wasn't true, network-attached storage (unlike local storage) has no semantics for communicating a "partially completed" write of a block. Your server either manages to send an iSCSI packet to the SAN with a completed checksum, or it doesn't. Which means that—for the problems that would arise from a sudden power-cut to a VM (let's say from unexpected hypervisor failure)—using a journalling filesystem…

Common filesystems only do metadata journaling, so your file contents are not protected by this. As an exception, the ext3 and ext4 filesystems support a data journaling mode using a special flag.

Even if you had data journaling, it won't give you consistency between different files. This post used Gitlab as an example, and git will break if some files in its databse are updated, but some not. Git doesn't use fsync to ensure their update order, I don't know if Gitlab enables it or if the performance hit is reasonable.

Re: Persisting state between AWS EC2 spot instances

#72

Post by subway »

solatic wrote: post #46
jen20 wrote: post #43

Earlier quoted context omitted.

This pattern is a lot safer if you use ZFS. Spot instances don't just disappear though, you get notification and have a chance to perform shutdown actions, except in the case of hardware failure - which is the same with non-spot instances.

- EBS, being block storage, doesn't recognize the filesystem format on top of it, and therefore doesn't recognize if you formatted the block storage as ZFS and therefore will not use ZFS snapshots when using Amazon's native EBS snapshotting. If you wish to use ZFS snapshots, you have to build that on top of what Amazon gives you, along with all the other aspects of ZFS storage, i.e. building a ZFS storage pool from s…

SIGPWR is anything but standard, and it's unclear how AWS would even send that signal to your processes without adding an agent to the instance.

Currently they initiate an ACPI shutdown event at the termination time. It's hard to initiate a shutdown in a more standardized manner. An instance shut down via this signal will generally see the init process begin gracefully stopping services, eventually halting on it's own. Typically your init process will get increasingly aggressive with kill signals, as defined by your service definitions, eventually getting to SIGKILL. If your init process fails to get the vcpu halted, after a (undocumented?) period AWS will halt the cpu(s) for you. This is about as graceful a shutdown as you're going to get with 'standard' interfaces.

Termination Notifications go out of their way to give you an extra heads up, in case your application is unlikely to gracefully handle being shut down by the init system. Think DB hosts with a craploads of dirty blocks that take a few minutes to sync to disk at shutdown.

Re: Persisting state between AWS EC2 spot instances

#73

Post by bdcravens »

archgoon wrote: post #68

So I was pleasantly surprised to discover that for the last several years, spot instances have provided a mechanism that give you 2 minutes notice prior to shutdown: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-inte... Learn something new everyday. :) https://aws.amazon.com/blogs/aws/new-ec2-spot-instance-termi...

See my top-level comment - you can now set "shutdown" behavior to stop instead of terminate (though 2-minute notice still useful)

Re: Persisting state between AWS EC2 spot instances

#74

Post by manigandham »

objectivefs wrote: post #67
manigandham wrote: post #24

Persistent storage remains a complicated problem. Attaching volumes on the fly with docker volume abstraction works well enough for most cloud workloads, whether on-demand or spot, but it's still easy to run into problems. This is leading to rapid progress in clustered/distributed filesystems and it's even built into the Linux kernel now with OrangeFS [1]. There are also commercial companies like Avere [2] who make f…

Using a clustered/distributed filesystem definitively simplifies persisting the state between EC2 spot instances. It also makes it easier to scale out the work load when you need more instances accessing the same data. To add to your list: there is also ObjectiveFS[1] that integrates well with AWS (uses S3 for storage, works with IAM roles, etc) and EC2 spot instances. [1]. https://objectivefs.com

This looks very interesting, good competition to Avere based on info so far. Is there any native kubernetes integration in the works?

Re: Persisting state between AWS EC2 spot instances

#75

Post by objectivefs »

manigandham wrote: post #74
objectivefs wrote: post #67

Earlier quoted context omitted.

Using a clustered/distributed filesystem definitively simplifies persisting the state between EC2 spot instances. It also makes it easier to scale out the work load when you need more instances accessing the same data. To add to your list: there is also ObjectiveFS[1] that integrates well with AWS (uses S3 for storage, works with IAM roles, etc) and EC2 spot instances. [1]. https://objectivefs.com

This looks very interesting, good competition to Avere based on info so far. Is there any native kubernetes integration in the works?

We are looking into the best way to add native kubernetes support. Currently, you can add a mount on the host or directly mount the file system inside the container. Both approaches work well, so it mainly depends on your preferred architecture.

Re: Persisting state between AWS EC2 spot instances

#76

Post by manigandham »

objectivefs wrote: post #75
manigandham wrote: post #74

Earlier quoted context omitted.

This looks very interesting, good competition to Avere based on info so far. Is there any native kubernetes integration in the works?

We are looking into the best way to add native kubernetes support. Currently, you can add a mount on the host or directly mount the file system inside the container. Both approaches work well, so it mainly depends on your preferred architecture.

A persistent volume provider would be great: https://kubernetes.io/docs/concepts/storage/persistent-volum...

This makes it easy to declare the volume as part of the deployment and automatically attach storage when the container is run. Mounting on the host isn't very easy (or even possible sometimes), especially with spot/preemptible instances and the increasing abstractions by managed K8S providers. The pricing model might need to be different though if billing on a container-mount level.

Re: Persisting state between AWS EC2 spot instances

#77

Post by colechristensen »

sitkack wrote: post #69
colechristensen wrote: post #64

Earlier quoted context omitted.

All you're doing is advocating larger, less frequent failures with people who know less. Robustness isn't just about your software or your ops setup, but also about your people and their knowledge and experience. I cannot see how less frequent, more intense failures with people who know less is preferable, and that anything else is "very dangerous advice" You will ultimately have many fewer resources available if you…

You are both right. But both wrong. If you want better consistency, use either object storage or a database. If you are mutating multiple entities and need consistency, now you need a distributed transaction. But ALL cloud providers provide warning before an instance is shutdown. There is absolutely no reason, other than a crash for an instance to have a hard shutdown.

He makes valid points, but in defense of an original ridiculous statement that the articles suggestions are extremely dangerous. There are all sorts of benefits to an ACID database, it's just not reasonable to scream about the necessity of it because reboots are scary.

Re: Persisting state between AWS EC2 spot instances

#78

Post by mayank »

yjftsjthsd-h wrote: post #6
olegkikin wrote: post #4

If you don't care about reliability, why not just get a cheap and powerful VPS? Paying $90/month for that machine is madness. I pay $6/month for 6GB RAM, 4 cores, 50GB disk.

Perhaps integration with other AWS services?

AWS Lightsail is AWS’s option there.

Re: Persisting state between AWS EC2 spot instances

#79

Post by sitkack »

colechristensen wrote: post #77
sitkack wrote: post #69

Earlier quoted context omitted.

You are both right. But both wrong. If you want better consistency, use either object storage or a database. If you are mutating multiple entities and need consistency, now you need a distributed transaction. But ALL cloud providers provide warning before an instance is shutdown. There is absolutely no reason, other than a crash for an instance to have a hard shutdown.

He makes valid points, but in defense of an original ridiculous statement that the articles suggestions are extremely dangerous. There are all sorts of benefits to an ACID database, it's just not reasonable to scream about the necessity of it because reboots are scary.

I agree.

But! Lots of applications aren't built to handle partial writes, which will absolutely occur if apps are hard killed. Any disucssion around this topic should reference Crash-only Software [0][1][2] and Micro Reboots [3]

[0] https://en.wikipedia.org/wiki/Crash-only_software

[1] https://www.usenix.org/conference/hotos-ix/crash-only-softwa...

[2] https://lwn.net/Articles/191059/

[3] https://www.usenix.org/legacy/event/osdi04/tech/full_papers/...

Post reply on HN