Live data from Hacker News

Amazon Elastic File System

aws.amazon.com

41–50 of 217 posts

Re: Amazon Elastic File System

#41

Earlier quoted context omitted.

Yeah, I have a WP Multisite cluster that's a real pain to manage for this reason. Probably will be my first use of EFS.

How do you manage it now? Plugin? S3?

Our uploads folder is an s3fs mountpoint with a custom plugin to rewrite URLs to a CloudFront distribution in front of it.

The rest gets updated on a dev server then deployed to the auto-scaling group via an AMI.

Re: Amazon Elastic File System

#42
post #5

As someone who doesn't know AWS much, what are the major differences between this and EBS?

As has been mentioned, EBS can only be mounted to one instance at a time. If you could mount it to multiple it would effectively be the same thing, but then you have all sorts of write-locking issues.

As the rest of the comments also allude, a lot of the cloud-entrenched world has abandoned NFS, or at least in AWS circles.

I'm not one of these people. Rather than relying solely on puppet->all instances to handle multiple deploys, the convenience of an NFS instance was appealing, so essentially I have a relatively small (read: medium) instance that does nothing but manage the deployment filesystems and allow new in-security groups to connect. I have often though of abandoning this for doing deploys in a more "modern" way but I'm still not sure what the benefit would be other than eliminating a minor source of shame.

The analog here is running a MySQL or Postgres database in an instance prior to RDS. RDS provided enough benefit that the minor price difference in rolling your own no longer factored in. A more reliable, fault-tolerant and extensible file system is, like RDS, a huge upgrade. It may not be for everyone, but for some of us it's just another reason why AWS keeps making it hard to even look anywhere else.

Re: Amazon Elastic File System

#44
post #36

Does anyone else cringe when someone suggestions using NFS in production? I can't be the only one that has been woken up at 2am because of an NFS outage.

Although skeptical about it going into the project, I've deployed a virtualized oracle rac environment over 10G NFS and with some tuning it was stable and performant. If it is good enough for rac, which has some of the most stringent latency / performance requirements that I've seen, I'd say that it is probably good enough for quite a few production use cases, although to be fair this was only an 8 node cluster.

Re: Amazon Elastic File System

#46
post #4

We have been needing this very thing for hosting a large number of assets across multiple ec2 instances. Hope they roll this out quickly.

For the record, there has never been anything stopping you from spinning up an NFS EC2 that all of your other EC2s can use.

Re: Amazon Elastic File System

#47
This is what should have existed instead of EBS all along.

I'll never consider this to be as reliable as S3, but if I'm going to have a network filesystem I'd rather be dealing with NFS as my abstraction instead of virtualized network block devices.

Re: Amazon Elastic File System

#48
post #12

Earlier quoted context omitted.

S3fs filesystems are really slow. We tested around 10mb/s for file upload. Where it really struggles is when you have a lot of files in a folder. Try doing an 'ls' on a folder with hundreds of files to see it break.

As always, it depends on your use case. Just because it can be slow doesn't mean it's not a viable (and in some cases superior) option. We use it to store petabytes of large video files and our system is structured such that no folder ever has more than a couple files in it (>20 is rare). With properly tuned caching this works fantastically well for our use case and I would take the simpler code and reduced points of…

Which s3fs are you using?

The fuse-based ones that I've tried were ridden with problems and poor error handling. Hangs and truncated files were the rule rather than the exception.

Re: Amazon Elastic File System

#50
post #47

This is what should have existed instead of EBS all along. I'll never consider this to be as reliable as S3, but if I'm going to have a network filesystem I'd rather be dealing with NFS as my abstraction instead of virtualized network block devices.

May I ask why? My experience with NFS is pretty bad performance wise.
Post reply on HN