Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Elastic File System

aws.amazon.com

Post reply on HN
Print view View original

81–90 of 217 posts

Re: Amazon Elastic File System

#81

Post by late2part »

alrs wrote: post #47

This is what should have existed instead of EBS all along. I'll never consider this to be as reliable as S3, but if I'm going to have a network filesystem I'd rather be dealing with NFS as my abstraction instead of virtualized network block devices.

I believe EBS was the right building block at the right time, and I'll still use it over EFS in the majority of my deployment designs.

I still like to "trifurcate" the storage into objects, local disposable, and local volumes. Having durable local volumes still makes sense in a lot of scenarios.

Re: Amazon Elastic File System

#84

Post by evol262 »

notacoward wrote: post #80
evol262 wrote: post #64

Earlier quoted context omitted.

It's almost certainly NFSv4 only so they can utilize pNFS

Not necessarily. IIRC, pNFS is an optional feature of NFSv4.1, so they might not have implemented that flavor. If they had, I'm pretty sure they'd be advertising it. They're not shy.

Technical details aren't out yet. They may not have implemented it, but given their comments about performance and scaling, it would be a very good reason not to support NFSv3.

Re: Amazon Elastic File System

#85

Post by notacoward »

As a GlusterFS developer, and furthermore the founder of a project to create a "cloud" version of GlusterFS aimed at exactly this use case, this is pretty darn interesting to me. I guess I'm supposed to pick away at all the feature differences between EFS and GlusterFS-on-EC2, or something like that, but for now I'm more pleased to see that this use case is finally being addressed and the solution seems well integrated with other Amazon features. Kudos to the EFS team.

Re: Amazon Elastic File System

#86

Post by tlrobinson »

pyre wrote: post #70
biot wrote: post #54

All services should eventually be like this. Just as you don't want to deal with provisioning BTUs of air conditioning or watts of power needed for your cloud infrastructure, why should you concern yourself about allocating a certain number of bytes of storage?

> Just as you don't want to deal with provisioning BTUs of air conditioning or watts of power needed for your cloud infrastructure Maybe you don't want to, but there is definitely someone out there dealing with these issues. E.g. during a heat-wave (100 F+) a transformer on top of the building (at a previous employer) started on fire. When the dust settled, we found out that the person in charge of it had not upgrade…

Right... which is why software should do it automatically.

Re: Amazon Elastic File System

#87

Post by graceofs »

b1twise wrote: post #65

This would be big for us. When we initially looked at the problem of sharing or keeping a large number of files in sync, the prospects were dim. DRBD? etc. So we ended up using Gluster. Gluster has been temperamental at best. We've been able to move some data out and into elasticsearch, but not all. So, I've nudged my AWS rep and signed up already. Reliable NFS is good for me.

We couldn't find a solution either, so we built a posix filesystem with S3 backend backend that is easy to run and scale. If you want to give ObjectiveFS (https://objectivefs.com) a try, I'll be happy to hear your feedback.

Re: Amazon Elastic File System

#88

Post by fi788 »

bmurphy1976 wrote: post #29
fi788 wrote: post #12

Earlier quoted context omitted.

S3fs filesystems are really slow. We tested around 10mb/s for file upload. Where it really struggles is when you have a lot of files in a folder. Try doing an 'ls' on a folder with hundreds of files to see it break.

As always, it depends on your use case. Just because it can be slow doesn't mean it's not a viable (and in some cases superior) option. We use it to store petabytes of large video files and our system is structured such that no folder ever has more than a couple files in it (>20 is rare). With properly tuned caching this works fantastically well for our use case and I would take the simpler code and reduced points of…

I agree with your points. It is indeed a viable, if somewhat clunky solution.

For getting the data into S3, we found exponential improvements in using the AWS CLI, as I believe it handles uploads in a multi-threaded way.

S3fs turned out to be viable for our use case, storing Magento Enterprise content assets which are then served directly from S3, so the app's upload features rely on s3fs as well as the file checks from the app itself (which are indeed quite slow).

I've always wanted to do it natively, mounting EBS volumes on more than one instance (which is not currently possible) or wishing for a native NFS service like AWS released.

All in all, it is a happy day for me. More options make us more powerful.

Re: Amazon Elastic File System

#90

Post by cwyers »

jstoiko wrote: post #75
DenisM wrote: post #66

Salient points: - NFS (v4). - Supports petabyte-scale file systems, thousands of concurrent NFS connections. - Automatically grows/shrinks in size. - Multi-zone storage and access. - $0.30 / (Gigabyte * month). - (Not mentioned) Both Linux and Windows have built-in NFS clients.

Thanks for the time-saving description. Osx also has a built-in NFS client.

Until Amazon starts offering EC2 OS X instances, I'm not sure that's as relevant.
Post reply on HN