Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Elastic File System – Production-Ready in Three Regions

aws.amazon.com

Post reply on HN
Print view View original

101–110 of 122 posts

Re: Amazon Elastic File System – Production-Ready in Three Regions

#101

Post by walden42 »

Corrado wrote: post #47
jedberg wrote: post #10

This is really cool but also kind of a shame. I feel like it will encourage bad behavior. I can see people using this to deploy code by deploying onto the shared volume so that all their instances get an "instant update". Which is great when it works, but lord help you when EFS goes down and every app server you have is hung on a broken NFS connection.

I think that deploying onto a shared volume is exactly the workload that EFS encourages. That things will stop functioning can be said about any (AWS) technology. "Lord help you when (S3|RDS|SQS|etc.) goes down and every (thing) you have is hung up on a broken (thing) connection." The rational argument here is that if uptime is important to you, the solution is to utilize multiple regions, of which EFS is now in 4. Y…

Assuming all assets get stored on S3, the other common writing scenario is logging. How would multiple machines writing to the same log file work?

Re: Amazon Elastic File System – Production-Ready in Three Regions

#102

Post by MOARDONGZPLZ »

omni wrote: post #96
willejs wrote: post #83

Earlier quoted context omitted.

HAHAHA. YES! So much this. THere are legit use cases for this, but there are many use cases that will cause you much pain.

I suspect you're being downvoted because you've described every piece of technology ever created.

Or the

"HAHAHA. YES! So much this."

part. Though I can't down vote so I wouldn't know.

Re: Amazon Elastic File System – Production-Ready in Three Regions

#104

Post by MrMullen »

Someone wrote: post #26
Sanddancer wrote: post #21

Earlier quoted context omitted.

It claims to present itself as a full filesystem so I don't see why it wouldn't, however, you're probably going to run into pretty hardcore performance issues, because the bandwidth amazon provides in general is on the stingy side, but under a terabyte, they're downright miserly, with a 10 gig filesystem only able to provide 0.5 MB/s sustained. This product is problematic at best.

Also, it is a full-fledged NFS-mounted file system. I would be very wary of running a database from that.

It sounds like a disaster waiting to happen. The latency of NFS is just too high (Even if it is very fast for NFS) for it to be safe for a database.

Re: Amazon Elastic File System – Production-Ready in Three Regions

#105

Post by KaiserPro »

merpnderp wrote: post #87
KaiserPro wrote: post #25

YES Finally we can start using standard posix semantics for things. Yes people might say its primitive, but currently all I can see is people trying to re-create shared files systems over protocols not designed for it cough cough HTTP Finally I can have a shared home, with a shared environment. A single readonly binary source (great for docker by the way.) also great for ensuring one version of scripts, without havin…

At 10x the price for S3.

You are correct. However its not for the same things.

S3 is high latency, HTTP semantics, no locking mechanism.

EFS is low latency, high throughput shared file system with standard posix interface. everything can write to standard files. Not many things can write using S3 effectively.

Re: Amazon Elastic File System – Production-Ready in Three Regions

#107

Post by [deleted] »

merpnderp wrote: post #87
KaiserPro wrote: post #25

YES Finally we can start using standard posix semantics for things. Yes people might say its primitive, but currently all I can see is people trying to re-create shared files systems over protocols not designed for it cough cough HTTP Finally I can have a shared home, with a shared environment. A single readonly binary source (great for docker by the way.) also great for ensuring one version of scripts, without havin…

At 10x the price for S3.

[deleted]

Re: Amazon Elastic File System – Production-Ready in Three Regions

#108

Post by SteveNuts »

MrMullen wrote: post #104
Someone wrote: post #26

Earlier quoted context omitted.

Also, it is a full-fledged NFS-mounted file system. I would be very wary of running a database from that.

It sounds like a disaster waiting to happen. The latency of NFS is just too high (Even if it is very fast for NFS) for it to be safe for a database.

We're running Oracle on NFS volumes, and some MySQL databases also with no problems whatsoever.

It'll never be as fast as local disk but the flexibility it provides is very nice.

Re: Amazon Elastic File System – Production-Ready in Three Regions

#109

Post by acdha »

takeda wrote: post #79
jedberg wrote: post #50

Earlier quoted context omitted.

The difference is that all of those services are accessed via an API, which means your client can do clever things when it fails, like timeout, give a fallback, find an equivalent resource in another zone or region, etc. With EFS, it's exposed as NFS, which hooks in much deeper. If it goes down, there isn't anything you can do to work around the problem, unless you start hacking your own file system kernel modules.

Can't you just use soft mounts? Especially since in the scenario you described you can use read only mounts.

It depends on the nature of the failure. If you're doing something which triggers hard failures – i.e. where a read() call on a ocket would unblock with an error – soft mounts will eventually recover. The problem is that, in my experience anyways, the vast majority of failures aren't that clean – things like a server which processes packets but never responds, a network connection which drops packets but doesn't change the link status, etc. — and in those cases soft mounts behave no better than hard mounts. There also used to be kernel bugs in *BSD, Darwin, and Linux where the client could deadlock in heavy activity, which were hard to reproduce and get fixed.

In all of those cases, anything which tries to access something on the NFS mount will block in the kernel (i.e. “kill -9“ won't work) and the mount cannot be unmounted normally.

I wrote https://github.com/acdha/mountstatus awhile back – if memory serves, 2004 or so – because we found that on Linux a lazy unmount would still work in this case and so you could have a process monitor the mount status (fork() a child to check the mount, alert if it doesn't get a response within a set interval) and a watchdog could respond to an alert by issuing a “umount -l” and remounting, which doesn't fix the blocked process but is less disruptive than rebooting and that new processes won't block because they tried to access that mount.

Re: Amazon Elastic File System – Production-Ready in Three Regions

#110

Post by jtrtoo »

Corrado wrote: post #49

I am excited by EFS but during the beta it didn't perform as well as I would have liked. I had a small Wordpress install that I wanted to make elastic in that I could have multiple web servers and all of the data was stored apart from the AMI. After testing my setup I found that latency was way too high and the performance of my site was terrible. I'm guessing it's because the amount of data was really low (100MB) an…

I think you might be going about solving this problem in a sub-optiimal way. WordPress data consists of db + asset files. Db gets addressed outside of this and asset files either go in s3 or are synced (rsync, etc)
Post reply on HN