Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A shared file system for lambda functions

aws.amazon.com

Post reply on HN
Print view View original

11–20 of 113 posts

Re: A shared file system for lambda functions

#11

Post by spullara »

tidepod12 wrote: post #8
kleebeesh wrote: post #4

I've found EFS enticing in theory but painfully slow and riddled with issues in practice. In the past I've tried it thinking "it's basically an EBS volume I can mount on > 1 EC2 instance," only to find terrible read performance and misc. low-level NFS errors.

Dunno your exact requirements or when you last tried it, but they did boost EFS's read speed (they claim by 400% [1]) as of this April, so it might be worth looking into again if you're still trying to find a solution. 1: https://aws.amazon.com/about-aws/whats-new/2020/04/amazon-el...

That is great to know, thanks. It was pretty unusable the last time I tried it.

Re: A shared file system for lambda functions

#12

Post by tw04 »

kleebeesh wrote: post #4

I've found EFS enticing in theory but painfully slow and riddled with issues in practice. In the past I've tried it thinking "it's basically an EBS volume I can mount on > 1 EC2 instance," only to find terrible read performance and misc. low-level NFS errors.

You should try one of the various netapp options. Marketplace but significantly better performance and a platform that’s got several decades of nas experience.

https://cloud.netapp.com/cloud-volumes-service-for-aws

Whether amazon will ever let you mount it to lambda is a very different discussion.

Re: A shared file system for lambda functions

#14

Post by actionowl »

I could see this being very useful.

I recall Joyent's solution to this (similar) problem where you have an object stored somewhere (e.g. S3) and you want to use that object in a container, but you have to copy it over HTTP or something to do any work on it and the object could be very large.

With Joyent's Manta[1] you would spin up a container right where an object is stored (instead of bringing the objects to the container via NFS.) Also has map reduce support.

[1] https://apidocs.joyent.com/manta/jobs-reference.html

Re: A shared file system for lambda functions

#15

Post by Havoc »

jupp0r wrote: post #13

This is a horrible idea. This gives lambda functions shared mutable state to interfere with each other, with very brittle semantics compared to most databases (even terrible ones).

It certainly has some potential for shooting oneself in the foot in some scenarios, but still nice to have the option

Re: A shared file system for lambda functions

#16

Post by scarface74 »

jupp0r wrote: post #13

This is a horrible idea. This gives lambda functions shared mutable state to interfere with each other, with very brittle semantics compared to most databases (even terrible ones).

I need this specifically as a part of a state machine. Most of the steps involve a Lambda loading and unloading csv data between S3, Redshift, and Aurora where no local storage is needed. The last step where we had to download the files locally and compress multiple files together was done manually via a script because they were greater than 512Mb.

We were just about to put the script in Fargate (Serverless Docker) and run a ECS task as part of the state machine. Now we don’t have to.

Re: A shared file system for lambda functions

#18

Post by riquito »

jupp0r wrote: post #13

This is a horrible idea. This gives lambda functions shared mutable state to interfere with each other, with very brittle semantics compared to most databases (even terrible ones).

"practicality beats purity"

Depending on the use case it can be a good option (some are listed in the article). If the dev doesn't get the disadvantages the rest of the architecture will hardly be correct anyway

Re: A shared file system for lambda functions

#19

Post by hansef »

scarface74 wrote: post #16
jupp0r wrote: post #13

This is a horrible idea. This gives lambda functions shared mutable state to interfere with each other, with very brittle semantics compared to most databases (even terrible ones).

I need this specifically as a part of a state machine. Most of the steps involve a Lambda loading and unloading csv data between S3, Redshift, and Aurora where no local storage is needed. The last step where we had to download the files locally and compress multiple files together was done manually via a script because they were greater than 512Mb. We were just about to put the script in Fargate (Serverless Docker) a…

This - if you have to fetch data from or output data to outside of the AWS ecosystem, the 512mb /tmp limit pushes you into the additional (relative) complexity of having to run on Fargate pretty quickly. Just had to deal with this for a content ingest job involving pulling a couple GB of data from an FTP server, processing it and pushing it into an RDS database on an hourly basis. Would have been super simple if the file was on S3 already.

Re: A shared file system for lambda functions

#20

Post by booi »

tanilama wrote: post #10
black_puppydog wrote: post #3

Can we please change the title? I was really afraid that AWS might have lost it and called the new file system "new", which must be the least practical name for anything from an SEO standpoint.

Amazon New will be the worst name for a AWS product, since they announce TONS of products every year, nothing is really new.

Amazon New, a full-featured and elastic system for releasing new products.
Post reply on HN