Amazon Elastic File System – Production-Ready in Three Regions
1–10 of 122 posts
Re: Amazon Elastic File System – Production-Ready in Three Regions
#2Can EFS be mounted from the Lambda environment?
Re: Amazon Elastic File System – Production-Ready in Three Regions
#3We use a columnar database and we insert data to it periodically (30s to 90s) using Kinesis as middleware. Files are immutable and each insert creates [count of columns] * [count of tables] files. The database also periodically merges the batch files in background.
Since EFS was not available, I was trying to develop a FUSE filesystem that writes data to both S3 and local filesystem (EBS) for durability and read only from local filesystem for performance. Even though it's cheaper than EFS, it's slow since each file needs to be sent to S3 individually. Our requirements are not that much because the files are immutable, there are not many files (I heard that EFS doesn't work well with many small files.), no need for concurrent access. I think I will give it a shot since it also fits our use-case.
Re: Amazon Elastic File System – Production-Ready in Three Regions
#4Can EFS be mounted from the Lambda environment?
Not at present. I would be interested in learning about use cases for this. Post here or find me online, as you wish.
Re: Amazon Elastic File System – Production-Ready in Three Regions
#5Can EFS be mounted from the Lambda environment?
Not at present. I would be interested in learning about use cases for this. Post here or find me online, as you wish.
You mention HPC and Big Data processing as use cases for EFS; Lambda is also very well suited for big processing tasks you want to execute quickly with a large amount of concurrency. You often need persistence across executions, for example when breaking tasks into multi-function workflows, and EFS would seem to simplify this over manually using S3 for it.
Re: Amazon Elastic File System – Production-Ready in Three Regions
#6Can EFS be used as a volume for ECS tasks?
Re: Amazon Elastic File System – Production-Ready in Three Regions
#7Can EFS be used as a volume for ECS tasks?
Here's a post we wrote earlier in the year about EFS and ECS: https://aws.amazon.com/blogs/compute/using-amazon-efs-to-per...
Re: Amazon Elastic File System – Production-Ready in Three Regions
#8Re: Amazon Elastic File System – Production-Ready in Three Regions
#9Can EFS be mounted from the Lambda environment?
Not at present. I would be interested in learning about use cases for this. Post here or find me online, as you wish.
One use case would be one that came up a while back here on HN -- The person had a large binary blob that they needed to load up every time their Lambda function ran. If they could mount EFS from Lambda, they could put the large binary blob there.
Depending on how they wrote their code, if the Lambda could seek to the right part of the file then this would work, but if it has to load the entire file into RAM anyway, then it wouldn't gain them much except perhaps a slightly easier way to get the file into RAM.
Re: Amazon Elastic File System – Production-Ready in Three Regions
#10This is really cool but also kind of a shame. I feel like it will encourage bad behavior.
I can see people using this to deploy code by deploying onto the shared volume so that all their instances get an "instant update". Which is great when it works, but lord help you when EFS goes down and every app server you have is hung on a broken NFS connection.