This is going to be a really interesting way to benchmark EBS performance.
A shared file system for lambda functions
21–30 of 113 posts
Re: A shared file system for lambda functions
#22Serverless is a bit like Stone Soup [1]. This I guess is the point at which the Tramp says: "Now if you just add a few onions it really helps the flavour..."
Re: A shared file system for lambda functions
#23Ooh. I wonder if efs is compatible with sqlite’s nfs mode.
More seriously, this is huge. Unix pipes over shared nfs has always been my big data platform of choice (since before the cloud, or even google map reduce). Things finally came full circle.
Re: A shared file system for lambda functions
#24I could see this being very useful. I recall Joyent's solution to this (similar) problem where you have an object stored somewhere (e.g. S3) and you want to use that object in a container, but you have to copy it over HTTP or something to do any work on it and the object could be very large. With Joyent's Manta[1] you would spin up a container right where an object is stored (instead of bringing the objects to the co…
Sort of. The better analogy would be spinning up compute localized to the s3 object; which would be pretty interesting.
This feature they did release deserves little fanfare.
Re: A shared file system for lambda functions
#25sharp knife to hand people -- because EFS is just NFS, it uses NFS for security / isolation. Everything that can mount a given volume needs to agree on what unix users are what, and you need to make sure to completely lock down root access, otherwise you can't enforce any kind of data isolation.
If your use case can deal with one EFS volume per isolation boundary, you can use IAM to control who can mount what volume, which might be easier to reason about.
Cloud-y DLP tools don't know about EFS.
Re: A shared file system for lambda functions
#26I could see this being very useful. I recall Joyent's solution to this (similar) problem where you have an object stored somewhere (e.g. S3) and you want to use that object in a container, but you have to copy it over HTTP or something to do any work on it and the object could be very large. With Joyent's Manta[1] you would spin up a container right where an object is stored (instead of bringing the objects to the co…
Sort of. The better analogy would be spinning up compute localized to the s3 object; which would be pretty interesting. This feature they did release deserves little fanfare.
Ummm. Linux’s NFS client includes a kernel page cache.
You can just mmap or read the file without doing anything else. That is zero or one memcpy overhead.
S3 clients have to copy the data over the network, assemble the tcp packets, decrypt and checksum for ssl, and then memcpy the result. That’s at a minimum. They may be doing other work, like verifying the s3 checksum, or allocating memory to store the object.
They have to do that once per lambda process, again, at a minimum. They might do it once per lambda invocation.
I wonder how amazon bills DRAM if multiple lambdas mmap the same thing read only.
Re: A shared file system for lambda functions
#27Curious if it adds significantly to cold start times.
Re: A shared file system for lambda functions
#28Curious if it adds significantly to cold start times.
They've already reduced cold starts significantly [0] for Lambas in a VPC, I can't see them tossing all that good work for this.
[0] - https://aws.amazon.com/blogs/compute/announcing-improved-vpc...
Re: A shared file system for lambda functions
#29Cool, so we're staring to curve more sharply around the full circle we'll eventually go on.
So now lambda functions can mount persistent block storage.
Next up: allow your lamda functions to run for longer
Then: allow multiple lamda functions to execute concurrently, and indefinitely, as a group, while having block storage mounted
And finally: use your EC2 instances as lambda functions
Re: A shared file system for lambda functions
#30Cool, so we're staring to curve more sharply around the full circle we'll eventually go on. So now lambda functions can mount persistent block storage. Next up: allow your lamda functions to run for longer Then: allow multiple lamda functions to execute concurrently, and indefinitely, as a group, while having block storage mounted And finally: use your EC2 instances as lambda functions
I agree that we will see all of those, except the last one. It's no surprise that we're bending around, that's normal. But when we get back to the "feature parity with the past" things will still look very different. We're talking about fully managed systems here that you can compose together to get parity with what you have today with unmanaged systems.
I also don't think the time limit on lambdas will extend much further than it is today. Not sure on that one.