Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A shared file system for lambda functions

aws.amazon.com

Post reply on HN
Print view View original

81–90 of 113 posts

Re: A shared file system for lambda functions

#81

Post by geertj »

philsnow wrote: post #25

sharp knife to hand people -- because EFS is just NFS, it uses NFS for security / isolation. Everything that can mount a given volume needs to agree on what unix users are what, and you need to make sure to completely lock down root access, otherwise you can't enforce any kind of data isolation. If your use case can deal with one EFS volume per isolation boundary, you can use IAM to control who can mount what volume,…

(PM-T on the EFS team)

The EFS/Lambda integration uses EFS Access Points, which allow you to enforce a specific POSIX identity and directory for NFS operations. You can also use IAM policies to require that specific IAM roles/users use a specific access point.

Re: A shared file system for lambda functions

#82

Post by matteuan »

VWWHFSfQ wrote: post #42

I'm curious what people actually use Lambda for? I tried Lambda for a use-case that I had in 2018: We published Polls and Predictions to people watching the 2018 World Cup. We set the vote callback URL to a function on AWS Lambda. It failed spectacularly during our load-testing because the ramp-up period was far too slow. We needed to go from 0 to 100,000 incoming requests/second in about 20 seconds. We had to switch…

Preprocessing and forwarding data to on-premise

Re: A shared file system for lambda functions

#83

Post by nickcw »

KaiserPro wrote: post #77
abrookewood wrote: post #34

What is the advantage of using this over S3? Is it just the speed & latency difference between S3 and EFS?

You have the ability to seek() without downloading first.

You can seek with S3 using the Range header. The latencies are poor though!

Re: A shared file system for lambda functions

#84

Post by matteuan »

ralusek wrote: post #2

Pretty sweet. Is anybody using Lambda to run huge MapReduce jobs? Do people still use Hadoop? Doesn't this basically just let you have something like HDFS for running large distributed computations with some shared state, without having to reach for S3 or redis?

We use Lambda with S3 as intermediate storage between different steps, sort of multiple-stages map-only MapReduce. And we still use Hadoop on premise :)

Re: A shared file system for lambda functions

#85

Post by hamandcheese »

KaiserPro wrote: post #76
jupp0r wrote: post #13

This is a horrible idea. This gives lambda functions shared mutable state to interfere with each other, with very brittle semantics compared to most databases (even terrible ones).

It can, but... As you can seek() on NFS, you can quickly scroll to the middle of a massive file. Unlike S3 where you have to download the whole file first.

S3 allows you to specify a byte range.

Re: A shared file system for lambda functions

#86

Post by vosper »

meritt wrote: post #46
vosper wrote: post #44

Earlier quoted context omitted.

We use it as a cron substitute quite often, which I guess isn’t the serverless use case, but it’s quite handy

Cron is a scheduler that executes shell commands. Lambda represents the shell command itself right? What are you using to specify when something should run?

Lambda has built in support for scheduling, including cron syntax https://docs.aws.amazon.com/lambda/latest/dg/services-cloudw...

Re: A shared file system for lambda functions

#88

Post by alex_duf »

VWWHFSfQ wrote: post #42

I'm curious what people actually use Lambda for? I tried Lambda for a use-case that I had in 2018: We published Polls and Predictions to people watching the 2018 World Cup. We set the vote callback URL to a function on AWS Lambda. It failed spectacularly during our load-testing because the ramp-up period was far too slow. We needed to go from 0 to 100,000 incoming requests/second in about 20 seconds. We had to switch…

We use it for a large variety of situation.

From direct exposition (via API gateway) to HTTP traffic, to very spiky loads such that we don't pay for hardware in between spikes. We've also have a kind of "hybrid pipeline" that is a mix of both HTTP requests from clients and SQS messages. The HTTP part sends messages to SQS, which triggers lambdas asynchronously so we get retries for free.

Your load doesn't strike me as being out of the ordinary, I suspect you had one of the following: - Slow start, which means AWS was unable to re-use an already warm container to run the function a second start. This is particularly true with Java, although they've massively improved it recently. - A dependency on another service which would have caused the slow start (loading a config is a common unexpected culprit) - You mention 2018, so at that time running a lambda within a VPC was notoriously slow to start - You had a limit on your AWS account (default is 1k concurrent I believe)

Re: A shared file system for lambda functions

#89

Post by saurik »

OMG THIS IS SO AMAZING I HAVE BEEN WANTING THIS FOR AN ENTIRE YEAR NOW. (I've been using Lambda to do massively distributed compile jobs, but had reached the throughput limits I could achieve with distcc-like techniques doing local preprocessing, and so was looking at doing limited synchronization of my codebase to S3 to then either link against the compiler or, for other tools and to let me use the gold standard compiler I want, do C runtime injection to make it so that when files are opened I pull them from S3... but that entire process sucked and doesn't really solve the general purpose problem: this does; this lets me trivially do the moral equivalent of make -j1000 and have all of the random sub-jobs get executed in lambda functions and have the compile complete nearly instantaneously. I can even have those jobs just directly share state and do "exactly what you'd expect" with respect to the inter-dependency stuff <- which like, is a tradeoff, but one that fits well with how most projects are already designed when using make... I'm so pumped to go back and work on that project again.)

Re: A shared file system for lambda functions

#90

Post by [deleted] »

KaiserPro wrote: post #76
jupp0r wrote: post #13

This is a horrible idea. This gives lambda functions shared mutable state to interfere with each other, with very brittle semantics compared to most databases (even terrible ones).

It can, but... As you can seek() on NFS, you can quickly scroll to the middle of a massive file. Unlike S3 where you have to download the whole file first.

[deleted]
Post reply on HN