A shared file system for lambda functions • Hacker News Board

Re: A shared file system for lambda functions

#71

Post by garethmcc » Wed, Jun 17, 2020, 6:36 AM UTC

VWWHFSfQ wrote: post #42

I'm curious what people actually use Lambda for? I tried Lambda for a use-case that I had in 2018: We published Polls and Predictions to people watching the 2018 World Cup. We set the vote callback URL to a function on AWS Lambda. It failed spectacularly during our load-testing because the ramp-up period was far too slow. We needed to go from 0 to 100,000 incoming requests/second in about 20 seconds. We had to switch…

There are an enormous number of people using Lambda's as a part of very complex workloads including customer facing platforms. Your specific use case of 100 000 incoming requests in 20 seconds is actually far more extreme than most users use cases; Lambda can do 3000 instantly and then adds more capacity over time unless you contact AWS and ask them to increase the limits in which it can do a lot more.

There are very many users of Lambda that run very high volume production workloads (billions of requests a month) and save an enormous amount of money doing so compared to traditional techniques. A lot of that saving comes from the ability to create solutions in days as opposed to weeks or even months and the fact that for most use cases the capacity exists to handle the volume most users need.

Re: A shared file system for lambda functions

#72

Post by garethmcc » Wed, Jun 17, 2020, 6:39 AM UTC

abrookewood wrote: post #52

VWWHFSfQ wrote: post #42

I'm curious what people actually use Lambda for? I tried Lambda for a use-case that I had in 2018: We published Polls and Predictions to people watching the 2018 World Cup. We set the vote callback URL to a function on AWS Lambda. It failed spectacularly during our load-testing because the ramp-up period was far too slow. We needed to go from 0 to 100,000 incoming requests/second in about 20 seconds. We had to switch…

It works pretty well for tasks that are executed infrequently (no need to have a full application deployed), out of band (no need for a rapid response; we'll email you) and for tasks that can consume excessive resources (threatening other apps running on the server). For us the perfect use case is the production of PDF reports: they aren't run that often; they can be emailed a few minutes later; and when too many of…

Not entirely true. The largest use case we see for Lambda is as the compute service for an API backend provisioned with API Gateway (so a full application backend) with synchronous responses over HTTP (so rapid response provided within milliseconds) and for mundane tasks that store data in a database just like a regular web application.

The perfect use case we see repeatedly is for high volume HTTP requests to API Gateway endpoints that trigger Lambda functions that respond in less than a second depending on what compute is running.

Re: A shared file system for lambda functions

#73

Post by garethmcc » Wed, Jun 17, 2020, 6:40 AM UTC

redis_mlc wrote: post #49

VWWHFSfQ wrote: post #42

I'm curious what people actually use Lambda for? I tried Lambda for a use-case that I had in 2018: We published Polls and Predictions to people watching the 2018 World Cup. We set the vote callback URL to a function on AWS Lambda. It failed spectacularly during our load-testing because the ramp-up period was far too slow. We needed to go from 0 to 100,000 incoming requests/second in about 20 seconds. We had to switch…

AWS lambda is used in 2 categories: 1) AWS administration-related actions (required for non-trivial mgmt.) 2) end-user actions (optional). AWS throttles and limits everything, so expecting 100,000 requests in 20 seconds on a default AWS account is unreasonable. The Cloud doesn't mean unlimited capacity. Also, I haven't seen anybody version control lambda code, even in compliance environments, so something to investig…

I just want to add in about version controlling Lambda code, this is entirely possible using the available tooling with something like the Serverless Framework at serverless.com

Re: A shared file system for lambda functions

#74

Post by garethmcc » Wed, Jun 17, 2020, 6:42 AM UTC

meritt wrote: post #46

vosper wrote: post #44

Earlier quoted context omitted.

We use it as a cron substitute quite often, which I guess isn’t the serverless use case, but it’s quite handy

Cron is a scheduler that executes shell commands. Lambda represents the shell command itself right? What are you using to specify when something should run?

If you use a tool like the Serverless Framework, you can trigger Lambda functions off of many different events, includinmg a schedule: https://www.serverless.com/framework/docs/providers/aws/even...

Re: A shared file system for lambda functions

#75

Post by KaiserPro » Wed, Jun 17, 2020, 6:46 AM UTC

QuinnyPig wrote: post #68

Konohamaru wrote: post #66

This is misdirection by naming, like how University of Phoenix is similar to Arizona State University, Phoenix. "Lambda functions" sounds like anonymous functions, but is actually referring to a proprietary interface to AWS Lambda™. They named it this way so that readers confuse AWS Lambda™ for programming lambdas.

One of AWS’s core competencies is being bad at naming things.

with useless logos.

Re: A shared file system for lambda functions

#76

Post by KaiserPro » Wed, Jun 17, 2020, 6:50 AM UTC

jupp0r wrote: post #13

This is a horrible idea. This gives lambda functions shared mutable state to interfere with each other, with very brittle semantics compared to most databases (even terrible ones).

It can, but...

As you can seek() on NFS, you can quickly scroll to the middle of a massive file. Unlike S3 where you have to download the whole file first.

Re: A shared file system for lambda functions

#77

Post by KaiserPro » Wed, Jun 17, 2020, 6:53 AM UTC

abrookewood wrote: post #34

What is the advantage of using this over S3? Is it just the speed & latency difference between S3 and EFS?

You have the ability to seek() without downloading first.

Re: A shared file system for lambda functions

#78

Post by redis_mlc » Wed, Jun 17, 2020, 7:20 AM UTC

VWWHFSfQ wrote: post #50

redis_mlc wrote: post #49

Earlier quoted context omitted.

AWS lambda is used in 2 categories: 1) AWS administration-related actions (required for non-trivial mgmt.) 2) end-user actions (optional). AWS throttles and limits everything, so expecting 100,000 requests in 20 seconds on a default AWS account is unreasonable. The Cloud doesn't mean unlimited capacity. Also, I haven't seen anybody version control lambda code, even in compliance environments, so something to investig…

I guess that I was expecting AWS Lambda to scale up immediately in the same way the ALB or SQS scales up with demand. That assumption was very wrong. Edit: my AWS support rep explained it: Without pre-provisioned capacity, it would take 34 minutes to scale our Lambda functions to the peak concurrency we needed. And it would cost several thousand dollars. Completely insane cost. Also, EFS is ridiculously expensive.

> scale up immediately in the same way the ALB

You would need to pre-arrange that with your account manager too.

An ALB is hosted on servers, and when you hog those servers, they would need to migrate to bigger.

Re: A shared file system for lambda functions

#79

Post by dmortin » Wed, Jun 17, 2020, 7:36 AM UTC

VWWHFSfQ wrote: post #42

I'm curious what people actually use Lambda for? I tried Lambda for a use-case that I had in 2018: We published Polls and Predictions to people watching the 2018 World Cup. We set the vote callback URL to a function on AWS Lambda. It failed spectacularly during our load-testing because the ramp-up period was far too slow. We needed to go from 0 to 100,000 incoming requests/second in about 20 seconds. We had to switch…

Not AWS, but I use Cloud Functions in Firebase Hosting to redirect dynamic urls to static website paths.

Of course, in this case I have to use it, because Firebase Hosting is static, so this is the only way to handle dynamic urls if the simple rules of firebase.json are not enough.

Re: A shared file system for lambda functions

#80

Post by geertj » Wed, Jun 17, 2020, 7:56 AM UTC

boulos wrote: post #53

Disclosure: I work on Google Cloud. It wasn't obvious to me if this is somehow mounting EFS over !NFS (since never says the words NFS in the post). My main fear when people say "Should I use Lambda / Google Cloud Functions / Cloud Run against my NFS server" my response isn't "How would you set that up" it's "Be careful. Cleaning up NFS locks held by clients that have gone away is fairly painful, and you have none of…

(PM-T in the EFS team)

This is NFS and locking is fully supported (the blog includes an example). Because EFS implements the NFS 4.0/4.1 protocols, locks are lease-based and there isn't a need to clean them up. In the unlikely event of a client crash where the client still held locks, they will automatically expire once the associated lease expires.