Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How was your experience with AWS Lambda in production?

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 165 posts

Re: Ask HN: How was your experience with AWS Lambda in production?

#121

Post by philliphaydon »

runamok wrote: post #109
munns wrote: post #72

Hey all, My name is Chris Munns and I am currently the lead Developer Advocate for Serverless at AWS (I am part of the Lambda PM team). We really appreciate this feedback and are always looking for ways to hear about these pain points. Can email me directly: munns@amazon.com if you ever get stuck. Thanks, - Chris

I am on a devops team trying to implement lambda by splitting pieces off from a monolithic Java app where applicable. The biggest pain point I have is because we have multiple "environments" (such as dev, da, staging) in the same amazon account and because lambdas are global I can't limit access to resources via IAM easily without hacks. Aka, because the same lambda will be used (but different versions and/or aliases…

To get around the IAM issue we deployed the lambda's with a naming convention like:

prod_lambda_name staging_lambda_name dev_lambda_name

Then the IAM's are written with resource access to prod_* staging_* etc.

It allows to give full permissions to the developer to create dev ones, modify the other ones, but the prod_ are all controlled by a smaller group of people.

It's a bit hacky but it works well enough.

Would be nicer to grant access by stages.

Re: Ask HN: How was your experience with AWS Lambda in production?

#122

Post by philliphaydon »

Prefinem wrote: post #107
chickenbane wrote: post #98

I worked on a project where the architect wanted to use Lambdas for the entire solution. This was a bad choice. Lambdas have a lot of benefits - for occasional tasks they are essentially free, the simple programming model makes them easy to understand in teams, you get Amazon's scaling and there's decent integration with caching and logging. However, especially since I had to use them for whole solution, I ran into a…

1. Use a framework for deploying your lambda functions. There are a few that will manage the API Gateway for you. 2. Don't put the Lambda inside a VPC if you want lower response times 3. Step Functions don't seem ready for prime time that I can tell. (This might have changed in the last couple of months) 4. Lambda Functions should be microservices. Small and lean. 5. There is a limit on resources for CloudFormation s…

Don't put the Lambda inside a VPC if you want lower response times

Response time's are fine inside a VPC... People keep saying gateway is slow but it isn't from my experience...

Re: Ask HN: How was your experience with AWS Lambda in production?

#123

Post by Prefinem »

philliphaydon wrote: post #122
Prefinem wrote: post #107

Earlier quoted context omitted.

1. Use a framework for deploying your lambda functions. There are a few that will manage the API Gateway for you. 2. Don't put the Lambda inside a VPC if you want lower response times 3. Step Functions don't seem ready for prime time that I can tell. (This might have changed in the last couple of months) 4. Lambda Functions should be microservices. Small and lean. 5. There is a limit on resources for CloudFormation s…

Don't put the Lambda inside a VPC if you want lower response times Response time's are fine inside a VPC... People keep saying gateway is slow but it isn't from my experience...

API Gateway adds around 200ms to each call for us based on our testing plus another 100ms if it’s inside a VPC. What region are you in? Our VPC has a NAT Gateway so that our Lambda functions can talk to the internet as well.

I am by no means an expert and am reporting what watching the time differences between Lambda reporting and our network calls shows.

Edit: this is by no means a deal breaker or anything. Just something that shocked me when I first noticed it.

Re: Ask HN: How was your experience with AWS Lambda in production?

#124

Post by zurn »

For a serverless system that uses Lambda together with eg CloudFormation, Dynamo, and S3, Cognito etc - it's pretty low level and you spend a lot of time understanding, refining & debugging basic things. The end-to-end logging and instrumentation throughout the services used by your app weren't great.

Doesn't like big app binaries/JARs and Amazon's API client libs are bloated - Clojure + Amazonica goes easily over the limit if you don't manually exclude some Amazon's API JDKs from the package.

On the plus side, you can test all the APIs from your dev box using the cli or boto3 before doing it from the lambda.

Would probably look into third party things like Serverless next time.

Re: Ask HN: How was your experience with AWS Lambda in production?

#125

Post by meekins »

We're doing both stream processing and small query APIs using Lambda.

A few pointers (from relatively short experience):

- The best UC for Lambda seems to be stream processing where latency due to start up times is not an issue

- For user/application-facing logic the major issue seems to be start-up-times (esp. JVM startup times when doing Java or your API gets called very rarely) and API Gateway configuration management using infrastructure as code tools (I'd be interested in good hints about this, especially concerning interface changes)

- The programming model is very simple and nice but it seems to make most sense to split each API over multiple lambdas to keep them as small as possible or use some serverless framework to make managing the whole app more easy

- This goes without saying, but be sure to use CI and do not deploy local builds (native binary deps)

Re: Ask HN: How was your experience with AWS Lambda in production?

#126

Post by philliphaydon »

Prefinem wrote: post #123
philliphaydon wrote: post #122

Earlier quoted context omitted.

Don't put the Lambda inside a VPC if you want lower response times Response time's are fine inside a VPC... People keep saying gateway is slow but it isn't from my experience...

API Gateway adds around 200ms to each call for us based on our testing plus another 100ms if it’s inside a VPC. What region are you in? Our VPC has a NAT Gateway so that our Lambda functions can talk to the internet as well. I am by no means an expert and am reporting what watching the time differences between Lambda reporting and our network calls shows. Edit: this is by no means a deal breaker or anything. Just som…

(crap I just typed stuff out and pressed F5...)

I haven't tried with a NAT Gateway.

This is what I know.

If the response is small, the request duration is small.

Cloudfront > Gateway > Lambda > RDS (PostgreSQL)

Response:

1k | 20ms-60ms

10k | 50ms-90ms

100k | 150ms-200ms

350k | 200ms-450ms

That's a rough gauge of what I've experienced.

I think the throughput on the gateway is the bottleneck.

Re: Ask HN: How was your experience with AWS Lambda in production?

#127

Post by ec109685 »

Prefinem wrote: post #123
philliphaydon wrote: post #122

Earlier quoted context omitted.

Don't put the Lambda inside a VPC if you want lower response times Response time's are fine inside a VPC... People keep saying gateway is slow but it isn't from my experience...

API Gateway adds around 200ms to each call for us based on our testing plus another 100ms if it’s inside a VPC. What region are you in? Our VPC has a NAT Gateway so that our Lambda functions can talk to the internet as well. I am by no means an expert and am reporting what watching the time differences between Lambda reporting and our network calls shows. Edit: this is by no means a deal breaker or anything. Just som…

Are you paying for ssl round trips in your tests? We haven't seen such slow response times with lambda or the gateway.

Re: Ask HN: How was your experience with AWS Lambda in production?

#128

Post by tracker1 »

CharlesW wrote: post #97
tracker1 wrote: post #8

Best to keep your workloads as small as possible, cold starts can be very bad, depending on the type of project. Been using mostly node myself, and it's worked out well. One thing to be careful of, if you're targeting input into dynamodb table(s), then it's really easy to flood your writes. Same goes for SQS writes. You might be better off with a data pipeline, and slower progress. It really just depends on your use…

> Best to keep your workloads as small as possible, cold starts can be very bad, depending on the type of project. Here's a recent, interesting article on the topic that quantifies some of this: https://read.acloud.guru/does-coding-language-memory-or-pack...

Thanks for the link... interesting how quick python is to start...

Re: Ask HN: How was your experience with AWS Lambda in production?

#129

Post by AlexanderC89 »

CSDude wrote: post #2

- Monitoring & debugging is little hard - CPU power also scales with Memory, you might need to increase it to get better responses - Ability to attach many streams (Kinesis, Dynamo) is very helpful, and it scales easily without explicitly managing servers - There can be a overhead, your function gets paused (if no data incoming) or can be killed undeterministically (even if it works all the time or per hour) and caus…

@CSDude you can check out this small tool i've written for debugging Lambda and API Gateway integration.

https://github.com/AlexanderC/lambdon (i know the name sucks)

Also @chetanmelkani as a hint: if you are using NodeJS runtime most optimal from the execution time and cost efficiency perspective is setting up 512mb of memory ;) it's about getting x2 performance boost over the 128mb configuration.

Re: Ask HN: How was your experience with AWS Lambda in production?

#130

Post by str3tch76 »

runamok wrote: post #109
munns wrote: post #72

Hey all, My name is Chris Munns and I am currently the lead Developer Advocate for Serverless at AWS (I am part of the Lambda PM team). We really appreciate this feedback and are always looking for ways to hear about these pain points. Can email me directly: munns@amazon.com if you ever get stuck. Thanks, - Chris

I am on a devops team trying to implement lambda by splitting pieces off from a monolithic Java app where applicable. The biggest pain point I have is because we have multiple "environments" (such as dev, da, staging) in the same amazon account and because lambdas are global I can't limit access to resources via IAM easily without hacks. Aka, because the same lambda will be used (but different versions and/or aliases…

check out the Serverless Framework (www.serverless.com)

This will take all this pain awawy from you. It makes it very easy to push to different stages (dev, uat, prod etc)

Post reply on HN