Live data from Hacker News

Ask HN: How was your experience with AWS Lambda in production?

news.ycombinator.com

21–30 of 165 posts

Re: Ask HN: How was your experience with AWS Lambda in production?

#21
I made an image hosting tool on Lambda and S3, for internal corporate use. Staff can upload images to S3 via an SPA. The front end contacts the Lambda service to request a pre-signed S3 upload URL, so the browser can upload directly to S3. It works really well. Observations:

1. Took too long to get something working. The common use case of hooking up a Lambda function to an HTTP endpoint is surprisingly fiddly and manual.

2. Very painful logging/monitoring.

3. The Node.js version of Lambda has a weird and ugly API that feels like it was designed by a comittee with little knowledge of Node.js idioms.

4. The Serverless framework produces a huge bundle unless you spend a lot of effort optimising it. It's also very slow to deploy incremental changes edit: – this is not only due to the large bundle size but also due to having to re-up the whole generated CloudFormation stack for most updates.

5. It was worth it in the end for making a useful little service that will exist forever with ultra-low running costs, but the developer experience could have been miles better, and I wouldn't want to have to work on that codebase again.

---

Edit: here's the code: https://github.com/Financial-Times/ig-images-backend

To address point 3 above, I wrote a wrapper function (in src/index.js) so I could write each HTTP Lambda endpoint as a straight async function that simply receives a single argument (the request event) and asynchronously returns the complete HTTP response. This wouldn't be good if you were returning a large response though; you'd probably be better streaming it.

Re: Ask HN: How was your experience with AWS Lambda in production?

#22
Have used it in production for > 2 years, mainly for ETL/Data processing type jobs which seems to work well.

We also use it to perform scheduled tasks (e.g. every hour) which is good as it means you don't have to have an EC2 instance just to run cron like jobs.

The main downside is Cloudwatch Logs, if you have a Lambda that runs very frequently (i.e. 100,000+ invocations a day) the logs become painful to search through, you have to end up exporting them to S3 or ElasticSearch.

Re: Ask HN: How was your experience with AWS Lambda in production?

#23
I've been using it for heavy background job for http://thefeed.press and overall, I think it's pretty ok (I use NodeJs). That said here are few things:

- No straight way to prevent retries. (Retries can crazily increase your bill if something goes wrong)

- API gateway to Lambda can be better. (For one, Multipart form-data support for API gateway is a mess)

- (For NodeJs) I don't see why the node_modules folder should be uploaded. (Google cloud functions downloads the modules from the package.json)

Re: Ask HN: How was your experience with AWS Lambda in production?

#24
post #19

For running Java in Lambda, I had to optimize it for Lambda. To decrease processing time (and in the end the bill), I got rid of all reflection for example and though twice when to initialize what and what to make static. Also, Java Cold Start is an issue. I fixed this with creating a Cloudwatch Trigger that executes the Lambda function every minute to keep it hot. Otherwise, after some minutes of no-one calling the…

The polling trigger is a fairly common pattern from what I've seen to keep it hot.

Re: Ask HN: How was your experience with AWS Lambda in production?

#26
post #19

For running Java in Lambda, I had to optimize it for Lambda. To decrease processing time (and in the end the bill), I got rid of all reflection for example and though twice when to initialize what and what to make static. Also, Java Cold Start is an issue. I fixed this with creating a Cloudwatch Trigger that executes the Lambda function every minute to keep it hot. Otherwise, after some minutes of no-one calling the…

Seems like a lot of hustle to me...

Re: Ask HN: How was your experience with AWS Lambda in production?

#27

- There is a surprisingly high amount of API gateway latency - The CPU power available seems to be really weak. Simple loops running in NodeJS run way way slower on Lambda compared to a 1.1 GHz Macbook by a significant magnitude. This is despite scaling the memory up to near 512mb. - Certain elements, such as DNS lookups, take a very long time. - The CloudWatch logging is a bit frustrating. If you have a cron job it…

RDS always running was just recently fixed...

https://aws.amazon.com/about-aws/whats-new/2017/06/amazon-rd...

Re: Ask HN: How was your experience with AWS Lambda in production?

#28
post #23

I've been using it for heavy background job for http://thefeed.press and overall, I think it's pretty ok (I use NodeJs). That said here are few things: - No straight way to prevent retries. (Retries can crazily increase your bill if something goes wrong) - API gateway to Lambda can be better. (For one, Multipart form-data support for API gateway is a mess) - (For NodeJs) I don't see why the node_modules folder should…

Agreed regarding downloading of libraries. I tried to get a python torrent library working in lambda the other day, and I had to manually dig through my /usr/lib to find the right shared objects. Should be much easier than that, I should be able to place a language appropriate set of requirements in the zip root and it should do what it needs to do.

Re: Ask HN: How was your experience with AWS Lambda in production?

#29
post #23

I've been using it for heavy background job for http://thefeed.press and overall, I think it's pretty ok (I use NodeJs). That said here are few things: - No straight way to prevent retries. (Retries can crazily increase your bill if something goes wrong) - API gateway to Lambda can be better. (For one, Multipart form-data support for API gateway is a mess) - (For NodeJs) I don't see why the node_modules folder should…

> I don't see why the node_modules folder should be uploaded.

Exactly! Especially if you're using modules that include some sort of binary and build your function on macOS it's a pain -- I ended up using a Docker-based workflow to get the correct binaries into the node_modules.

Re: Ask HN: How was your experience with AWS Lambda in production?

#30
post #23

I've been using it for heavy background job for http://thefeed.press and overall, I think it's pretty ok (I use NodeJs). That said here are few things: - No straight way to prevent retries. (Retries can crazily increase your bill if something goes wrong) - API gateway to Lambda can be better. (For one, Multipart form-data support for API gateway is a mess) - (For NodeJs) I don't see why the node_modules folder should…

What do you mean by retries? Also, why would you upload node_modules? Why don't you build it first and only upload what is required...
Post reply on HN