Ask HN: How was your experience with AWS Lambda in production? • Hacker News Board

Re: Ask HN: How was your experience with AWS Lambda in production?

#111

Post by mfrye0 » Wed, Jun 21, 2017, 5:45 PM UTC

I started off doing manual build / deploy for a project and it was a total pain in the ass. From packaging the code, to versioning, rollbacks, deploy. Then that doesn't even include setting up API Gateway if you want an endpoint for the function.

Since then I've been using Serverless for all my projects and it's the best thing I've tried thus far. It's not perfect, but now I'm able to abstract everything away as you configure pretty much everything from a .yml file.

With that said, there are still some rough spots with Lambda:

1) Working with env vars. Default is to store them in plain text in the Lambda config. Fine for basic stuff, but I didn't want that for DB creds. You can store them encrypted, but then you have to setup logic to decrypt in the function. Kind of a pain.

2) Working within a subnet to access private resources incurs an extra delay. There is already a cold start time for Lambda functions, but to access the subnet adds more time... Apparently AWS is aware and is exploring a fix.

3) Monitoring could be better. Cloudwatch is not the most user friendly tool for trying to find something specific.

With that said, as a whole Lambda is pretty awesome. We don't have to worry about setting up ec2 instances, load balancing, auto scaling, etc for a new api. We can just focus on the logic and we're able to roll out new stuff so much faster. Then our costs are pretty much nothing.

Re: Ask HN: How was your experience with AWS Lambda in production?

#112

Post by z3t4 » Wed, Jun 21, 2017, 6:01 PM UTC

callumlocke wrote: post #21

I made an image hosting tool on Lambda and S3, for internal corporate use. Staff can upload images to S3 via an SPA. The front end contacts the Lambda service to request a pre-signed S3 upload URL, so the browser can upload directly to S3. It works really well. Observations: 1. Took too long to get something working. The common use case of hooking up a Lambda function to an HTTP endpoint is surprisingly fiddly and ma…

how do you make sure it will run forever ? what about breaking changes and discontinued services ? can you move to another vendor without changes to your codebase ?

Re: Ask HN: How was your experience with AWS Lambda in production?

#113

Post by Prefinem » Wed, Jun 21, 2017, 6:10 PM UTC

callumlocke wrote: post #21

I made an image hosting tool on Lambda and S3, for internal corporate use. Staff can upload images to S3 via an SPA. The front end contacts the Lambda service to request a pre-signed S3 upload URL, so the browser can upload directly to S3. It works really well. Observations: 1. Took too long to get something working. The common use case of hooking up a Lambda function to an HTTP endpoint is surprisingly fiddly and ma…

1. I think a framework / tool will ensure this isn't an issue

2. Dear heavens yes. I ended up building a wrapper (similar to what you did to address 3) that handles logging, etc for any internal events. Everything else is a pass / fail check

3. Also had to build a wrapper. Context / Callback params are confusing

4. I wouldn't use Serverless unless you need it. Try something smaller. Apex is a nice, simple start. Shameless plug: I built a deployment tool because serverless wouldn't work for us and I wanted something in node (no binary like Apex - integrate into our build process) [1]

[1]: https://github.com/Prefinem/lambdify

Re: Ask HN: How was your experience with AWS Lambda in production?

#114

Post by adjohn » Wed, Jun 21, 2017, 8:12 PM UTC

kesor wrote: post #83

callumlocke wrote: post #21

I made an image hosting tool on Lambda and S3, for internal corporate use. Staff can upload images to S3 via an SPA. The front end contacts the Lambda service to request a pre-signed S3 upload URL, so the browser can upload directly to S3. It works really well. Observations: 1. Took too long to get something working. The common use case of hooking up a Lambda function to an HTTP endpoint is surprisingly fiddly and ma…

The problem of logs is actually a problem of CloudWatch Logs being just not a very good service. A great way to solve that is to push all logs from CloudWatch Logs into an ElasticSearch cluster (using a Lambda function). AWS even has the code already done for you if you click the "subscribe" button in CWL. Then with Kibana/ElasticSearch the experience of inspecting and analysing logs is MUCH better.

We built IOpipe[1] to address these issues by offering our own wrapper[2] that sends telemetry to our service. IOpipe aggregates metrics, and errors, and allows the creation of alerts with multiple rules per alert.

[1] - https://iopipe.com [2] - https://github.com/iopipe/iopipe/

Re: Ask HN: How was your experience with AWS Lambda in production?

#115

Post by dlanger » Wed, Jun 21, 2017, 9:03 PM UTC

Hey everyone, I'm Daniel Langer and I help build lambda monitoring products over at Datadog. I see lots of you are unhappy with the current monitoring solutions available to you. If anyone has thoughts on what they'd like in a Lambda monitoring service feel free to email me at daniel.langer@datadoghq.com

Re: Ask HN: How was your experience with AWS Lambda in production?

#116

Post by aeikenberry » Wed, Jun 21, 2017, 10:14 PM UTC

We run many microservices on Lambda and it has been a pleasant experience for us. We use Terraform for creating, managing environment variables, and permissions/log groups/etc. We use CodeShip for testing, and validating and applying Terraform across multiple accounts and environments.

For logging, we pipe all of our logs out of CloudWatch to LogEntries with a custom Lambda, although looking at CloudWatch logs works fine most of the time.

Re: Ask HN: How was your experience with AWS Lambda in production?

#117

Post by lindydonna » Wed, Jun 21, 2017, 11:08 PM UTC

kesor wrote: post #83

callumlocke wrote: post #21

I made an image hosting tool on Lambda and S3, for internal corporate use. Staff can upload images to S3 via an SPA. The front end contacts the Lambda service to request a pre-signed S3 upload URL, so the browser can upload directly to S3. It works really well. Observations: 1. Took too long to get something working. The common use case of hooking up a Lambda function to an HTTP endpoint is surprisingly fiddly and ma…

The problem of logs is actually a problem of CloudWatch Logs being just not a very good service. A great way to solve that is to push all logs from CloudWatch Logs into an ElasticSearch cluster (using a Lambda function). AWS even has the code already done for you if you click the "subscribe" button in CWL. Then with Kibana/ElasticSearch the experience of inspecting and analysing logs is MUCH better.

Have you tried out Azure Functions? It has pretty good features for continuous integration, a CLI where you can run locally, and a really nice monitoring experience. Obviously it's probably not worth migrating an existing project, but it might be useful for future projects. Plus, we have a Serverless Framework plugin.

Here's a talk that walks through some of our features: https://www.youtube.com/watch?v=TgB-fs1hwlw&t=18s

Disclosure: I'm a Program Manager on Azure Functions.

Re: Ask HN: How was your experience with AWS Lambda in production?

#118

Post by mikemann7121 » Wed, Jun 21, 2017, 11:28 PM UTC

callumlocke wrote: post #21

I made an image hosting tool on Lambda and S3, for internal corporate use. Staff can upload images to S3 via an SPA. The front end contacts the Lambda service to request a pre-signed S3 upload URL, so the browser can upload directly to S3. It works really well. Observations: 1. Took too long to get something working. The common use case of hooking up a Lambda function to an HTTP endpoint is surprisingly fiddly and ma…

Hi,

Most people failed to make money online because they don't have a proven system that works for them...

Good news is, this system has helped more than 100 ordinary people, including newbies to make real money.

You can be one of them too:

http://rurl.us/v1GxM

The best part is...

-You don't need to write or produce any unique content

-You don't need to create any products to sell

-You don't need to write up any sales letters

-You don't need to provide any customer service or follow up support

-You don't even need to learn about internet marketing!

See how this can be done with real proof here:

http://rurl.us/v1GxM

Don't be left behind.

Re: Ask HN: How was your experience with AWS Lambda in production?

#119

Post by milesvp » Thu, Jun 22, 2017, 12:23 AM UTC

jamroom wrote: post #87

kesor wrote: post #83

Earlier quoted context omitted.

The problem of logs is actually a problem of CloudWatch Logs being just not a very good service. A great way to solve that is to push all logs from CloudWatch Logs into an ElasticSearch cluster (using a Lambda function). AWS even has the code already done for you if you click the "subscribe" button in CWL. Then with Kibana/ElasticSearch the experience of inspecting and analysing logs is MUCH better.

This is what drives me nuts about AWS, but is genius on their part - they've got you to spend more $$ on ElasticSearch and Lambda to overcome the fact that one of their other products just doesn't work very well.

I don't think this is intentional. My experience with AWS is that they aim to constantly improve products and create hosted services that are cheaper than if you frankensteined the same thing yourself. They don't always succeed, but I'm willing to give them the benefit of the doubt for how bad cloudwatch truly is, and just assume either they're blind to the pain because they know what not to do internally, or they're just really hamstrung in trying to modify that feature since so much relies on it.

Hopefully this isn't just stockholm syndrome speaking though...

Re: Ask HN: How was your experience with AWS Lambda in production?

#120

Post by thenewwazoo » Thu, Jun 22, 2017, 12:49 AM UTC

lindydonna wrote: post #117

kesor wrote: post #83

Earlier quoted context omitted.

The problem of logs is actually a problem of CloudWatch Logs being just not a very good service. A great way to solve that is to push all logs from CloudWatch Logs into an ElasticSearch cluster (using a Lambda function). AWS even has the code already done for you if you click the "subscribe" button in CWL. Then with Kibana/ElasticSearch the experience of inspecting and analysing logs is MUCH better.

Have you tried out Azure Functions? It has pretty good features for continuous integration, a CLI where you can run locally, and a really nice monitoring experience. Obviously it's probably not worth migrating an existing project, but it might be useful for future projects. Plus, we have a Serverless Framework plugin. Here's a talk that walks through some of our features: https://www.youtube.com/watch?v=TgB-fs1hwlw&t…

I used Azure Functions (when I was a MSFT employee, in fact), and found it to be unusable. A list of complaints:

* setup is 100% completely clicky-clicky UI driven, which was a huge pain to scale. instantiation of a Function on behalf of a developer for production use was a huge time sink

* it's clearly a thin veneer on Azure Web Services, and the abstractions leak badly in the portal (deployment credentials, for example)

* the web UI breaks completely and mysteriously if you enable authentication

* management of service princi- uh, I mean, Azure AD Applications was weird, and the (internal to MSFT, I suspect) permissions model to the Graph API was a huge barrier to ease of use

* management of NPM packages required me to start a terminal session in the UI and run commands manually, which was a huge turnoff (and had to be repeated ad nauseam with every new Function created)

* the configuration files for the runtime are utterly undocumented, with the sole exception of the bits used to plug Azure inputs/outputs together. this makes automating things exceedingly difficult. I recall there even being a magic value in the topmost config file

* the edit-commit-push-test cycle was VERY slow, with new commits sometimes taking tens of minutes to "appear" in my function

* I never found a way to run it locally, making the previous point that much worse

* log output is very difficult to find, and can live in a few different places. I spent too much time hunting for errors, especially things like syntax errors that make the runtime itself go kaboom. This was the thing that really killed it for me; if I had an error that resulted in anything but a "clean" return, it was torture trying to figure out where I'd missed the paren.