Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Lambda Cold Start Times

filia-aleks.medium.com

Post reply on HN
Print view View original

71–80 of 233 posts

Re: AWS Lambda Cold Start Times

#71

Post by danielvaughn »

cmcconomy wrote: post #32

My experience with cold starts in Azure Functions Serverless is pretty awful. Like most other Azure services, their affordable consumer grade offerings are designed from the ground up not to be good enough for "serious" use. Cold start times compared to Lambda are worse, and in addition, we would get random 404s which do not appear in any logs; inspecting these 404s indicated they were emitted by nginx, leading me to…

Same experience with Firebase. I just joined a team that has been using it. I've never worked with serverless before, and it boggles my mind how anyone thought it would be a good idea.

The cold starts are horrendous. In one case, it's consistently taking about 7 seconds to return ~10K of data. I investigated the actual runtime of the function and it completes in about 20ms, so the only real bottleneck is the fucking cold start.

Re: AWS Lambda Cold Start Times

#72

Post by roland35 »

berkes wrote: post #30
VWWHFSfQ wrote: post #15

Earlier quoted context omitted.

Provisioned concurrency is insanely expensive. If you have any kind of a thundering herd access pattern then Lambda is a complete non-starter because of the warm-up and scaling characteristics. We eventually just put an nginx/openresty server on a regular medium EC2 instance and got rid of Lambda from our stack completely and now we're paying about 1/300th the cost we were previously and the performance is infinitely…

> I'm sure it has some use-cases in some kind of backoffice task queue scenario, but Lambda is nearly unusable in a web context unless you have a very trivial amount of traffic. This has been the outcome for me on several projects too. Just use loadbalanced EC2 (or EB, for simplification) and pay for a few instances running 24/7. It's actually cheaper than having a busy lambda in all my cases. The only other case (ot…

Have you run into issues with Lambda with complex tasks? I thought there was a 15 minute limit to tasks, plus a maximum storage size when importing large dependencies, etc?

Re: AWS Lambda Cold Start Times

#73

Post by [deleted] »

_fat_santa wrote: post #4

At my last job we built an entire API on top of serverless. One of the things we had to figure out was this cold start time. If a user were to hit an endpoint for the first time, it would take 2x as long as it normally would at first. To combat this we wrote a "runWarm" function that kept the API alive at all times. Sure kind of defeats the purpose of serverless but hey, enterprise software.

[deleted]

Re: AWS Lambda Cold Start Times

#74

Post by sudhirj »

wongarsu wrote: post #67
d0gsg0w00f wrote: post #55

Earlier quoted context omitted.

In my experience, if you can't say "get over it" to your customers when they complain about performance then Lambda is not the right tool. Just use EC2.

It's an excellent product for glue code between the various AWS services. Just about every AWS product can trigger Lambda functions, so if you want to run image recognition whenever a new image is uploaded to S3, Lambda is the way to do that. They also make great cron jobs. But for some reason Amazon likes to sell it as a way to run any web application backend, as if that was a good use case.

It can be. We run dynamic image resizing (have a couple million image high quality originals in S3, and customers request sizes based on their screen). Each request is handled by a lambda, and even though these are memory intensive operations we never need to worry about servers or running out or RAM or circuit breakers or anything. Whatever the load it just works. The actual operations take on the order of 100ms, so the cold start is negligible to us. And the end product is cached on a CDN anyway. Costs less than one m5.large, but at peak loads it does work 100 times what's possible on the m5.large.

Say you open a page with a 100 images on it for example. With lambda the all images are resized for you in parallel, so total 100ms. If this was servers, would have to run 100 servers to give you the same performance. A single servers could resize images in sequence all day and might be cheaper than running a lambda repeatedly all day, but that's not the requirement. The requirement is to suddenly do 100 things in parallel within 100ms just once when you open the app.

Re: AWS Lambda Cold Start Times

#75

Post by earthboundkid »

danielvaughn wrote: post #71
cmcconomy wrote: post #32

My experience with cold starts in Azure Functions Serverless is pretty awful. Like most other Azure services, their affordable consumer grade offerings are designed from the ground up not to be good enough for "serious" use. Cold start times compared to Lambda are worse, and in addition, we would get random 404s which do not appear in any logs; inspecting these 404s indicated they were emitted by nginx, leading me to…

Same experience with Firebase. I just joined a team that has been using it. I've never worked with serverless before, and it boggles my mind how anyone thought it would be a good idea. The cold starts are horrendous. In one case, it's consistently taking about 7 seconds to return ~10K of data. I investigated the actual runtime of the function and it completes in about 20ms, so the only real bottleneck is the fucking…

7s cold start is weird. I think there has to be more to the story than that.

Re: AWS Lambda Cold Start Times

#76

Post by wmfiv »

VWWHFSfQ wrote: post #15
jpdb wrote: post #8

Earlier quoted context omitted.

> To combat this we wrote a "runWarm" function that kept the API alive at all times. This doesn't really work like you'd expect and isn't recommended, as it only helps a particular use-case. The reason is that AWS Lambda will only keep a single instance of your function alive. That means if two requests come in at the same time, you'd see a cold start on one of those invocations. Instead, you want to look at somethin…

Provisioned concurrency is insanely expensive. If you have any kind of a thundering herd access pattern then Lambda is a complete non-starter because of the warm-up and scaling characteristics. We eventually just put an nginx/openresty server on a regular medium EC2 instance and got rid of Lambda from our stack completely and now we're paying about 1/300th the cost we were previously and the performance is infinitely…

Insanely expensive is definitely a flexible term. I think numbers help here.

Provisions Concurrency $8.64 / GB / month

256 MB per Lambda (Assuming Python, Ruby, NodeJS, or Rust)

$2.16 per Lambda per month

A lot of organizations can probably make a good business case for keeping 100s or even 1000s of Lambda's warm. You also don't need to keep them warm 24x7, can get an additional 12% discount using savings plans, and if you're a big guy you get your EDP discount.

Re: AWS Lambda Cold Start Times

#77

Post by nuerow »

_fat_santa wrote: post #4

At my last job we built an entire API on top of serverless. One of the things we had to figure out was this cold start time. If a user were to hit an endpoint for the first time, it would take 2x as long as it normally would at first. To combat this we wrote a "runWarm" function that kept the API alive at all times. Sure kind of defeats the purpose of serverless but hey, enterprise software.

> To combat this we wrote a "runWarm" function that kept the API alive at all times.

This sort of hack is not needed in AWS Lambdas, as they support provisioned concurrency.

Re: AWS Lambda Cold Start Times

#78

Post by gokhan »

moonchrome wrote: post #13
jabart wrote: post #5

We run a few .net core lambdas and a few things that make a big difference for latency. 1. pre-jit the package, this reduces cold start times as the JIT doesn't need to run on most items. Still does later to optimize some items. 2 is sticking to the new .net json seralizer. The reference code uses both the new and old newtsonsoft package. The old package has higher memory allocations as it doesn't make use of the Spa…

Great tip for running the JIT AOT, for anyone interested Microsoft calls this "ReadyToRun" compilation [1]. [1] https://docs.microsoft.com/en-us/dotnet/core/deploying/ready... I wonder did you test if the increased size results in an actual win for the startup time ?

The test code is quite small and might not benefit from R2R that much, libs it relies on are already jitted. Ditching Newtonsoft would affect response time though.

Re: AWS Lambda Cold Start Times

#79

Post by dharmab »

secondaryacct wrote: post #46
cmcconomy wrote: post #32

My experience with cold starts in Azure Functions Serverless is pretty awful. Like most other Azure services, their affordable consumer grade offerings are designed from the ground up not to be good enough for "serious" use. Cold start times compared to Lambda are worse, and in addition, we would get random 404s which do not appear in any logs; inspecting these 404s indicated they were emitted by nginx, leading me to…

My company is moving away from datacenter and into Azure and I have to get the az900 this week, it doesnt bode well. And I was so happy to leave the clusterfuck of 300 aws lambda I was working with in my prev company. What an expensive fad, and no engineer is ever consulted ...

Never use Basic SKU, plan your network carefully before you even create the vnets, monitor NAT capacity, beware undocumented ARM rate limits. Good luck

Re: AWS Lambda Cold Start Times

#80

Post by earthboundkid »

ctvo wrote: post #58
VWWHFSfQ wrote: post #15

Earlier quoted context omitted.

Provisioned concurrency is insanely expensive. If you have any kind of a thundering herd access pattern then Lambda is a complete non-starter because of the warm-up and scaling characteristics. We eventually just put an nginx/openresty server on a regular medium EC2 instance and got rid of Lambda from our stack completely and now we're paying about 1/300th the cost we were previously and the performance is infinitely…

This is another example of AWS over marketing Lambda. Lambda is horrendously expensive when requests pass a certain level per second. You can graph it against ECS / EC2 to see the point it stops becoming economical. Taking all of this into account, Lambda is then useful for a very small niche: - Tasks that don't care about low P99 latency. These tend to be asynchronous processing workflows, as APIs in the customer re…

I use Netlify serverless functions (which is just a wrapper around AWS Lambda) because it basically fits the criteria for me. I have a low but bursty access pattern that fits into the free tier, and there's a static SPA page that can serve up instantly while the XHR triggers to do the cold start fetch. I don't think I would use it for anything consumer facing though. This is just a backend where an extra 300ms isn't going to make a big difference to the admins.
Post reply on HN