Live data from Hacker News

Comparing Serverless Performance for CPU Bound Tasks

blog.cloudflare.com

41–50 of 68 posts

Re: Comparing Serverless Performance for CPU Bound Tasks

#41
post #2

I'd love to see an honest comparison across other providers, throwing in Google's Firebase Functions and Azure Cloud Functions.

Here's a good comparison of Lambda vs Azure Functions performance and scaling up to 400 concurrent requests: https://www.azurefromthetrenches.com/azure-functions-vs-aws-...

"Since I published this piece Microsoft have made significant improvements to HTTP scaling on Azure Functions and the below is out of date. Please see this post for a revised comparison. https://www.azurefromthetrenches.com/azure-functions-signifi...

Re: Comparing Serverless Performance for CPU Bound Tasks

#42
post #31

Each and every one of these posts from Cloudflare is direct targeting and completely biased. Should be noted that their response times are indicative of them being far fewer hops away. Unless they can run in the same DC, or even make the RTT fair, their 'Webpage response' metric is utterly useless. Notice how they admit that they don't know how lambda really works. They switch between lambda@edge and Region-based lam…

Hi, I'm the author of the post, thanks for sharing your thoughts! You're absolutely right we don't know how Lambda works. We have read what we could find that's publicly available, and done a bunch of testing, but Amazon doesn't share all that much about their architecture. I agree that the cold-start times of Lambda are slow, particularly with languages like Java and with VPCs. My plan at the moment is to write a bl…

Thanks for being forthcoming; and I do appreciate the post. I'm partially venting, and I've been using AWS for a while. Vendor lock-in can be a pain in the rear sometimes.

A lot of the gripe I've got with these posts is that they seem somewhat incomplete. Hard to say things if you've only been messing with it for a week, and don't have a lot of the ins-and-outs of the services. That being said, I've spent at least 2 years diving into lambda and weird issues with it.

I'm not an employee of Amazon, and so my understanding can be off-base as well.

Lambdas are just managed EC2 instances. Each lambda code is stored in an AWS controlled S3 bucket; and on initial execution (cold start) pulled down, and run in their own chroot jail. I've dealt with java lambdas the most, and I can say that they take your zip, and run the jail inside the zip. They keep your java process open, and just call the handler on each call (warm start.) Each concurrent call can start another jail on another managed instance; getting the cold start time again.

You can get cold starts by uploading a new zip, or changing any of the lambda compute parameters.

The Golang works in a similar way, jails the zip, and keeps the program running, calling the handler as invokes come in.

I haven't done the python, node, or .NET enough to know if those are the same principles; I'd assume they are.

Interestingly, API Gateway really is just Cloudfront. Cloudfront is just the AWS managed API Gateway.

Re: Comparing Serverless Performance for CPU Bound Tasks

#43
post #17
post #4

Assuming the author's tests are single-thread, I'm pretty sure 1024 MB doesn't give you a full CPU core on Lambda. I could be wrong though; I haven't payed attention to Lambda in a long time. Last I remember it was 1.5 GB that gave you a full core. This alone makes the comparison between a mid-range server and Lambda unfair, not to mention the differences between language runtimes. That said, if you are using Lambda…

Anyone using Lambda should _absolutely_ do load testing with different memory configurations. You will get different results, and should analyze what is best for your application. When calculating the overall cost of managing your own instances you should also include time spent by your engineering team. There are particular tipping points in terms of overall requests per second at which point you'd save money by mov…

> And then even above that, you should be running your own instances in a colo space

Why are people jumping from EC2 to colo and skipping dedicated servers? Mystery of my life. We were running the 75th largest site in the US some years ago (as measured by Quantcast), ran the numbers and colo was ridiculously expensive and way more troublesome.

Re: Comparing Serverless Performance for CPU Bound Tasks

#44
post #43
post #17

Earlier quoted context omitted.

Anyone using Lambda should _absolutely_ do load testing with different memory configurations. You will get different results, and should analyze what is best for your application. When calculating the overall cost of managing your own instances you should also include time spent by your engineering team. There are particular tipping points in terms of overall requests per second at which point you'd save money by mov…

> And then even above that, you should be running your own instances in a colo space Why are people jumping from EC2 to colo and skipping dedicated servers? Mystery of my life. We were running the 75th largest site in the US some years ago (as measured by Quantcast), ran the numbers and colo was ridiculously expensive and way more troublesome.

Oh, I was defining "EC2" as everything from small shared virtual boxes to EC2 bare metal. I suppose on-site dedicated servers is definitely a point I left out of my hypothetical list though! I used to work at an unnamed company that had a server rack plugged in between the kitchen and the ping pong table. Though I never witnessed such a thing, I'd be shocked if there was never an outage due to a particularly enthusiastic game.

Re: Comparing Serverless Performance for CPU Bound Tasks

#45

Earlier quoted context omitted.

Technically CloudFlare workers is better compared with Lambda @ Edge no? I don't think you'd be using API Gateway, instead CloudFront right?

Originally we were thinking about it in those terms (as an alternative to Lambda@Edge), but based on our recent results I am happy to have people compare it to Lambda as well. Lambda with API Gateway is at least eight times more expensive than Workers, and only runs your code in one location instead of 151. Unless you're using Lambda to do something ultra-Amazon-specific (like process S3 changes), I don't see why it…

Gotcha. Thanks for explaining!

Re: Comparing Serverless Performance for CPU Bound Tasks

#46
post #3

I've recently been exploring AWS Lambda in a stack which contained API Gateway + Python Flask under Lambda for a task I was working on. I deployed it using Zappa and its purpose was to be a simple REST frontend for transferring files to S3. After experimenting with uploads from Lambda to S3 I was noticing that the time to upload a tiny 4MB file changed dramatically when I reconfigured the Lambda function's memory siz…

This is why people using Lambda at scale are concerned with keeping the containers “warm” https://aws.amazon.com/blogs/compute/container-reuse-in-lamb...

Keep in mind that if you have any kind of fanout at scale, keeping a few lambda instances “warm” probably won’t improve your throughout much.

Update: found a nice article with metrics re: lambda-backed api gateway but the premise applies to any fan-out.

https://hackernoon.com/im-afraid-you-re-thinking-about-aws-l...

Re: Comparing Serverless Performance for CPU Bound Tasks

#47
post #5

I remember a few years ago we tried to implement a scheduled Lambda that needed to download a bunch of files from an S3 prefix, perform some aggregation on the data and then write the result to a database. Our EC2 prototype of this on one of the m3 class instances could do the work in about 2 minutes which seemed a perfect opportunity to port to Lambda. Even on the top memory instance at the time (1536mb), the job ju…

We’ve found lambda’s x-ray feature to be very helpful wrt finding the source of slowdowns. I know it wasn’t available during the project you were writing about but wanted to mention it for others.

Re: Comparing Serverless Performance for CPU Bound Tasks

#48
post #5

I remember a few years ago we tried to implement a scheduled Lambda that needed to download a bunch of files from an S3 prefix, perform some aggregation on the data and then write the result to a database. Our EC2 prototype of this on one of the m3 class instances could do the work in about 2 minutes which seemed a perfect opportunity to port to Lambda. Even on the top memory instance at the time (1536mb), the job ju…

This is a fight I currently have with our dev-team atm. Situational awareness is a real concern, especially if the code is misbehaving in productive loads. If a solution doesn't give us situational awareness if things go wrong, I object to that solution.

Re: Comparing Serverless Performance for CPU Bound Tasks

#49

Earlier quoted context omitted.

Here's a good comparison of Lambda vs Azure Functions performance and scaling up to 400 concurrent requests: https://www.azurefromthetrenches.com/azure-functions-vs-aws-...

"Since I published this piece Microsoft have made significant improvements to HTTP scaling on Azure Functions and the below is out of date. Please see this post for a revised comparison. https://www.azurefromthetrenches.com/azure-functions-signifi...

Excellent, though 200ms seems like a long time for a service to respond to something, especially after being warmed up.

I'm only so-so happy about GCF's response time. I honestly wonder why these cloud functions take so long to execute after being warmed up.

Re: Comparing Serverless Performance for CPU Bound Tasks

#50
post #49

Earlier quoted context omitted.

"Since I published this piece Microsoft have made significant improvements to HTTP scaling on Azure Functions and the below is out of date. Please see this post for a revised comparison. https://www.azurefromthetrenches.com/azure-functions-signifi...

Excellent, though 200ms seems like a long time for a service to respond to something, especially after being warmed up. I'm only so-so happy about GCF's response time. I honestly wonder why these cloud functions take so long to execute after being warmed up.

hmm firebase (google) cloud functions take 2,3s in average for me for a respond (simple json output out of firebase)
Post reply on HN