Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Comparing Serverless Performance for CPU Bound Tasks

blog.cloudflare.com

Post reply on HN
Print view View original

31–40 of 68 posts

Re: Comparing Serverless Performance for CPU Bound Tasks

#31

Post by cremp »

Each and every one of these posts from Cloudflare is direct targeting and completely biased. Should be noted that their response times are indicative of them being far fewer hops away. Unless they can run in the same DC, or even make the RTT fair, their 'Webpage response' metric is utterly useless.

Notice how they admit that they don't know how lambda really works. They switch between lambda@edge and Region-based lambdas, and don't seem to be able to be consistent with it.

Java Lambdas have horrible cold start times, and I'm not seeing any of this reflected anywhere in their report.

> Our Lambda is deployed to with the default 128MB of memory behind an API Gateway in us-east-1

Well duh the lambda is slower; it's going through API Gateway, and that does authentication processing as well.

All in all, these blog posts from Cloudflare are turning me off from them entirely, because they aren't even saying 'yeah, AWS got us beat in this case here.'

Re: Comparing Serverless Performance for CPU Bound Tasks

#32

Post by [deleted] »

hobls wrote: post #17
sreque wrote: post #4

Assuming the author's tests are single-thread, I'm pretty sure 1024 MB doesn't give you a full CPU core on Lambda. I could be wrong though; I haven't payed attention to Lambda in a long time. Last I remember it was 1.5 GB that gave you a full core. This alone makes the comparison between a mid-range server and Lambda unfair, not to mention the differences between language runtimes. That said, if you are using Lambda…

Anyone using Lambda should _absolutely_ do load testing with different memory configurations. You will get different results, and should analyze what is best for your application. When calculating the overall cost of managing your own instances you should also include time spent by your engineering team. There are particular tipping points in terms of overall requests per second at which point you'd save money by mov…

[deleted]

Re: Comparing Serverless Performance for CPU Bound Tasks

#33

Post by cremp »

microcolonel wrote: post #29

Good work at CloudFlare. I personally figure that Amazon ought to be doing more interesting things with Lambda, like maybe starting the workers from a memory snapshot.

They do. After the cold start, things are just run in memory.

Java for example will keep your static variables and such in memory, and keep /tmp until you haven't called the lambda for a while.

With GoLang, they call start your program, and just call the handler method as required.

Re: Comparing Serverless Performance for CPU Bound Tasks

#34

Post by zackbloom »

cremp wrote: post #31

Each and every one of these posts from Cloudflare is direct targeting and completely biased. Should be noted that their response times are indicative of them being far fewer hops away. Unless they can run in the same DC, or even make the RTT fair, their 'Webpage response' metric is utterly useless. Notice how they admit that they don't know how lambda really works. They switch between lambda@edge and Region-based lam…

Hi, I'm the author of the post, thanks for sharing your thoughts!

You're absolutely right we don't know how Lambda works. We have read what we could find that's publicly available, and done a bunch of testing, but Amazon doesn't share all that much about their architecture.

I agree that the cold-start times of Lambda are slow, particularly with languages like Java and with VPCs. My plan at the moment is to write a blog post focused on cold-start times specifically, when I can figure out how to accurately test that around the world.

I'm not entirely sure why API Gateway would add hundreds of ms of latency. We also do authentication processing with our Access product, for example, and it certainly doesn't add anywhere near that. I also don't have any of those API Gateway features enabled to begin with. If you would very much like it, I'm happy to test a Lambda by hitting the Invoke API directly, but I doubt you'll see much of a difference. As the post says, Lambda is granting us a much smaller quantity of CPU time, there's not much you can do to get around that.

I apologize if the transitions between global and region-specific tests are unclear. The majority of the tests are being done from DC, specifically to focus the comparison around execution time, not global latency. I did my best in the post to specify where I was running the test from. If you have an idea of how that can be better expressed please share and I'll do my best to incorporate it in the future.

Re: Comparing Serverless Performance for CPU Bound Tasks

#35

Post by zackbloom »

NathanKP wrote: post #28
zackbloom wrote: post #20

Earlier quoted context omitted.

I have a Worker running bcrypt here: https://cloudflareworkers.com/#4addaef33b10b6a58954ffbb310e7... Based on this code: https://gist.github.com/zackbloom/c0064838cbf85e7b81df9d4690... That means it would cost you $0.50 / million requests. AWS Lambda would be $1.84 / million, $3.50 / million for API Gateway, $0.40 / million for AWS Route 53, and various other charges.

Technically CloudFlare workers is better compared with Lambda @ Edge no? I don't think you'd be using API Gateway, instead CloudFront right?

Originally we were thinking about it in those terms (as an alternative to Lambda@Edge), but based on our recent results I am happy to have people compare it to Lambda as well. Lambda with API Gateway is at least eight times more expensive than Workers, and only runs your code in one location instead of 151. Unless you're using Lambda to do something ultra-Amazon-specific (like process S3 changes), I don't see why it would be the better choice.

Re: Comparing Serverless Performance for CPU Bound Tasks

#37

Post by reilly3000 »

philipodonnell wrote: post #24
hobls wrote: post #17

Earlier quoted context omitted.

Anyone using Lambda should _absolutely_ do load testing with different memory configurations. You will get different results, and should analyze what is best for your application. When calculating the overall cost of managing your own instances you should also include time spent by your engineering team. There are particular tipping points in terms of overall requests per second at which point you'd save money by mov…

Are there any tools that allow you to load test Lambda services with different memory configurations?

In Serverless Framework you can specify memory per function so it stands to reason making multiple copies at different memory levels would be pretty easy to test, even concurrently.

Re: Comparing Serverless Performance for CPU Bound Tasks

#38

Post by zlynx »

hobls wrote: post #17
sreque wrote: post #4

Assuming the author's tests are single-thread, I'm pretty sure 1024 MB doesn't give you a full CPU core on Lambda. I could be wrong though; I haven't payed attention to Lambda in a long time. Last I remember it was 1.5 GB that gave you a full core. This alone makes the comparison between a mid-range server and Lambda unfair, not to mention the differences between language runtimes. That said, if you are using Lambda…

Anyone using Lambda should _absolutely_ do load testing with different memory configurations. You will get different results, and should analyze what is best for your application. When calculating the overall cost of managing your own instances you should also include time spent by your engineering team. There are particular tipping points in terms of overall requests per second at which point you'd save money by mov…

Has anyone done research on cooling datacenters on the Moon?

Re: Comparing Serverless Performance for CPU Bound Tasks

#39

Post by chrisco255 »

com2kid wrote: post #2

I'd love to see an honest comparison across other providers, throwing in Google's Firebase Functions and Azure Cloud Functions.

Here's a good comparison of Lambda vs Azure Functions performance and scaling up to 400 concurrent requests: https://www.azurefromthetrenches.com/azure-functions-vs-aws-...

Re: Comparing Serverless Performance for CPU Bound Tasks

#40

Post by hobls »

reilly3000 wrote: post #37
philipodonnell wrote: post #24

Earlier quoted context omitted.

Are there any tools that allow you to load test Lambda services with different memory configurations?

In Serverless Framework you can specify memory per function so it stands to reason making multiple copies at different memory levels would be pretty easy to test, even concurrently.

The real fun starts when you need to start spinning up a distributed load testing fleet to actually generate enough load. :P
Post reply on HN