Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Comparing Serverless Performance for CPU Bound Tasks

blog.cloudflare.com

Post reply on HN
Print view View original

21–30 of 68 posts

Re: Comparing Serverless Performance for CPU Bound Tasks

#21

Post by manigandham »

Dunedan wrote: post #7
wolf550e wrote: post #6

I'll copy from Twitter[1]: @zackbloom @jgrahamc I can't find it in the docs on AWS site, but I've read that AWS Lambda scales CPU linearly until 1.5GB, then gives you 2nd thread/core and again scales linearly until 3GB. If your PBKDF2 was single threaded, Lambda bigger than 1.5GB is wasted. 11:12 AM - 9 Jul 2018 reply by blog post author[2]: Replying to @ZTarantov @Cloudflare @jgrahamc I can't think of a way to test…

Yes, Lambda functions use a second core above 1536MB of memory. Back in the past they had it in their documentation, but removed it at a certain point. Also see: https://stackoverflow.com/questions/34135359/whats-the-maxim...

It might not be true any longer if they removed it from the documentation.

Re: Comparing Serverless Performance for CPU Bound Tasks

#22

Post by manigandham »

djhworld wrote: post #19
RhodesianHunter wrote: post #18

Earlier quoted context omitted.

If your job is easily parallelizable then you can run multiple lambdas in parallel. For the above use case they probably should have kicked off one lambda per prefix or similar.

That's exactly what we were doing. 1 Lambda to download and aggregate all files under a prefix. The problem was the task just couldn't complete in < 5 minutes.

You have to fan out further then, process each file separately and aggregate the aggregates, using SQS or something else to queue up the processing.

Azure's Durable Functions have an advantage here in making extreme fan-out situations easy.

Re: Comparing Serverless Performance for CPU Bound Tasks

#23

Post by djhworld »

manigandham wrote: post #22
djhworld wrote: post #19

Earlier quoted context omitted.

That's exactly what we were doing. 1 Lambda to download and aggregate all files under a prefix. The problem was the task just couldn't complete in < 5 minutes.

You have to fan out further then, process each file separately and aggregate the aggregates, using SQS or something else to queue up the processing. Azure's Durable Functions have an advantage here in making extreme fan-out situations easy.

We considered it, but at the time we just felt implementing map/reduce over Lambda would just introduce a more complex architecture for such a simple problem.

Maybe the recently introduced SQS->Lambda support might make it a bit cleaner, but in the end we opted for EC2.

Re: Comparing Serverless Performance for CPU Bound Tasks

#24

Post by philipodonnell »

hobls wrote: post #17
sreque wrote: post #4

Assuming the author's tests are single-thread, I'm pretty sure 1024 MB doesn't give you a full CPU core on Lambda. I could be wrong though; I haven't payed attention to Lambda in a long time. Last I remember it was 1.5 GB that gave you a full core. This alone makes the comparison between a mid-range server and Lambda unfair, not to mention the differences between language runtimes. That said, if you are using Lambda…

Anyone using Lambda should _absolutely_ do load testing with different memory configurations. You will get different results, and should analyze what is best for your application. When calculating the overall cost of managing your own instances you should also include time spent by your engineering team. There are particular tipping points in terms of overall requests per second at which point you'd save money by mov…

Are there any tools that allow you to load test Lambda services with different memory configurations?

Re: Comparing Serverless Performance for CPU Bound Tasks

#25

Post by hobls »

philipodonnell wrote: post #24
hobls wrote: post #17

Earlier quoted context omitted.

Anyone using Lambda should _absolutely_ do load testing with different memory configurations. You will get different results, and should analyze what is best for your application. When calculating the overall cost of managing your own instances you should also include time spent by your engineering team. There are particular tipping points in terms of overall requests per second at which point you'd save money by mov…

Are there any tools that allow you to load test Lambda services with different memory configurations?

Not to my knowledge. I'd just do something simple by changing your CloudFormation (or Terraform or whatever) templates, deploying the new config, and then running another load test. Could also just spin multiple different versions of the Lambda up, but at some point you need to start getting creative to actually generate enough traffic and running multiple load tests at the same time becomes a pain.

When it comes to load testing tools I like Vegeta[1], personally. (Though I've also used some much more complicated proprietary tools when testing at great scale.)

1: https://github.com/tsenart/vegeta

Re: Comparing Serverless Performance for CPU Bound Tasks

#26

Post by [deleted] »

hobls wrote: post #17
sreque wrote: post #4

Assuming the author's tests are single-thread, I'm pretty sure 1024 MB doesn't give you a full CPU core on Lambda. I could be wrong though; I haven't payed attention to Lambda in a long time. Last I remember it was 1.5 GB that gave you a full core. This alone makes the comparison between a mid-range server and Lambda unfair, not to mention the differences between language runtimes. That said, if you are using Lambda…

Anyone using Lambda should _absolutely_ do load testing with different memory configurations. You will get different results, and should analyze what is best for your application. When calculating the overall cost of managing your own instances you should also include time spent by your engineering team. There are particular tipping points in terms of overall requests per second at which point you'd save money by mov…

[deleted]

Re: Comparing Serverless Performance for CPU Bound Tasks

#27

Post by [deleted] »

NathanKP wrote: post #13
sreque wrote: post #4

Assuming the author's tests are single-thread, I'm pretty sure 1024 MB doesn't give you a full CPU core on Lambda. I could be wrong though; I haven't payed attention to Lambda in a long time. Last I remember it was 1.5 GB that gave you a full core. This alone makes the comparison between a mid-range server and Lambda unfair, not to mention the differences between language runtimes. That said, if you are using Lambda…

It's not quite that straightforward. Lambda is more expensive per cycle if you are capable of keeping an instance fully utilized and getting every cycle out of that machine. After all if your Node.js or Go code takes say 20ms per request and can handle many concurrent requests you can squeeze a LOT of requests per second from a single instance. But many workloads don't have that high of a request volume and can't act…

[deleted]

Re: Comparing Serverless Performance for CPU Bound Tasks

#28

Post by NathanKP »

zackbloom wrote: post #20
NathanKP wrote: post #15

Earlier quoted context omitted.

No idea. Given that a secure password hash will probably take about 100ms-200ms of execution time to calculate that would fall under Cloudflares "custom pricing" tier that you need to call them and negotiate. The baseline Cloudflare worker tiers are limited to less than 5ms, less than 10ms, and less than 50ms, which isn't going to be enough time to calculate a 12 round bcrypt for example.

I have a Worker running bcrypt here: https://cloudflareworkers.com/#4addaef33b10b6a58954ffbb310e7... Based on this code: https://gist.github.com/zackbloom/c0064838cbf85e7b81df9d4690... That means it would cost you $0.50 / million requests. AWS Lambda would be $1.84 / million, $3.50 / million for API Gateway, $0.40 / million for AWS Route 53, and various other charges.

Technically CloudFlare workers is better compared with Lambda @ Edge no? I don't think you'd be using API Gateway, instead CloudFront right?

Re: Comparing Serverless Performance for CPU Bound Tasks

#30

Post by [deleted] »

hobls wrote: post #17
sreque wrote: post #4

Assuming the author's tests are single-thread, I'm pretty sure 1024 MB doesn't give you a full CPU core on Lambda. I could be wrong though; I haven't payed attention to Lambda in a long time. Last I remember it was 1.5 GB that gave you a full core. This alone makes the comparison between a mid-range server and Lambda unfair, not to mention the differences between language runtimes. That said, if you are using Lambda…

Anyone using Lambda should _absolutely_ do load testing with different memory configurations. You will get different results, and should analyze what is best for your application. When calculating the overall cost of managing your own instances you should also include time spent by your engineering team. There are particular tipping points in terms of overall requests per second at which point you'd save money by mov…

[deleted]
Post reply on HN