Comparing Serverless Performance for CPU Bound Tasks • Hacker News Board

Re: Comparing Serverless Performance for CPU Bound Tasks

#61

Post by js2 » Wed, Jul 11, 2018, 3:17 AM UTC

cremp wrote: post #42

zackbloom wrote: post #34

Earlier quoted context omitted.

Hi, I'm the author of the post, thanks for sharing your thoughts! You're absolutely right we don't know how Lambda works. We have read what we could find that's publicly available, and done a bunch of testing, but Amazon doesn't share all that much about their architecture. I agree that the cold-start times of Lambda are slow, particularly with languages like Java and with VPCs. My plan at the moment is to write a bl…

Thanks for being forthcoming; and I do appreciate the post. I'm partially venting, and I've been using AWS for a while. Vendor lock-in can be a pain in the rear sometimes. A lot of the gripe I've got with these posts is that they seem somewhat incomplete. Hard to say things if you've only been messing with it for a week, and don't have a lot of the ins-and-outs of the services. That being said, I've spent at least 2…

On the Python bits, see this blog post - "Reverse engineering AWS Lambda":

https://www.denialof.services/lambda/

Re: Comparing Serverless Performance for CPU Bound Tasks

#62

Post by [deleted] » Wed, Jul 11, 2018, 3:20 AM UTC

NathanKP wrote: post #13

sreque wrote: post #4

Assuming the author's tests are single-thread, I'm pretty sure 1024 MB doesn't give you a full CPU core on Lambda. I could be wrong though; I haven't payed attention to Lambda in a long time. Last I remember it was 1.5 GB that gave you a full core. This alone makes the comparison between a mid-range server and Lambda unfair, not to mention the differences between language runtimes. That said, if you are using Lambda…

It's not quite that straightforward. Lambda is more expensive per cycle if you are capable of keeping an instance fully utilized and getting every cycle out of that machine. After all if your Node.js or Go code takes say 20ms per request and can handle many concurrent requests you can squeeze a LOT of requests per second from a single instance. But many workloads don't have that high of a request volume and can't act…

[deleted]

Re: Comparing Serverless Performance for CPU Bound Tasks

#63

Post by sudhirj » Wed, Jul 11, 2018, 4:22 AM UTC

zackbloom you’ve made your point already, but remember that these posts represent a moving target. AWS could crush CF performance on pretty much all these numbers with a few configuration changes, which they might well do. And you’re not acknowledging the rest of the Lambda moat, like SQS integration, free S3 bandwidth etc.

Workers has a clear advantage over Lambda@Edge, but not because of the current resource configuration differences across the two products - the advantage is your choice of V8 and adoption of the Service Worker API standard, which brilliantly outshines the L@Edge API choices. Harp on that, most of what you’re talking about now will likely be invalidated by the next reinvent, and they’ll make it a point to tell the world.

Re: Comparing Serverless Performance for CPU Bound Tasks

#64

Post by mmt » Wed, Jul 11, 2018, 4:33 AM UTC

chx wrote: post #43

hobls wrote: post #17

Earlier quoted context omitted.

Anyone using Lambda should _absolutely_ do load testing with different memory configurations. You will get different results, and should analyze what is best for your application. When calculating the overall cost of managing your own instances you should also include time spent by your engineering team. There are particular tipping points in terms of overall requests per second at which point you'd save money by mov…

> And then even above that, you should be running your own instances in a colo space Why are people jumping from EC2 to colo and skipping dedicated servers? Mystery of my life. We were running the 75th largest site in the US some years ago (as measured by Quantcast), ran the numbers and colo was ridiculously expensive and way more troublesome.

> colo was ridiculously expensive

That's slightly surprising to me, unless you were limiting yourself to a particular geographic market for colo that was unusually supply-constrained at the time. (The SFBA during certain years comes to mind).

Sometimes rented servers, for a specific use case (or if the provider is overstocked on a particular model) are a great deal, but I've never seen that at the high-performance end of the spectrum, if they even offer such a model in the first place.

For the average and middle-performance cases, though, for truly comparable servers and connectivity (internal and external, which can be tough to find in the first place), I found rented servers to be moderately more expensive than colo plus buying hardware amortized over 3 years [1].

> and way more troublesome

This one, is the mystery of my life. You mention "magic smoke" downthread, but I've only experienced that once in my entire career and that was with proprietary hardware 2 decades ago.

Conversely, my experience with rented servers is that when there is a hardware problem, other than obvious failure of a replaceable part, "troublesome" doesn't begin to describe it.

[1] yes, including all the costs like installation/rack/stack, network ports, spares, etc. They're not de minimis, but it's maybe a few extra percentage points on the overall cost.

Re: Comparing Serverless Performance for CPU Bound Tasks

#65

Post by kentonv » Wed, Jul 11, 2018, 5:07 AM UTC

sudhirj wrote: post #63

zackbloom you’ve made your point already, but remember that these posts represent a moving target. AWS could crush CF performance on pretty much all these numbers with a few configuration changes, which they might well do. And you’re not acknowledging the rest of the Lambda moat, like SQS integration, free S3 bandwidth etc. Workers has a clear advantage over Lambda@Edge, but not because of the current resource config…

> AWS could crush CF performance on pretty much all these numbers with a few configuration change

Eh? I can see how they could match Workers' raw CPU throughput by simply turning off throttling. But how would they "crush" it? And how can they easily improve other performance measures like network latency, cold start time, or deploy time? Honestly curious what you're getting at here.

> the advantage is your choice of V8 and adoption of the Service Worker API standard, which brilliantly outshines the L@Edge API choices.

Thanks for the kind words.

Re: Comparing Serverless Performance for CPU Bound Tasks

#66

Post by penagwin » Wed, Jul 11, 2018, 5:59 AM UTC

kentonv wrote: post #65

sudhirj wrote: post #63

zackbloom you’ve made your point already, but remember that these posts represent a moving target. AWS could crush CF performance on pretty much all these numbers with a few configuration changes, which they might well do. And you’re not acknowledging the rest of the Lambda moat, like SQS integration, free S3 bandwidth etc. Workers has a clear advantage over Lambda@Edge, but not because of the current resource config…

> AWS could crush CF performance on pretty much all these numbers with a few configuration change Eh? I can see how they could match Workers' raw CPU throughput by simply turning off throttling. But how would they "crush" it? And how can they easily improve other performance measures like network latency, cold start time, or deploy time? Honestly curious what you're getting at here. > the advantage is your choice of…

Hello, I know you're the tech lead for the web workers at cloudflare so pardon my ignorance if I'm wrong.

At least for pure computer speed, I think he means that if you (cloudflare) and AWS got into an arms race in terms of allocated CPU/Memory to the webworkers/lambda, they have more raw resources to do so. They also have a global presence, not necessarily to the degree that you do obviously.

I highly doubt they would do this, and I think you have the superior product. I'm just a student/hobbyist so I admittedly don't have a ton of experience. I'm very biased towards CF, you guys are great! :D

Re: Comparing Serverless Performance for CPU Bound Tasks

#67

Post by zackbloom » Wed, Jul 11, 2018, 6:37 AM UTC

penagwin wrote: post #66

kentonv wrote: post #65

Earlier quoted context omitted.

> AWS could crush CF performance on pretty much all these numbers with a few configuration change Eh? I can see how they could match Workers' raw CPU throughput by simply turning off throttling. But how would they "crush" it? And how can they easily improve other performance measures like network latency, cold start time, or deploy time? Honestly curious what you're getting at here. > the advantage is your choice of…

Hello, I know you're the tech lead for the web workers at cloudflare so pardon my ignorance if I'm wrong. At least for pure computer speed, I think he means that if you (cloudflare) and AWS got into an arms race in terms of allocated CPU/Memory to the webworkers/lambda, they have more raw resources to do so. They also have a global presence, not necessarily to the degree that you do obviously. I highly doubt they wou…

Honestly, if that's what happens and Lambda end up priced more reasonably because of us I would consider that a victory, and a lovely outcome for the world.

That said, I also want people to build applications which run all around the world. I can only imagine what it's like for people in Australia to browse the modern Internet, but I doubt it's particularly fun and I'd like to help fix it if I can.

Re: Comparing Serverless Performance for CPU Bound Tasks

#68

Post by sudhirj » Wed, Jul 11, 2018, 7:05 AM UTC

kentonv wrote: post #65

sudhirj wrote: post #63

zackbloom you’ve made your point already, but remember that these posts represent a moving target. AWS could crush CF performance on pretty much all these numbers with a few configuration changes, which they might well do. And you’re not acknowledging the rest of the Lambda moat, like SQS integration, free S3 bandwidth etc. Workers has a clear advantage over Lambda@Edge, but not because of the current resource config…

> AWS could crush CF performance on pretty much all these numbers with a few configuration change Eh? I can see how they could match Workers' raw CPU throughput by simply turning off throttling. But how would they "crush" it? And how can they easily improve other performance measures like network latency, cold start time, or deploy time? Honestly curious what you're getting at here. > the advantage is your choice of…

Because the pure geographic latency numbers are quite comparable. Workers latency when compared to Lambda@Edge/CloudFront isn't that different, and is also a moving target because both of you are adding locations all over the world continuously and buying more and more local racks. There's no clear winner here.

Cold start time is fixable, there's already been large improvements, and this is more a VPC problem. Easiest way would be to overprovision aggressively / make sure Lambdas tied to APIGW always have lambda running, or ever use some variation of ML predictions to keep things warm. But again, this isn't comparable - Lambda is a truck compared to Workers/Lambda@Edge being bikes. Parallel scalability is more important there than speed. There are enough ways to keep a few warm ones ready.

Deploy time is really download time from S3, AWS could cache more aggressively on the local cloudfront caches. I'm not seeing deployment time as being a big factor, though.

By "crush" I mean make claims about performance irrelevant. To claim that AWS cannot equalize performance between Lambda@Edge and Workers doesn't make sense, they can. And they can improve Lambda price-performance as well, and are already doing so. I'm saying this cannot and is not the Workers USP - no one in the AWS ecosystem is going to jump to Workers based on this because it lacks the rest of the AWS ecosystem.

> > the advantage is your choice of V8 and adoption of the Service Worker API standard, which brilliantly outshines the L@Edge API choices.

That's really the big differentiator for Workers. I think you should blow that trumpet a lot more. If you only publicize performance numbers, what happens to the Workers story when that advantage is lost?

> Thanks for the kind words.

You made a good decision and built something great, you're welcome.