Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serverless Map/Reduce

tothestars.io

Post reply on HN
Print view View original

21–30 of 161 posts

Re: Serverless Map/Reduce

#21

Post by teej »

hueving wrote: post #17
ralusek wrote: post #15

This is a screenshot of my google search from 2 days ago: http://i.imgur.com/BNAcSsn.png I've been using Lambda quite a bit, I think it's SO amazingly useful. Tasks that are highly parallelized and CPU intensive can literally be infinitely scaled out. I find it weird that their poster child use case is still always a reactive event like watching S3 and formatting images. There are so many use cases for directly invok…

>Theoretically, you've taken something that would take 70 hours and had it run in 250ms without having to set up any additional infrastructure. And you've spent the cost of building out that 8 server infrastructure in one batch.

The cost to execute a 250ms Lambda on the most expensive tier 1M times is about $7.70. If your data is in S3 the cost of 1M GETs at the most expensive is $0.40. You don't pay for transfer between S3 and Lambda.

Re: Serverless Map/Reduce

#22

Post by ceejayoz »

ronack wrote: post #20
teej wrote: post #16

Earlier quoted context omitted.

I just restructured a job to be parallel and run on Lambda and I couldn't be happier. Glad to see this idea is gaining mindshare. I found that it was easy to test, easy to debug, easy to maintain, not very expensive, and just kinda worked. The only caveat for those considering taking this path - AWS has an account-wide limit of 100 concurrent lambdas running at once. There's no way to know how many are currently runn…

We had it raised to several thousand concurrent requests with no questions asked. YMMV based on age of account, amount of spend, etc.

Yeah, most of the limits are there just to prevent "whoops, we fired up 500 $2/hour servers and left them running all month, can you refund us?" sort of newbie situations. Very few are technological and AWS has been happy to bump them up for me in each occasion I've had a need.

Re: Serverless Map/Reduce

#23

Post by teej »

ronack wrote: post #20
teej wrote: post #16

Earlier quoted context omitted.

I just restructured a job to be parallel and run on Lambda and I couldn't be happier. Glad to see this idea is gaining mindshare. I found that it was easy to test, easy to debug, easy to maintain, not very expensive, and just kinda worked. The only caveat for those considering taking this path - AWS has an account-wide limit of 100 concurrent lambdas running at once. There's no way to know how many are currently runn…

We had it raised to several thousand concurrent requests with no questions asked. YMMV based on age of account, amount of spend, etc.

Very useful, thanks!

Re: Serverless Map/Reduce

#24

Post by teej »

etrain wrote: post #19
hueving wrote: post #17

Earlier quoted context omitted.

>Theoretically, you've taken something that would take 70 hours and had it run in 250ms without having to set up any additional infrastructure. And you've spent the cost of building out that 8 server infrastructure in one batch.

One of the nice things about lambda is the billing is super granular - you get billed at 100ms intervals. Assuming 70 hours at $0.000000834/100ms [1] The whole job costs $2.10. [1] - https://aws.amazon.com/lambda/pricing/

70 * 60 * 60 * 10 * $0.000000834 = $2.1

Re: Serverless Map/Reduce

#25

Post by teej »

ceejayoz wrote: post #22
ronack wrote: post #20

Earlier quoted context omitted.

We had it raised to several thousand concurrent requests with no questions asked. YMMV based on age of account, amount of spend, etc.

Yeah, most of the limits are there just to prevent "whoops, we fired up 500 $2/hour servers and left them running all month, can you refund us?" sort of newbie situations. Very few are technological and AWS has been happy to bump them up for me in each occasion I've had a need.

Some are, some aren't. For example, my experience has been that it's harder to get the 100 S3 bucket-per-account limit raised.

Re: Serverless Map/Reduce

#26

Post by jedberg »

teej wrote: post #25
ceejayoz wrote: post #22

Earlier quoted context omitted.

Yeah, most of the limits are there just to prevent "whoops, we fired up 500 $2/hour servers and left them running all month, can you refund us?" sort of newbie situations. Very few are technological and AWS has been happy to bump them up for me in each occasion I've had a need.

Some are, some aren't. For example, my experience has been that it's harder to get the 100 S3 bucket-per-account limit raised.

That used to be a hard technological limitation but they fixed it a few months back. You should be able to get more now (but you have to have a good justification for it).

Re: Serverless Map/Reduce

#27

Post by ralusek »

hueving wrote: post #17
ralusek wrote: post #15

This is a screenshot of my google search from 2 days ago: http://i.imgur.com/BNAcSsn.png I've been using Lambda quite a bit, I think it's SO amazingly useful. Tasks that are highly parallelized and CPU intensive can literally be infinitely scaled out. I find it weird that their poster child use case is still always a reactive event like watching S3 and formatting images. There are so many use cases for directly invok…

>Theoretically, you've taken something that would take 70 hours and had it run in 250ms without having to set up any additional infrastructure. And you've spent the cost of building out that 8 server infrastructure in one batch.

As the other poster who replied to you showed, it's actually incredibly cheap. Additionally, it's perfectly elastic. That expensive hardware you provision or buy for your server that can handle this task (at a much, MUCH slower rate) would cost you significantly more. In addition to the cost of the server and running it, you also have a much harder job developing it. If we're talking about multicore, you're now managing the job concurrently on a single machine, and need the code to facilitate it.

I also feel like your comment is implying that the outcomes of the lambda option vs the worker server are comparable. As mentioned, the worker machine will still run in hours, rather than less than a second.

Re: Serverless Map/Reduce

#28

Post by jedberg »

stcredzero wrote: post #18

I wonder if something like AWS Lambda could be applied to multiplayer games? It seems like game-loop based games would be a good domain for such a programming model. The entire game could be expressed as a function that turns tick N into tick N+1. Such a function would be composed of many other functions, of course. So for example, there would also be a function that took as an argument the player at time N and gave…

This was exactly the demo they gave and their AWS Dev Day in SF in July. Using Lambda and the IOT gateway to run a massively multiplayer game (they had everyone in the room play from the web via their phone).

Re: Serverless Map/Reduce

#29

Post by kuschku »

luhn wrote: post #13
CaptSpify wrote: post #12

Earlier quoted context omitted.

Just because you abstract something away doesn't mean it no longer exists. I don't manually manage cache on my laptop's HD, but that doesn't mean it's "cacheless".

I'm not saying "serverless" is a good term, I'm saying that it's the term. It's won. You can argue all you want that it's a terribly misleading/incorrect term, but people aren't going to stop using it. So let's move on.

> It's won.

That’s the linguistic descriptivist position.

We can also require all papers and journals and conferences to use "Function as a Service" everywhere, and force all professors to teach "Function as a Service", and require all official publications to use "Function as a Service", by defining an authoritative dictionary, which gets its authority by law.

Then wait a few months, and the term "serverless" will be gone.

Some countries handle their entire language that way – and have an official institution tasked with updating the language every few years, and the updates become mandatory for business communication, press releases, and schools.

Germany and France are some examples.

IMO, having grown up right after one of the largest such changes in recent German history, it’s a better system than letting the mob decide how to call things, or how to write words, because that leads to pure chaos.

Post reply on HN