How does it compare to 3 years old Joyent's Manta ?
AFAIK it was especially designed for this kind of purposes. The processing is made directly on the servers storing the data..
Serverless Map/Reduce
11–20 of 161 posts
Re: Serverless Map/Reduce
#12If it doesn't run on a server, then what does this plumbing-work run on? Clickbait name?
Oh come on. Is it still not accepted that "serverless" is the colloquial name for AWS Lambda and comparable services? Stop trying to make "FaaS" happen. It's not going to happen.
Just because you abstract something away doesn't mean it no longer exists. I don't manually manage cache on my laptop's HD, but that doesn't mean it's "cacheless".
Re: Serverless Map/Reduce
#13Earlier quoted context omitted.
Oh come on. Is it still not accepted that "serverless" is the colloquial name for AWS Lambda and comparable services? Stop trying to make "FaaS" happen. It's not going to happen.
Just because you abstract something away doesn't mean it no longer exists. I don't manually manage cache on my laptop's HD, but that doesn't mean it's "cacheless".
I'm not saying "serverless" is a good term, I'm saying that it's the term. It's won. You can argue all you want that it's a terribly misleading/incorrect term, but people aren't going to stop using it. So let's move on.
Re: Serverless Map/Reduce
#14If it doesn't run on a server, then what does this plumbing-work run on? Clickbait name?
Call it Function as a Service if you prefer.
Re: Serverless Map/Reduce
#15This is a screenshot of my google search from 2 days ago:
http://i.imgur.com/BNAcSsn.png
I've been using Lambda quite a bit, I think it's SO amazingly useful. Tasks that are highly parallelized and CPU intensive can literally be infinitely scaled out. I find it weird that their poster child use case is still always a reactive event like watching S3 and formatting images. There are so many use cases for directly invoking a lambda directly from your code.
Imagine a case where you had to parse a million documents with a relatively expensive computation, let's say 250MS per document. Maybe you have a solid machine with a few cores that's running your server, but even then you can't have the server cpu locked for so long, so naturally you'd need some sort of worker server set up. With a good machine and multiple cores, maybe you get 8 running at once. With a lambda, you can forego the worker server altogether. Just invoke a million lambdas directly from your application server, completely parallelized.
Theoretically, you've taken something that would take 70 hours and had it run in 250ms without having to set up any additional infrastructure.
Re: Serverless Map/Reduce
#16This is a screenshot of my google search from 2 days ago: http://i.imgur.com/BNAcSsn.png I've been using Lambda quite a bit, I think it's SO amazingly useful. Tasks that are highly parallelized and CPU intensive can literally be infinitely scaled out. I find it weird that their poster child use case is still always a reactive event like watching S3 and formatting images. There are so many use cases for directly invok…
I just restructured a job to be parallel and run on Lambda and I couldn't be happier. Glad to see this idea is gaining mindshare. I found that it was easy to test, easy to debug, easy to maintain, not very expensive, and just kinda worked.
The only caveat for those considering taking this path - AWS has an account-wide limit of 100 concurrent lambdas running at once. There's no way to know how many are currently running unless you keep track of it yourself. You'll only find out when you try to kick off a job and it get's throttled. I haven't contacted support yet to find out how hard it is for the limit to get raised or what they'll raise it to.
Re: Serverless Map/Reduce
#17This is a screenshot of my google search from 2 days ago: http://i.imgur.com/BNAcSsn.png I've been using Lambda quite a bit, I think it's SO amazingly useful. Tasks that are highly parallelized and CPU intensive can literally be infinitely scaled out. I find it weird that their poster child use case is still always a reactive event like watching S3 and formatting images. There are so many use cases for directly invok…
>Theoretically, you've taken something that would take 70 hours and had it run in 250ms without having to set up any additional infrastructure.
And you've spent the cost of building out that 8 server infrastructure in one batch.
Re: Serverless Map/Reduce
#18I wonder if something like AWS Lambda could be applied to multiplayer games? It seems like game-loop based games would be a good domain for such a programming model. The entire game could be expressed as a function that turns tick N into tick N+1. Such a function would be composed of many other functions, of course. So for example, there would also be a function that took as an argument the player at time N and gave the player at time N+1.
Such a model would allow infrastructure developers to abstract away most of the concerns around networking, collisions, security, etc., and let game developers concentrate their efforts on simply making the game.
I currently have a game server cluster written in Golang, where the locations are instantiated with an idempotent request operation. It doesn't matter if a particular location-instance exists at a particular moment. It's sufficient for the "master control" server to only approximately know the loads of the different cluster server processes. My experience leads me to believe that something like AWS Lambda, but optimized for implementing game loops would work well, so long as game developers could get their heads around pure functional programming and implement with soft real-time requirements in mind. (John Carmack already advocates the use of pure functions, and game devs in general already do the latter.)
Re: Serverless Map/Reduce
#19This is a screenshot of my google search from 2 days ago: http://i.imgur.com/BNAcSsn.png I've been using Lambda quite a bit, I think it's SO amazingly useful. Tasks that are highly parallelized and CPU intensive can literally be infinitely scaled out. I find it weird that their poster child use case is still always a reactive event like watching S3 and formatting images. There are so many use cases for directly invok…
>Theoretically, you've taken something that would take 70 hours and had it run in 250ms without having to set up any additional infrastructure. And you've spent the cost of building out that 8 server infrastructure in one batch.
One of the nice things about lambda is the billing is super granular - you get billed at 100ms intervals.
Assuming 70 hours at $0.000000834/100ms [1]
The whole job costs $2.10.
Re: Serverless Map/Reduce
#20This is a screenshot of my google search from 2 days ago: http://i.imgur.com/BNAcSsn.png I've been using Lambda quite a bit, I think it's SO amazingly useful. Tasks that are highly parallelized and CPU intensive can literally be infinitely scaled out. I find it weird that their poster child use case is still always a reactive event like watching S3 and formatting images. There are so many use cases for directly invok…
I just restructured a job to be parallel and run on Lambda and I couldn't be happier. Glad to see this idea is gaining mindshare. I found that it was easy to test, easy to debug, easy to maintain, not very expensive, and just kinda worked. The only caveat for those considering taking this path - AWS has an account-wide limit of 100 concurrent lambdas running at once. There's no way to know how many are currently runn…
We had it raised to several thousand concurrent requests with no questions asked. YMMV based on age of account, amount of spend, etc.