Show HN: Corral – A Serverless MapReduce Framework
1–10 of 39 posts
Re: Show HN: Corral – A Serverless MapReduce Framework
#2Hi HN, author here. Corral is my attempt at a performant, easy-to-deploy MapReduce. Unlike traditional frameworks like Hadoop, it uses AWS Lambda for execution and is “serverless” as a result. It was initially kicked-off by AWS adding Lambda support for Go, but draws on experience I’ve had using Lambda tools like Zappa[1] and Serverless[2] in the past.
I think there’s a lot of interesting applications for using function-as-a-service platforms as executors in data processing frameworks such as this.
If you’re interested more in the development/internals of this project, I wrote a blog post with more details: https://benjamincongdon.me/blog/2018/05/02/Introducing-Corra...
[1]: https://github.com/Miserlou/Zappa [2]: https://serverless.com/
Re: Show HN: Corral – A Serverless MapReduce Framework
#3Hi HN, author here. Corral is my attempt at a performant, easy-to-deploy MapReduce. Unlike traditional frameworks like Hadoop, it uses AWS Lambda for execution and is “serverless” as a result. It was initially kicked-off by AWS adding Lambda support for Go, but draws on experience I’ve had using Lambda tools like Zappa[1] and Serverless[2] in the past. I think there’s a lot of interesting applications for using funct…
Hi.
How do you deal with the 5min (IIRC) execution time limit of Lambda ?
Re: Show HN: Corral – A Serverless MapReduce Framework
#4Hi HN, author here. Corral is my attempt at a performant, easy-to-deploy MapReduce. Unlike traditional frameworks like Hadoop, it uses AWS Lambda for execution and is “serverless” as a result. It was initially kicked-off by AWS adding Lambda support for Go, but draws on experience I’ve had using Lambda tools like Zappa[1] and Serverless[2] in the past. I think there’s a lot of interesting applications for using funct…
Hi,
For a small map reduce load, say a terabyte (to replace a single MR node), how much would you estimate the aws cost would be?
Re: Show HN: Corral – A Serverless MapReduce Framework
#5Hi HN, author here. Corral is my attempt at a performant, easy-to-deploy MapReduce. Unlike traditional frameworks like Hadoop, it uses AWS Lambda for execution and is “serverless” as a result. It was initially kicked-off by AWS adding Lambda support for Go, but draws on experience I’ve had using Lambda tools like Zappa[1] and Serverless[2] in the past. I think there’s a lot of interesting applications for using funct…
Hi. How do you deal with the 5min (IIRC) execution time limit of Lambda ?
Yeah, max execution time (and max memory usage) are the main constraints of using Lambda.
Corral deals with this by splitting input data into small enough chunks that each chunk can be processed within the timeout -- I exposed options for setting the amount of data that each Lambda function has to process. However, if each data item requires more than 5 min of processing, then corral won't work for you.
The "driver" that coordinates the Lambda functions runs locally (not in Lambda), so it doesn't have this constraint.
Re: Show HN: Corral – A Serverless MapReduce Framework
#6Hi HN, author here. Corral is my attempt at a performant, easy-to-deploy MapReduce. Unlike traditional frameworks like Hadoop, it uses AWS Lambda for execution and is “serverless” as a result. It was initially kicked-off by AWS adding Lambda support for Go, but draws on experience I’ve had using Lambda tools like Zappa[1] and Serverless[2] in the past. I think there’s a lot of interesting applications for using funct…
Hi, For a small map reduce load, say a terabyte (to replace a single MR node), how much would you estimate the aws cost would be?
Pricing depends a lot on how much memory your job requires[1] and how much processing each record requires -- i.e. the pricing is more sensitive to usage.
As a very rough estimate, for a light-to-medium load of 1Tb, the cost would probably be in the ballpark of ~$0.50. AWS's own reference MR framework[2] (which is mostly a tech demo) quotes prices in a similar order of magnitude.
Corral isn't great for processing-heavy MR jobs, as Lambda pricing rises quickly if you need a lot of memory or take a lot of time with each record. But, for small-ish low-overhead jobs, it can pretty easily beat the pricing and hassle of using something like EMR.
[1]: https://aws.amazon.com/lambda/pricing/#Lambda_pricing_detail... [2]: https://github.com/awslabs/lambda-refarch-mapreduce/
Re: Show HN: Corral – A Serverless MapReduce Framework
#7Hi HN, author here. Corral is my attempt at a performant, easy-to-deploy MapReduce. Unlike traditional frameworks like Hadoop, it uses AWS Lambda for execution and is “serverless” as a result. It was initially kicked-off by AWS adding Lambda support for Go, but draws on experience I’ve had using Lambda tools like Zappa[1] and Serverless[2] in the past. I think there’s a lot of interesting applications for using funct…
Nice! I've been thinking about doing this kind of thing for a while. I have long experimented with doing map reduce style work outside of things like Hadoop and gotten much better results due to being able to tune for different things much more quickly.
Re: Show HN: Corral – A Serverless MapReduce Framework
#8Why call it "serverless" if I need to provide stuff like:
AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION, AWS_TEST_BUCKET
Re: Show HN: Corral – A Serverless MapReduce Framework
#9Why call it "serverless" if I need to provide stuff like: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION, AWS_TEST_BUCKET
Because you don't need to manage any servers.
Re: Show HN: Corral – A Serverless MapReduce Framework
#10Hi HN, author here. Corral is my attempt at a performant, easy-to-deploy MapReduce. Unlike traditional frameworks like Hadoop, it uses AWS Lambda for execution and is “serverless” as a result. It was initially kicked-off by AWS adding Lambda support for Go, but draws on experience I’ve had using Lambda tools like Zappa[1] and Serverless[2] in the past. I think there’s a lot of interesting applications for using funct…
I'd like to see your readme expanded with some figures:
* Processing speed - that is, how long does it take to do that word count example on a nontrivial dataset? Something that takes hours on a local machine, vs minutes in map/reduce. Comparing local to this to e.g. Hadoop or Google BigQuery or whatever viable alternative there is. * Cost. I think that's probably the biggest factor here. I don't get the impression that Lambda was intended for big data or highly resource / i/o / processing intensive operations, but, I'd love to be proven wrong. * Actually, mostly just cost vs performance.
I mean it's a neat idea but if the serverless benefit is outweighed by difficulty in setting up, cost, performance, etc compared to dedicated big data solutions it's going to stay a proof of concept.