Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Corral – A Serverless MapReduce Framework

github.com

Post reply on HN
Print view View original

21–30 of 39 posts

Re: Show HN: Corral – A Serverless MapReduce Framework

#21

Post by mayank »

jedimastert wrote: post #19

I'm a complete novice when it comes to distributed computing, so I'm not too sure I get the idea of a MapReduce framework. I'm gonna try to suss it out here as I understand it, and if someone could correct me where I'm wrong, that'd be awesome. First off, when I see "map" and "reduce" I think of the functional programming/data processing equivalents of mapping, meaning to apply a function to every element in a set (l…

> I would think that the reduce function would need some sort of "accumulator" input, and that you'd only get one thing as an output, as opposed to more files of data.

It may be easier to think of the reduce step more like a SQL GROUP BY rather than a function of a list. The map phase emits a bunch of (key, value) pairs, and all values with the same key are processed by the same reducer function (but each key gets a new reducer, modulo implementation details).

So in your paradigm, there are many reduce functions, each starting with a null accumulated value, resulting in many outputs rather than a single one.

Re: Show HN: Corral – A Serverless MapReduce Framework

#22

Post by [deleted] »

bcongdon wrote: post #2

Hi HN, author here. Corral is my attempt at a performant, easy-to-deploy MapReduce. Unlike traditional frameworks like Hadoop, it uses AWS Lambda for execution and is “serverless” as a result. It was initially kicked-off by AWS adding Lambda support for Go, but draws on experience I’ve had using Lambda tools like Zappa[1] and Serverless[2] in the past. I think there’s a lot of interesting applications for using funct…

[deleted]

Re: Show HN: Corral – A Serverless MapReduce Framework

#23

Post by jedimastert »

mayank wrote: post #21
jedimastert wrote: post #19

I'm a complete novice when it comes to distributed computing, so I'm not too sure I get the idea of a MapReduce framework. I'm gonna try to suss it out here as I understand it, and if someone could correct me where I'm wrong, that'd be awesome. First off, when I see "map" and "reduce" I think of the functional programming/data processing equivalents of mapping, meaning to apply a function to every element in a set (l…

> I would think that the reduce function would need some sort of "accumulator" input, and that you'd only get one thing as an output, as opposed to more files of data. It may be easier to think of the reduce step more like a SQL GROUP BY rather than a function of a list. The map phase emits a bunch of (key, value) pairs, and all values with the same key are processed by the same reducer function (but each key gets a…

Interesting. That actually makes a lot of sense. I've done a touch more reading, and I actually only just now realized that the "word count" example doesn't count all of the words in a document, but rather all occurrences of each word. That clears a lot of questions up for me.

Re: Show HN: Corral – A Serverless MapReduce Framework

#25

Post by jessaustin »

Annatar wrote: post #13

Any time I read "serverless", I get a violent allergic reaction. There is no such thing, as software requires hardware to run on, and the person or persons who went "serverless" simply chose to stick their head(s) in the sand and punt the OS engineering and hardware design and maintenance off to someone else, hoping that it will just work. But it does not, and eventually there will be an outage and lost money. One ca…

Any time I read "serverless", I get a violent allergic reaction.

You're even, then, because I've got the allergy to this trivial monotonous whinging about the by-now-well-understood meaning of the term "serverless". I'm not the only one.

Re: Show HN: Corral – A Serverless MapReduce Framework

#26

Post by shaunray »

bcongdon wrote: post #6
quickben wrote: post #4

Earlier quoted context omitted.

Hi, For a small map reduce load, say a terabyte (to replace a single MR node), how much would you estimate the aws cost would be?

Pricing depends a lot on how much memory your job requires[1] and how much processing each record requires -- i.e. the pricing is more sensitive to usage. As a very rough estimate, for a light-to-medium load of 1Tb, the cost would probably be in the ballpark of ~$0.50. AWS's own reference MR framework[2] (which is mostly a tech demo) quotes prices in a similar order of magnitude. Corral isn't great for processing-hea…

Hi OP, I am the person that made the most recent changes to the AWS Labs refarch. I had been working on a golang version and wanted to clean up the python one. Sunil the original author used the AMPLABS benchmark to calculate the results table. I was planning on updating it with the 1 and 5 node test. Would be happy to include Corral as well.

Re: Show HN: Corral – A Serverless MapReduce Framework

#27

Post by Annatar »

scarface74 wrote: post #16
Annatar wrote: post #13

Any time I read "serverless", I get a violent allergic reaction. There is no such thing, as software requires hardware to run on, and the person or persons who went "serverless" simply chose to stick their head(s) in the sand and punt the OS engineering and hardware design and maintenance off to someone else, hoping that it will just work. But it does not, and eventually there will be an outage and lost money. One ca…

Any time I read "serverless", I get a violent allergic reaction. There is no such thing, as software requires hardware to run on, The meanings of words morph over time. When developers mention serverless everyone knows what it means it that context. Just like when someone says there is a bug in their code no one thinks that there are roaches running around in their computer. and the person or persons who went "server…

Netflix has an army of professional system and kernel engineers who optimize and engineer the OSes they run on; they just use AWS for virtual machine and datacenter capacity.

Re: Show HN: Corral – A Serverless MapReduce Framework

#28

Post by Annatar »

rockostrich wrote: post #18
Annatar wrote: post #13

Any time I read "serverless", I get a violent allergic reaction. There is no such thing, as software requires hardware to run on, and the person or persons who went "serverless" simply chose to stick their head(s) in the sand and punt the OS engineering and hardware design and maintenance off to someone else, hoping that it will just work. But it does not, and eventually there will be an outage and lost money. One ca…

The term "serverless" means you don't need to manage the server resources. There is an "infinite" on-demand amount of resources that are available at request and they are available to run any process you want with a single command/hook/etc. It's a misnomer, but it's no worse than "the cloud" or how "artificial intelligence" has come to mean anything to do with machine learning.

There is no such thing as infinite server resources.

Re: Show HN: Corral – A Serverless MapReduce Framework

#29

Post by Annatar »

jedimastert wrote: post #20
Annatar wrote: post #13

Any time I read "serverless", I get a violent allergic reaction. There is no such thing, as software requires hardware to run on, and the person or persons who went "serverless" simply chose to stick their head(s) in the sand and punt the OS engineering and hardware design and maintenance off to someone else, hoping that it will just work. But it does not, and eventually there will be an outage and lost money. One ca…

>punt the OS engineering and hardware design and maintenance off to someone else Isn't hiring other people who know better than you to do this kind of stuff kind of the point? Like, a lot of people's jobs are based on that idea, including almost everyone in the IT industry. I'm confused by your point. It almost looks like sarcasm. Getting some serious "Poe's Law" here.

One cannot design and write high quality software without throughly understanding the substrate underneath and how to engineer for it; from doing this for over 30 years, I’m telling you it’s impossible.

Re: Show HN: Corral – A Serverless MapReduce Framework

#30

Post by scarface74 »

Annatar wrote: post #27
scarface74 wrote: post #16

Earlier quoted context omitted.

Any time I read "serverless", I get a violent allergic reaction. There is no such thing, as software requires hardware to run on, The meanings of words morph over time. When developers mention serverless everyone knows what it means it that context. Just like when someone says there is a bug in their code no one thinks that there are roaches running around in their computer. and the person or persons who went "server…

Netflix has an army of professional system and kernel engineers who optimize and engineer the OSes they run on; they just use AWS for virtual machine and datacenter capacity.

They use a lot of AWS services - not just virtual machines

- First they use Ubuntu Linux

- SES (Email)

- ElasticSearch (AWS has their own managed version)

- SQS (queueing system)

- S3 (storage)

-auto scaling

Most of the optimization's they do are standard things at scale where they are tuning by measuring performance.

https://youtu.be/89fYOo1V2pA

and the slide deck from the presentation.

https://www.slideshare.net/brendangregg/how-netflix-tunes-ec...

But that's even more of a reason to choose AWS, Netflix has open sourced dozens of tools specifically related to AWS. You get to take advantage of their tools and knowledge.

Post reply on HN