Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Corral – A Serverless MapReduce Framework

github.com

Post reply on HN
Print view View original

11–20 of 39 posts

Re: Show HN: Corral – A Serverless MapReduce Framework

#11

Post by eranation »

bcongdon wrote: post #2

Hi HN, author here. Corral is my attempt at a performant, easy-to-deploy MapReduce. Unlike traditional frameworks like Hadoop, it uses AWS Lambda for execution and is “serverless” as a result. It was initially kicked-off by AWS adding Lambda support for Go, but draws on experience I’ve had using Lambda tools like Zappa[1] and Serverless[2] in the past. I think there’s a lot of interesting applications for using funct…

Nice! Always thought this would be cool. Few thought questions though: how do you get things like consistent hashing? Spark for example can shuffle data somewhat efficiently by sending the data to the right node / getting data from the right node by the hash key, right? How in a serverless stateless world you call a specific Serverless function instance? Assuming it can’t be done, arent you losing performance gained by data locality? Eg data has to be saved in a massive and efficient key value store? Isn’t it much slower than spark’s in memory / data locality (bring the compute to the data and not vice versa). Would love to see benchmarks on this. This is the future IMHO... well done.

Re: Show HN: Corral – A Serverless MapReduce Framework

#13

Post by Annatar »

Any time I read "serverless", I get a violent allergic reaction. There is no such thing, as software requires hardware to run on, and the person or persons who went "serverless" simply chose to stick their head(s) in the sand and punt the OS engineering and hardware design and maintenance off to someone else, hoping that it will just work. But it does not, and eventually there will be an outage and lost money. One can punt this responsibility off to someone else, but there will be consequences.

If you want reliable infrastructure, first you must become a master system, database and network administrator, then you must apprentice with a mentor to become a system engineer, and finally after several decades of practitioning as one, you will have enough experience and insight to become a system architect. There is no way around that, no punting will help.

Re: Show HN: Corral – A Serverless MapReduce Framework

#14

Post by cle »

Nice!

The use of S3 ListObjects is an immediate deal breaker though, its eventual consistency can cause silent data corruption. To avoid the List, you'd need to write a file manifest somewhere that contains a list of all S3 objects. If it were me, I'd use DynamoDB and append keys to a StringSet on a single item (if you use S3 for the manifest, it needs to be a single object, which means you need to aggregate the keys first, which sounds tricky with Lambda). You'll hit a scaling limit with DDB's item size limit, if you want to avoid that, perhaps writing an item per mapper with the same hash key and a different range key might be better, then you'd do a strongly consistent query to reconstruct the manifest.

Re: Show HN: Corral – A Serverless MapReduce Framework

#15

Post by neolefty »

amelius wrote: post #8

Why call it "serverless" if I need to provide stuff like: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION, AWS_TEST_BUCKET

Serverless doesn't mean there isn't a server—it just means you don't have to manage server instances. You can treat all of AWS as your server. Which means there must be a way to separate your work from everybody else's work who is sharing AWS with you. Thus the configuration.

Re: Show HN: Corral – A Serverless MapReduce Framework

#16

Post by scarface74 »

Annatar wrote: post #13

Any time I read "serverless", I get a violent allergic reaction. There is no such thing, as software requires hardware to run on, and the person or persons who went "serverless" simply chose to stick their head(s) in the sand and punt the OS engineering and hardware design and maintenance off to someone else, hoping that it will just work. But it does not, and eventually there will be an outage and lost money. One ca…

Any time I read "serverless", I get a violent allergic reaction. There is no such thing, as software requires hardware to run on,

The meanings of words morph over time. When developers mention serverless everyone knows what it means it that context. Just like when someone says there is a bug in their code no one thinks that there are roaches running around in their computer.

and the person or persons who went "serverless" simply chose to stick their head(s) in the sand and punt the OS engineering and hardware design and maintenance off to someone else,

When I write a program, I'm not writing assembly language. I'm also "sticking my head in the sand" about the how assembly works. AWS has a whole team of people that know how to do that stuff.

hoping that it will just work. But it does not, and eventually there will be an outage and lost money. One can punt this responsibility off to someone else, but there will be consequences.

AWS is probably more reliable than what you could do on prem or at a colo,

If you want reliable infrastructure, first you must become a master system, database and network administrator, then you must apprentice with a mentor to become a system engineer, and finally after several decades of practitioning as one, you will have enough experience and insight to become a system architect. There is no way around that, no punting will help.

Tell that to Netflix. They host everything on AWS. They purposefully moved from an on prem architecture to AWS because they realized where their core competence was.

Re: Show HN: Corral – A Serverless MapReduce Framework

#17

Post by mjn »

Very different technical construction, but reminds me a bit of Joyent's Manta, that also lets you run map-reduce jobs over an object store without explicit server management: https://news.ycombinator.com/item?id=5939340

Source code: https://github.com/joyent/manta

I believe the cloud version has since been renamed to "Converged Analytics", so this is probably the same thing: https://www.joyent.com/triton/analytics

Re: Show HN: Corral – A Serverless MapReduce Framework

#18

Post by rockostrich »

Annatar wrote: post #13

Any time I read "serverless", I get a violent allergic reaction. There is no such thing, as software requires hardware to run on, and the person or persons who went "serverless" simply chose to stick their head(s) in the sand and punt the OS engineering and hardware design and maintenance off to someone else, hoping that it will just work. But it does not, and eventually there will be an outage and lost money. One ca…

The term "serverless" means you don't need to manage the server resources. There is an "infinite" on-demand amount of resources that are available at request and they are available to run any process you want with a single command/hook/etc.

It's a misnomer, but it's no worse than "the cloud" or how "artificial intelligence" has come to mean anything to do with machine learning.

Re: Show HN: Corral – A Serverless MapReduce Framework

#19

Post by jedimastert »

I'm a complete novice when it comes to distributed computing, so I'm not too sure I get the idea of a MapReduce framework. I'm gonna try to suss it out here as I understand it, and if someone could correct me where I'm wrong, that'd be awesome.

First off, when I see "map" and "reduce" I think of the functional programming/data processing equivalents of mapping, meaning to apply a function to every element in a set (like capitalizing strings or dividing everything by two or something) and reducing, meaning to iterate over a set, processing it and combining it with some accumulator (like taking a sum).

What a MapReduce framework seems to do is take these two function and run them in parallel, splitting the data to take advantage of the independent nature of these two functions. Data can be split however is convenient because the map function doesn't need to worry about another data than itself, and run in as many processes you can manage. Any mapped-data can be put into parallel reduce processes, which can be run in any order because the order of the data shouldn't matter.

All of that I get (although if I'm wrong that might explain why I'm confused). I guess my main confusion is why the reduce function doesn't really fit with the idea that I just put forward. I would think that the reduce function would need some sort of "accumulator" input, and that you'd only get one thing as an output, as opposed to more files of data. Perhaps the idea is that the reduce is actually just any function that can only work on post-mapped functions, or even the only one that's supposed to change state in some way?

Can anyone shed some light on my confusion? What is the reduce function actually supposed to do, if not what I just laid out.

Re: Show HN: Corral – A Serverless MapReduce Framework

#20

Post by jedimastert »

Annatar wrote: post #13

Any time I read "serverless", I get a violent allergic reaction. There is no such thing, as software requires hardware to run on, and the person or persons who went "serverless" simply chose to stick their head(s) in the sand and punt the OS engineering and hardware design and maintenance off to someone else, hoping that it will just work. But it does not, and eventually there will be an outage and lost money. One ca…

>punt the OS engineering and hardware design and maintenance off to someone else

Isn't hiring other people who know better than you to do this kind of stuff kind of the point? Like, a lot of people's jobs are based on that idea, including almost everyone in the IT industry. I'm confused by your point. It almost looks like sarcasm. Getting some serious "Poe's Law" here.

Post reply on HN