Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MapReduce is making a comeback

estuary.dev

Post reply on HN
Print view View original

31–39 of 39 posts

Re: MapReduce is making a comeback

#31

Post by vidarh »

whalesalad wrote: post #29

Did this ever go away? 99% of programs do map/reduce all-day long.

Yes, but when talking about MapReduce we generally talk about distributed frameworks for doing it on big data.

But most people don't have "big data" in the sense of having data that requires more than a single machine to process.

Most people who think they have "big data" still don't have big data (e.g. I've done work on datasets where people insisted on using "big data" solutions when it could all easily fit in a Postgres instance with or without a columnar store with most of the working set cached in memory for a fraction of the cost).

It "went away" in the sense that more people realised they could avoid it with a few simple steps (e.g. pre-processing during ingestion), and/or fit the data they needed on fast-growing individual servers, and so the number of people continuing to use it more closer approximated the set of people who actually work on big data.

For those who actually needed it, it of course never went away.

Re: MapReduce is making a comeback

#33

Post by psfried »

abeppu wrote: post #26

I question the premise that MapReduce really ever went away. Many migrated away from Hadoop, but in frameworks that succeeded it, MapReduce was still a core pattern. And in some cases, moving away from Hadoop wasn't ideal because later frameworks still got some things wrong. Maybe we stopped talking about MapReduce because we were focused on new patterns and challenges -- how to support many complex jobs and pipeline…

First let me say that I think Timely Dataflow and Materialize are both super cool. The two approaches are quite different, in part because they solve slightly different problems. Or maybe it's more fair to say that they think of the world in somewhat different ways. Probably most of the differences can be traced back to how Timely Dataflow relies on the expiration of timestamps in order to coordinate updates to its results. You can read the details on that in their docs (https://timelydataflow.github.io/timely-dataflow/chapter_5/c...).

I think a reasonable TLDR might be to say that continuous map reduce has a better fault-tolerance story, while timely dataflow is more efficient for things like reactive joins. They both have their purpose, though, and I imagine that both Flow and Materialize will go on to co-exists as successful products.

Re: MapReduce is making a comeback

#34

Post by brundolf »

vidarh wrote: post #31
whalesalad wrote: post #29

Did this ever go away? 99% of programs do map/reduce all-day long.

Yes, but when talking about MapReduce we generally talk about distributed frameworks for doing it on big data. But most people don't have "big data" in the sense of having data that requires more than a single machine to process. Most people who think they have "big data" still don't have big data (e.g. I've done work on datasets where people insisted on using "big data" solutions when it could all easily fit in a Po…

Forgive me if this is naive, but could smaller-scale cases be served by a version that uses the MapReduce model as a way to cleanly break up operations across cores instead of machines? Or do the benefits of the model become mostly irrelevant in that case?

I'm sure it wouldn't take the form of a dedicated process; probably just a language-agnostic programming pattern

Re: MapReduce is making a comeback

#35

Post by exikyut »

lumost wrote: post #10
ziggus wrote: post #2

MapReduce is a useful tool in some contexts, but the the hype surrounding it always felt like people patting themselves on the back for re-inventing relational algebra.

The big hype was always due to the ability to shed large oracle based data warehouses. When Hadoop was full of hype in 2010 Oracle was charging 150k/cpu core/year for a rac cluster license. Considering that oracle is not in fact magic, this meant that a large number of firms were spending 7-8 figures annually on oracle licenses. Map reduce/Hadoop was the first accepted alternative that didn’t involve spending outrage…

Enlightenment moment.

Con: OSS is less optimized than proprietary solution, requiring bigger hardware

Pro: OSS allows you to buy bigger hardware, use all of it without logical restrictions, and scale infinitely beyond the arbitrary point you were locked into with licensing.

And then the new-found efficiency frees up time to discover/identify $(x,)xxx,xxx+ in manual work that can also now be done with your new-found compute...

Wow. Way to prevent us from progressing beyond the industrial revolution.

($catchup_speed++)

Re: MapReduce is making a comeback

#36

Post by dekhn »

brundolf wrote: post #34
vidarh wrote: post #31

Earlier quoted context omitted.

Yes, but when talking about MapReduce we generally talk about distributed frameworks for doing it on big data. But most people don't have "big data" in the sense of having data that requires more than a single machine to process. Most people who think they have "big data" still don't have big data (e.g. I've done work on datasets where people insisted on using "big data" solutions when it could all easily fit in a Po…

Forgive me if this is naive, but could smaller-scale cases be served by a version that uses the MapReduce model as a way to cleanly break up operations across cores instead of machines? Or do the benefits of the model become mostly irrelevant in that case? I'm sure it wouldn't take the form of a dedicated process; probably just a language-agnostic programming pattern

most of the benefits go away, but yes, you can do this. MapReduce had a flag to use multiple cores for multiple workers on a machine and this was often the way to get the greatest throughput.

Re: MapReduce is making a comeback

#37

Post by citrin_ru »

vidarh wrote: post #31
whalesalad wrote: post #29

Did this ever go away? 99% of programs do map/reduce all-day long.

Yes, but when talking about MapReduce we generally talk about distributed frameworks for doing it on big data. But most people don't have "big data" in the sense of having data that requires more than a single machine to process. Most people who think they have "big data" still don't have big data (e.g. I've done work on datasets where people insisted on using "big data" solutions when it could all easily fit in a Po…

If all you have is a hummer everything look like a nail: when Hadoop first appeared there was almost no other open source systems to process 'big data' and it was widely adopted. Now there are many options to choose from. We don't have to use map-reduce for every task which could be solved using map-redude. E. g. for some tasks a columnar store, like ClickHouse is a better fit.

Re: MapReduce is making a comeback

#38

Post by CRConrad »

citrin_ru wrote: post #37
vidarh wrote: post #31

Earlier quoted context omitted.

Yes, but when talking about MapReduce we generally talk about distributed frameworks for doing it on big data. But most people don't have "big data" in the sense of having data that requires more than a single machine to process. Most people who think they have "big data" still don't have big data (e.g. I've done work on datasets where people insisted on using "big data" solutions when it could all easily fit in a Po…

If all you have is a hummer everything look like a nail: when Hadoop first appeared there was almost no other open source systems to process 'big data' and it was widely adopted. Now there are many options to choose from. We don't have to use map-reduce for every task which could be solved using map-redude. E. g. for some tasks a columnar store, like ClickHouse is a better fit.

If all you have is a Hummer everything looks like an enemy vehicle.

Re: MapReduce is making a comeback

#39

Post by CRConrad »

prasadjoglekar wrote: post #11
amelius wrote: post #5

Earlier quoted context omitted.

It was more about scaling than about relational algebra though.

Exactly this. I remember in 2007 being able to process TBs of data on commodity hardware with Hadoop. You got decent throughput, decent fault tolerance out of the box wrapped in Java that many average software developers (yours truly included) were comfortable with. You could scale data and people. It dramatically reduced the cost of entry for many ad-tech applications.

> It dramatically reduced the cost of entry for many ad-tech applications.

You say that as if it was unequivocally a point in its favour.

Post reply on HN