MapReduce is making a comeback • Hacker News Board

Re: MapReduce is making a comeback

#11

Post by prasadjoglekar » Tue, Aug 10, 2021, 2:03 PM UTC

amelius wrote: post #5

ziggus wrote: post #2

MapReduce is a useful tool in some contexts, but the the hype surrounding it always felt like people patting themselves on the back for re-inventing relational algebra.

It was more about scaling than about relational algebra though.

Exactly this. I remember in 2007 being able to process TBs of data on commodity hardware with Hadoop. You got decent throughput, decent fault tolerance out of the box wrapped in Java that many average software developers (yours truly included) were comfortable with. You could scale data and people.

It dramatically reduced the cost of entry for many ad-tech applications.

Re: MapReduce is making a comeback

#12

Post by _huayra_ » Tue, Aug 10, 2021, 2:05 PM UTC

hadrianpaulo wrote: post #3

Interesting but how is this as an alternative to Apache Flink's stream processing model?

dataflow things like Flink (or even better differential datatflow [0]) are far more flexible and subsume map-reduce. This article feels like hyping up the durability of the Model T.

[0] https://github.com/TimelyDataflow/differential-dataflow

Re: MapReduce is making a comeback

#13

Post by londons_explore » Tue, Aug 10, 2021, 2:10 PM UTC

In reality, such continuous mapreduce jobs lead to unchangeable code and versioning nightmares.

Imagine you want to change part of your pipelines logic. Now either all data needs to be reprocessed (expensive, depends on you having retained past data, will your low latency continuous pipeline keep running while the backlog is cleared, is the code really idempotent or will a rerun lead to half the records failing to be reprocessed?). Or you need to not reprocess old data (now there is inconsistency in historical records, what do you do if you make a bad release which just outputs zeros?).

In any real organisation, you'll need both approaches. And it'll end up a mess with versions of code and versions of data. Now some customer comes along and demands a GDPR deletion of their session records and you have no way to even find all the versions of all the copies of the records let alone delete them and make everything else consistent...

Re: MapReduce is making a comeback

#14

Post by _huayra_ » Tue, Aug 10, 2021, 2:10 PM UTC

amelius wrote: post #5

ziggus wrote: post #2

MapReduce is a useful tool in some contexts, but the the hype surrounding it always felt like people patting themselves on the back for re-inventing relational algebra.

It was more about scaling than about relational algebra though.

The scale factor was really overblown and without caveats. If you have Google-scale computations where it is not unlikely that at least 1 / 10k machines serving that request will bite the dust during the query, then of course MapReduce makes sense.

However, in most other cases there are now far better alternatives (although tbh I'm not sure how many were around when MapReduce was introduced).

The main limitation around mapreduce is the barries imposed by the shuffle stage and after the end of the reduce if chaining together multiple mapreduce operations. Dataflow frameworks remove these barriers to various degrees, which often lowers latency and can improve resource utilization.

Re: MapReduce is making a comeback

#15

Post by jgraettinger1 » Tue, Aug 10, 2021, 2:11 PM UTC

hadrianpaulo wrote: post #3

Interesting but how is this as an alternative to Apache Flink's stream processing model?

A big difference is the removal of windowing: Flink lets you aggregate or join events only so long as they arrive in the same temporally-bound window. You're required to have a window, and it's core to the semantics of your workflow.

Flow's model doesn't use windows, and allows for long-distance (in time) joins and aggregations. There's no concept of "late" data in Flow: it just keeps on updating the desired aggregate.

Re: MapReduce is making a comeback

#16

Post by rytis » Tue, Aug 10, 2021, 2:11 PM UTC

lumost wrote: post #10

ziggus wrote: post #2

MapReduce is a useful tool in some contexts, but the the hype surrounding it always felt like people patting themselves on the back for re-inventing relational algebra.

The big hype was always due to the ability to shed large oracle based data warehouses. When Hadoop was full of hype in 2010 Oracle was charging 150k/cpu core/year for a rac cluster license. Considering that oracle is not in fact magic, this meant that a large number of firms were spending 7-8 figures annually on oracle licenses. Map reduce/Hadoop was the first accepted alternative that didn’t involve spending outrage…

And there's also an added cost of engineers supporting all that hardware and the Hadoop components running on it.

Re: MapReduce is making a comeback

#17

Post by zippy5 » Tue, Aug 10, 2021, 2:14 PM UTC

ziggus wrote: post #2

MapReduce is a useful tool in some contexts, but the the hype surrounding it always felt like people patting themselves on the back for re-inventing relational algebra.

My interpretation is that is why it’s so brilliant.

It’s incredibly simple for the end user conceptually but encapsulates optimizing processing across a distributed file system, fault tolerance, shuffling key value pairs, job stage planning, handling intermediates ect.

Hadoop a big data framework that reduces the level of competence required to write data pipelines because it was able to hide a massive amount of complexity behind the map reduce abstraction.

Id even argue that hive, snowflake, and other sql data warehouses have taken this idea further, where most sql primitives can be implemented as map reduce derivatives. With this next level of abstraction, dbas and non-engineers are witting map reduce computations.

I think my point is that abstractions like map reduce have had a democratizing effect on who can implement high scale data processing and their value is that they took something incredibly complex and made it simple.

Re: MapReduce is making a comeback

#18

Post by willvarfar » Tue, Aug 10, 2021, 2:26 PM UTC

zippy5 wrote: post #17

ziggus wrote: post #2

MapReduce is a useful tool in some contexts, but the the hype surrounding it always felt like people patting themselves on the back for re-inventing relational algebra.

My interpretation is that is why it’s so brilliant. It’s incredibly simple for the end user conceptually but encapsulates optimizing processing across a distributed file system, fault tolerance, shuffling key value pairs, job stage planning, handling intermediates ect. Hadoop a big data framework that reduces the level of competence required to write data pipelines because it was able to hide a massive amount of comp…

It reminds me more of timescale’s continuous aggregates and the new snowflake slayer, firebolt’s, aggregation indexes.

Re: MapReduce is making a comeback

#19

Post by dekhn » Tue, Aug 10, 2021, 2:26 PM UTC

alexnewman wrote: post #6

Every section in this article is a mistake. For instance - Map Reduce was not invented in 2004. it's a technique from the punch card days. - Map reduce is not solely useful because it allows you to delete source data, it's a trivial method of parallel processing. It's not the most trivial or most modern. Just common

Really, they should be calling it "Distributed MapShuffleReduce", I let Sanjay and Jeff know this (in person) but they didn't really seem to care. Neither map nor reduce is a method of parallel processing, the shuffle stage is.

Re: MapReduce is making a comeback

#20

Post by psfried » Tue, Aug 10, 2021, 2:29 PM UTC

londons_explore wrote: post #13

In reality, such continuous mapreduce jobs lead to unchangeable code and versioning nightmares. Imagine you want to change part of your pipelines logic. Now either all data needs to be reprocessed (expensive, depends on you having retained past data, will your low latency continuous pipeline keep running while the backlog is cleared, is the code really idempotent or will a rerun lead to half the records failing to be…

Versioning is indeed an issue, but that's the case for anything with long-lived state. Our current rely on JSON schemas, TypeScript, and built-in testing support to help ensure compatibility. Those things actually help quite a bit in practice. But I think we may also want to build some more powerful features for managing versions of datasets, since there's a real need there, regardless of the processing model you use to derive the data.