Excluding the Hadoop ecosystem, I see some references to MapReduce in other database and analysis tools (e.g., MatLab). My perception was that Spark completely superseded MapReduce. Are there just different implementations of MapReduce and the one that Hadoop implemented was replaced by Spark?
Ask HN: Does (or why does) anyone use MapReduce anymore?
1–10 of 70 posts
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#2That's my understanding. MR is very simplistic and awkward/impossible to express many problems in, whereas dataflow processors like Spark and Apache Beam support creating complex DAGs of rich set of operators for grouping, windowing, joining, etc. that you just don't have in MR. You can do MR within a DAG, so you could say that dataflows are a generalization or superset of the MR model.
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#3That's my understanding. MR is very simplistic and awkward/impossible to express many problems in, whereas dataflow processors like Spark and Apache Beam support creating complex DAGs of rich set of operators for grouping, windowing, joining, etc. that you just don't have in MR. You can do MR within a DAG, so you could say that dataflows are a generalization or superset of the MR model.
> You can do MR within a DAG, so you could say that dataflows are a generalization or superset of the MR model.
I think it's the opposite of this. MapReduce is a very generic mechanism for splitting computation up so that it can be distributed. It would be possible to build Spark/Beam and all their higher level DAG components out of MapReduce operations.
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#4I feel like that’s kind of like saying we don’t use Assembly anymore now that we have C. We’ve just built higher level abstractions on top of it.
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#5That's my understanding. MR is very simplistic and awkward/impossible to express many problems in, whereas dataflow processors like Spark and Apache Beam support creating complex DAGs of rich set of operators for grouping, windowing, joining, etc. that you just don't have in MR. You can do MR within a DAG, so you could say that dataflows are a generalization or superset of the MR model.
> You can do MR within a DAG, so you could say that dataflows are a generalization or superset of the MR model. I think it's the opposite of this. MapReduce is a very generic mechanism for splitting computation up so that it can be distributed. It would be possible to build Spark/Beam and all their higher level DAG components out of MapReduce operations.
I don't mean generalization that way. Dataflow operators can be expressed as MR as the underlying primitive, as you say. But MR itself, as described in the original paper at least, only has the two stages, map and reduce; it's not a dataflow system. And it turns out people want dataflow systems, not hand-code MR and do the DAG manually.
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#6That's my understanding. MR is very simplistic and awkward/impossible to express many problems in, whereas dataflow processors like Spark and Apache Beam support creating complex DAGs of rich set of operators for grouping, windowing, joining, etc. that you just don't have in MR. You can do MR within a DAG, so you could say that dataflows are a generalization or superset of the MR model.
> You can do MR within a DAG, so you could say that dataflows are a generalization or superset of the MR model. I think it's the opposite of this. MapReduce is a very generic mechanism for splitting computation up so that it can be distributed. It would be possible to build Spark/Beam and all their higher level DAG components out of MapReduce operations.
I'm not sure what you describe is the opposite?
I mean, you can implement function calls (and other control flow operators like exceptions or loops) as GOTOs and conditional branches, and that's what your compiler does.
But that doesn't really mean it's useful to think of GOTOs being the generalisation.
Most of the time, it's just the opposite: you can think of a GOTO as a very specific kind of function call, a tail-call without any arguments. See eg https://www2.cs.sfu.ca/CourseCentral/383/havens/pubs/lambda-...
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#7You have no idea how long the tail of legacy MR-based daily stat aggregation workflows is in BigCorps.
The batch daily log processor jobs will last longer than Fortran. Longer than Cobol. Longer than earth itself.
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#8(2nd user & developer of spark here). It depends on what you ask.
MapReduce the framework is proprietary to Google, and some pipelines are still running inside google.
MapReduce as a concept is very much in use. Hadoop was inspired by MapReduce. Spark was originally built around the primitives of MapReduce, and you see still see that in the description of its operations (exchange, collect). However, spark and all the other modern frameworks realized that:
- users did not care mapping and reducing, they wanted higher level primitives (filtering, joins, ...)
- mapreduce was great for one-shot batch processing of data, but struggled to accomodate other very common use cases at scale (low latency, graph processing, streaming, distributed machine learning, ...). You can do it on top of mapreduce, but if you really start tuning for the specific case, you end up with something rather different. For example, kafka (scalable streaming engine) is inspired by the general principles of MR but the use cases and APIs are now quite different.
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#9(2nd user & developer of spark here). It depends on what you ask. MapReduce the framework is proprietary to Google, and some pipelines are still running inside google. MapReduce as a concept is very much in use. Hadoop was inspired by MapReduce. Spark was originally built around the primitives of MapReduce, and you see still see that in the description of its operations (exchange, collect). However, spark and all the…
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#10You have no idea how long the tail of legacy MR-based daily stat aggregation workflows is in BigCorps. The batch daily log processor jobs will last longer than Fortran. Longer than Cobol. Longer than earth itself.
> The batch daily log processor jobs will last longer than Fortran. Longer than Cobol.
Nonsense... They'll end at the same time. Which is approximately concurrently with the universe.