As for the framework called MapReduce, it isn't used much, but its descendant https://beam.apache.org very much is. Nowadays people often use "map reduce" as a shorthand for whatever batch processing system they're building on top of.
Ask HN: Does (or why does) anyone use MapReduce anymore?
21–30 of 70 posts
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#22Earlier quoted context omitted.
I know I'm replying to a troll comment, but: > “I have data and I know SQL. What is it about your database that makes retrieving it better?” Because my data comes from a variety of unstructured, possibly dirty sources which need cleaning and transforming before they can be made sense of.
> Because my data comes from a variety of unstructured, possibly dirty sources which need cleaning and transforming before they can be made sense of. Seattle data guy had a great end of year top 10 memes post recently and one of them went like this > oh cool you’ve hired a data scientist. so you have a collection of reliable and easy to query data sources, right? > … > you do have a collection of reliable and easy to…
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#23I think because it looked sorta like an automatic dictionary to multi-thread converter it became popular. But its pretty useless unless you know how to split up and process your data.
basically, if you can cut your data up into a queue, you can MapReduce. But, most pipelines are more complex than that, so you probably need a proper DAG with dependencies.
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#24Earlier quoted context omitted.
It’s crazy to think how old I am now. But give it 20 more years and you’ll come around.
Being old doesn’t automatically make you more right. You don’t get wisdom as a birthday gift.
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#25It was necessary as a first step, but as soon as we had better abstraction, everyone stopped using it directly except for legacy maintenance of course.
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#26MapReduce was basically a very verbose/imperative way to perform scalable, larger than memory aggregate-by-key operation. It was necessary as a first step, but as soon as we had better abstraction, everyone stopped using it directly except for legacy maintenance of course.
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#27MapReduce was basically a very verbose/imperative way to perform scalable, larger than memory aggregate-by-key operation. It was necessary as a first step, but as soon as we had better abstraction, everyone stopped using it directly except for legacy maintenance of course.
Can you point, please, to the better abstractions?
Every time you run an SQL query on BigQuery, for example, you are executing those same fundamental map shuffle primitives on underlying data, it's just that the interface is very different.
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#28Abstraction layers on top of this infrastructure now can optimize pipeline as a whole by merging several steps into one when possible, add combiners(partial reduce before shuffle). It requires whole processing pipeline to be defined in more specific operations. Some of them propose to use SQL to formulate task, but it can be done using other primitives. And given this pipeline it is easy to implement optimizations making whole system much more user-friendly and efficient compared to MapReduce, when user has to think about all the optimizations and implement them inside single map/reduce/(combine) operations.
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#29You have no idea how long the tail of legacy MR-based daily stat aggregation workflows is in BigCorps. The batch daily log processor jobs will last longer than Fortran. Longer than Cobol. Longer than earth itself.
> The batch daily log processor jobs will last longer than Fortran. Longer than Cobol. Nonsense... They'll end at the same time. Which is approximately concurrently with the universe.
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#30I feel like that’s kind of like saying we don’t use Assembly anymore now that we have C. We’ve just built higher level abstractions on top of it.