Live data from Hacker News

Ask HN: Does (or why does) anyone use MapReduce anymore?

news.ycombinator.com

21–30 of 70 posts

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#21
The "streaming systems" book answers your question and more: https://www.oreilly.com/library/view/streaming-systems/97814.... It gives you a history of how batch processing started with MapReduce, and how attempts at scaling by moving towards streaming systems gave us all the subsequent frameworks (Spark, Beam, etc.).

As for the framework called MapReduce, it isn't used much, but its descendant https://beam.apache.org very much is. Nowadays people often use "map reduce" as a shorthand for whatever batch processing system they're building on top of.

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#22

Earlier quoted context omitted.

I know I'm replying to a troll comment, but: > “I have data and I know SQL. What is it about your database that makes retrieving it better?” Because my data comes from a variety of unstructured, possibly dirty sources which need cleaning and transforming before they can be made sense of.

> Because my data comes from a variety of unstructured, possibly dirty sources which need cleaning and transforming before they can be made sense of. Seattle data guy had a great end of year top 10 memes post recently and one of them went like this > oh cool you’ve hired a data scientist. so you have a collection of reliable and easy to query data sources, right? > … > you do have a collection of reliable and easy to…

That's true. With dbt (=SQL+Jinja-Templating in an opionated framework) a large SQL codebase actually becomes maintainable. If in any way possible I'll usually load my raw data in an OLAP table (Snowflake, BigQuery) and do all the transforms there. At least for JSON data that works really well. Combine it with dbt tests and you're safe.

See https://www.getdbt.com/

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#23
From what I recall Map:Reduce was basically a weird representation of DAG based pipeline.

I think because it looked sorta like an automatic dictionary to multi-thread converter it became popular. But its pretty useless unless you know how to split up and process your data.

basically, if you can cut your data up into a queue, you can MapReduce. But, most pipelines are more complex than that, so you probably need a proper DAG with dependencies.

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#24
post #19

Earlier quoted context omitted.

It’s crazy to think how old I am now. But give it 20 more years and you’ll come around.

Being old doesn’t automatically make you more right. You don’t get wisdom as a birthday gift.

But if you did a lot of things you might know what does not work and why. Similar to science articles, nobody talks enough about "X technology does not work for Y domain", so a lot of people try to "reinvent the wheel" only to realize "X does not work for Y". Occasionally there is a surprise (because of some technology advancement) but being old definitely gives you more insight in what can go wrong.

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#25
MapReduce was basically a very verbose/imperative way to perform scalable, larger than memory aggregate-by-key operation.

It was necessary as a first step, but as soon as we had better abstraction, everyone stopped using it directly except for legacy maintenance of course.

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#26
post #25

MapReduce was basically a very verbose/imperative way to perform scalable, larger than memory aggregate-by-key operation. It was necessary as a first step, but as soon as we had better abstraction, everyone stopped using it directly except for legacy maintenance of course.

Can you point, please, to the better abstractions?

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#27
post #25

MapReduce was basically a very verbose/imperative way to perform scalable, larger than memory aggregate-by-key operation. It was necessary as a first step, but as soon as we had better abstraction, everyone stopped using it directly except for legacy maintenance of course.

Can you point, please, to the better abstractions?

SQL comes to mind.

Every time you run an SQL query on BigQuery, for example, you are executing those same fundamental map shuffle primitives on underlying data, it's just that the interface is very different.

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#28
Now you rarely use basic MapReduce primitives, you have another layer of abstraction that can run on infrastructure that was running MR jobs before. This infrastructure allows to efficiently allocate some compute resources for "long" running tasks in a large cluster with respect to memory/cpu/network and other constraints. So basically schedulers of MapReduce jobs and cluster management tools became that good, because MR methodology had trivial abstractions, but required efficient implementation to make it work seamlessly.

Abstraction layers on top of this infrastructure now can optimize pipeline as a whole by merging several steps into one when possible, add combiners(partial reduce before shuffle). It requires whole processing pipeline to be defined in more specific operations. Some of them propose to use SQL to formulate task, but it can be done using other primitives. And given this pipeline it is easy to implement optimizations making whole system much more user-friendly and efficient compared to MapReduce, when user has to think about all the optimizations and implement them inside single map/reduce/(combine) operations.

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#29

You have no idea how long the tail of legacy MR-based daily stat aggregation workflows is in BigCorps. The batch daily log processor jobs will last longer than Fortran. Longer than Cobol. Longer than earth itself.

> The batch daily log processor jobs will last longer than Fortran. Longer than Cobol. Nonsense... They'll end at the same time. Which is approximately concurrently with the universe.

Or 2037.

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#30

I feel like that’s kind of like saying we don’t use Assembly anymore now that we have C. We’ve just built higher level abstractions on top of it.

Yeah, that's exactly how I read the question, i.e. analogous to "does anyone still code in assembly, or has everyone switched to using abstractions?" and I think it's a very interesting one.
Post reply on HN