31–40 of 70 posts
Earlier quoted context omitted.
It’s crazy to think how old I am now. But give it 20 more years and you’ll come around.
Being old doesn’t automatically make you more right. You don’t get wisdom as a birthday gift.
Earlier quoted context omitted.
It’s crazy to think how old I am now. But give it 20 more years and you’ll come around.
Being old doesn’t automatically make you more right. You don’t get wisdom as a birthday gift.
I don't know about automatically, but definitely more likely.
I feel like that’s kind of like saying we don’t use Assembly anymore now that we have C. We’ve just built higher level abstractions on top of it.
Why use map:reduce when you can have an entire DAG for fanout/in?
Earlier quoted context omitted.
Being old doesn’t automatically make you more right. You don’t get wisdom as a birthday gift.
It does depend on experience, which everyone gets one way or the other. I don't know about automatically, but definitely more likely.
The "streaming systems" book answers your question and more: https://www.oreilly.com/library/view/streaming-systems/97814... . It gives you a history of how batch processing started with MapReduce, and how attempts at scaling by moving towards streaming systems gave us all the subsequent frameworks (Spark, Beam, etc.). As for the framework called MapReduce, it isn't used much, but its descendant https://beam.apache.o…
Earlier quoted context omitted.
It’s crazy to think how old I am now. But give it 20 more years and you’ll come around.
Being old doesn’t automatically make you more right. You don’t get wisdom as a birthday gift.
oh right, new language. that’ll definitely fix it. :eyeroll:
(2nd user & developer of spark here). It depends on what you ask. MapReduce the framework is proprietary to Google, and some pipelines are still running inside google. MapReduce as a concept is very much in use. Hadoop was inspired by MapReduce. Spark was originally built around the primitives of MapReduce, and you see still see that in the description of its operations (exchange, collect). However, spark and all the…
There really was always only Map and Shuffle (Reduce is just Shuffle+Map; also another name for Shuffle is GroupByKey). And you see those primitives under the hood of most parallel systems.
The correct language for querying data is, as always, SQL. No one cares about the implementation details. “I have data and I know SQL. What is it about your database that makes retrieving it better?” Any other paradigm is going to be a niche at best, likely outright fail.
I know I'm replying to a troll comment, but: > “I have data and I know SQL. What is it about your database that makes retrieving it better?” Because my data comes from a variety of unstructured, possibly dirty sources which need cleaning and transforming before they can be made sense of.
Ignore that statement, and fight the uphill battle.
MapReduce was basically a very verbose/imperative way to perform scalable, larger than memory aggregate-by-key operation. It was necessary as a first step, but as soon as we had better abstraction, everyone stopped using it directly except for legacy maintenance of course.