At a high level, most distributed data systems look something like MapReduce, and that's really just fancy divide-and-conquer. It's hard to reason about, and most data at this size is tabular, so you're usually better off using something where you can write a SQL query and let the query engine do the low-level map-reduce work.
Ask HN: Does (or why does) anyone use MapReduce anymore?
11–20 of 70 posts
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#12The concept is quite alive, and the fancy deep learning have it: jax.lax.map, jax.lax.reduce.
It's going to stay because it is useful:
Any operation that you can express with an associative behavior is automatically parallelizeable. And both in Spark and Torch/Jax this means scalable to a cluster, with the code going to the data. This is the unfair advantage of solving bigger problems.
If you were talking about the Hadoop ecosystem, then yes Spark pretty much nailed it and is dominant (no need to have another implementation)
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#13(2nd user & developer of spark here). It depends on what you ask. MapReduce the framework is proprietary to Google, and some pipelines are still running inside google. MapReduce as a concept is very much in use. Hadoop was inspired by MapReduce. Spark was originally built around the primitives of MapReduce, and you see still see that in the description of its operations (exchange, collect). However, spark and all the…
There really was always only Map and Shuffle (Reduce is just Shuffle+Map; also another name for Shuffle is GroupByKey). And you see those primitives under the hood of most parallel systems.
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#14The correct language for querying data is, as always, SQL. No one cares about the implementation details.
“I have data and I know SQL. What is it about your database that makes retrieving it better?”
Any other paradigm is going to be a niche at best, likely outright fail.
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#15I feel like a lot of the underlying concepts of mapreduce live large in multi-threaded applications even on a single machine.
Its definitely not a dead concept, I guess its not sexy to talk about though.
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#16The correct language for querying data is, as always, SQL. No one cares about the implementation details. “I have data and I know SQL. What is it about your database that makes retrieving it better?” Any other paradigm is going to be a niche at best, likely outright fail.
Spark is really failing, all right.
SQL lacks type safety, testability, and composability.
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#17The correct language for querying data is, as always, SQL. No one cares about the implementation details. “I have data and I know SQL. What is it about your database that makes retrieving it better?” Any other paradigm is going to be a niche at best, likely outright fail.
Spark is really failing, all right. SQL lacks type safety, testability, and composability.
It’s crazy to think how old I am now. But give it 20 more years and you’ll come around.
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#18The correct language for querying data is, as always, SQL. No one cares about the implementation details. “I have data and I know SQL. What is it about your database that makes retrieving it better?” Any other paradigm is going to be a niche at best, likely outright fail.
I know I'm replying to a troll comment, but:
> “I have data and I know SQL. What is it about your database that makes retrieving it better?”
Because my data comes from a variety of unstructured, possibly dirty sources which need cleaning and transforming before they can be made sense of.
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#19Earlier quoted context omitted.
Spark is really failing, all right. SQL lacks type safety, testability, and composability.
It’s crazy to think how old I am now. But give it 20 more years and you’ll come around.
Being old doesn’t automatically make you more right. You don’t get wisdom as a birthday gift.
Re: Ask HN: Does (or why does) anyone use MapReduce anymore?
#20The correct language for querying data is, as always, SQL. No one cares about the implementation details. “I have data and I know SQL. What is it about your database that makes retrieving it better?” Any other paradigm is going to be a niche at best, likely outright fail.
I know I'm replying to a troll comment, but: > “I have data and I know SQL. What is it about your database that makes retrieving it better?” Because my data comes from a variety of unstructured, possibly dirty sources which need cleaning and transforming before they can be made sense of.
> Because my data comes from a variety of unstructured, possibly dirty sources which need cleaning and transforming before they can be made sense of.
Seattle data guy had a great end of year top 10 memes post recently and one of them went like this
> oh cool you’ve hired a data scientist. so you have a collection of reliable and easy to query data sources, right?
> …
> you do have a collection of reliable and easy to query data sources, right?
—-
Like, most of the time in businesses… if the data can’t be queried with SQL then it’s not ready to be used by the rest of the business. Whether that’s for dashboards, monitoring, downstream analytics or reporting. Data engineers do the dirty data cleaning. Data scientists do the actual science.
That’s what I took from the parent at least.
YMMV obviously depending on your domain. ML being a good example where things like end to end speech-to-text operates on wav files directly.