Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Does (or why does) anyone use MapReduce anymore?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 70 posts

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#41

Post by nwsm »

dtoma wrote: post #21

The "streaming systems" book answers your question and more: https://www.oreilly.com/library/view/streaming-systems/97814... . It gives you a history of how batch processing started with MapReduce, and how attempts at scaling by moving towards streaming systems gave us all the subsequent frameworks (Spark, Beam, etc.). As for the framework called MapReduce, it isn't used much, but its descendant https://beam.apache.o…

This book looks interesting, should I buy it or does anyone else have newer recommendations? I have Designing Data-Intensive Applications which is a fantastic overview and still holds up well.

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#42

Post by erikerikson »

nwsm wrote: post #41
dtoma wrote: post #21

The "streaming systems" book answers your question and more: https://www.oreilly.com/library/view/streaming-systems/97814... . It gives you a history of how batch processing started with MapReduce, and how attempts at scaling by moving towards streaming systems gave us all the subsequent frameworks (Spark, Beam, etc.). As for the framework called MapReduce, it isn't used much, but its descendant https://beam.apache.o…

This book looks interesting, should I buy it or does anyone else have newer recommendations? I have Designing Data-Intensive Applications which is a fantastic overview and still holds up well.

That was one of "the" books in the space prior to DDIA. In my opinion Akidao mixes the logic for processing events with the stream infrastructure implementation because he was writing from the context of his particular use cases. The time that I spoke with him it seemed that his influence had driven to the design of Google's systems and GCP such that they didn't properly prioritize ordering/linearity/consistency requirements. At this point my copy is of historic interest to me.

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#43

Post by ramses0 »

monero-xmr wrote: post #17
esafak wrote: post #16

Earlier quoted context omitted.

Spark is really failing, all right. SQL lacks type safety, testability, and composability.

It’s crazy to think how old I am now. But give it 20 more years and you’ll come around.

Agree/disagree. I wish (and maybe this is "the answer") that you could take a basic SQL query and "invert it" into composable components.

A common thing I ended up doing for some "small data" hack projects was extremely liberal usage of SQLite: SELECT ... UNION ( SELECT ... ... GROUP BY ... ( UNION ... etc ) ) ... absolutely terrible SQL, but it got the job done to return the 100 or so records I was interested in.

It'd be great if I could write me some SQL then pop it out into: fn_group001, fn_join(g1, g2, cols(c1, c2)), ...etc...

...and then have composable sub-components of what the janky SQL-COBOL syntax supports, but in a group().chain().join(...) style.

I think I keep running across DataLog as something that's recommended, and of course ProLog has some similarities.

Nothing has been compelling enough to warrant jumping off of SQL, but I really do agree with the grandparent comment: SQL (aka: COBOL) is pretty clunky and non-composable in a way that complicates what you'd think would be straightforward for interactive, non-programming usage.

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#44

Post by revscat »

monero-xmr wrote: post #17
esafak wrote: post #16

Earlier quoted context omitted.

Spark is really failing, all right. SQL lacks type safety, testability, and composability.

It’s crazy to think how old I am now. But give it 20 more years and you’ll come around.

20 years ago SQL lacked type safety, testability, and composability. Today the same is true. I doubt it will be different 2 decades from now.

SQL is powerful. It is also very old and has very large warts.

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#45

Post by api »

revscat wrote: post #44
monero-xmr wrote: post #17

Earlier quoted context omitted.

It’s crazy to think how old I am now. But give it 20 more years and you’ll come around.

20 years ago SQL lacked type safety, testability, and composability. Today the same is true. I doubt it will be different 2 decades from now. SQL is powerful. It is also very old and has very large warts.

I think one of the biggest missed opportunities in language design is the integration of powerful relational database and query models directly into a modern language. Not as a bolt-on or an ORM but as a first class part of the language in the same way as maps and arrays. Make a language that deals with data relationally and where relational queries are a core part of the language and relational query execution is baked into its runtime.

If persistence hooks were also baked in then you'd have something a little bit like stored procedures in databases but far more powerful and with a modern syntax. Couple this with a distributed database layer supporting either eventual consistency built on CRDTs or synchronization via raft/paxos and you'd have an amazing application platform.

It's always seemed dumb to me that data, which is in the very center of everything we do, feels like a bolted-on second class citizen from the perspective of pretty much all programming languages and runtime environments. "Oh, you want to work with your data? Well we didn't think about that..." Accessing the data requires weird incantations and hacks that feel like you're entering a 1970s time warp into a PDP-11 mainframe.

Instead the language and runtime environment should be built around the data. Put the data in the center like Copernicus did with the sun.

Why has nobody done this? Has anyone even tried?

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#46

Post by monero-xmr »

revscat wrote: post #44
monero-xmr wrote: post #17

Earlier quoted context omitted.

It’s crazy to think how old I am now. But give it 20 more years and you’ll come around.

20 years ago SQL lacked type safety, testability, and composability. Today the same is true. I doubt it will be different 2 decades from now. SQL is powerful. It is also very old and has very large warts.

All data query languages eventually reduce themselves into SQL, or something equivalent to it.

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#47

Post by refulgentis »

H8crilA wrote: post #13
tjhunter wrote: post #8

(2nd user & developer of spark here). It depends on what you ask. MapReduce the framework is proprietary to Google, and some pipelines are still running inside google. MapReduce as a concept is very much in use. Hadoop was inspired by MapReduce. Spark was originally built around the primitives of MapReduce, and you see still see that in the description of its operations (exchange, collect). However, spark and all the…

There really was always only Map and Shuffle (Reduce is just Shuffle+Map; also another name for Shuffle is GroupByKey). And you see those primitives under the hood of most parallel systems.

Shuffle is interesting, I gotta read up on that. Maybe I've been hearing reduce for too long and have too much of a built-in visual sense of it but...shuffle does not seem like the right name at all, then I picture randomizing some set N, where the input and output counts are the same.

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#48

Post by revscat »

monero-xmr wrote: post #46
revscat wrote: post #44

Earlier quoted context omitted.

20 years ago SQL lacked type safety, testability, and composability. Today the same is true. I doubt it will be different 2 decades from now. SQL is powerful. It is also very old and has very large warts.

All data query languages eventually reduce themselves into SQL, or something equivalent to it.

Ok? Even if that were true — and I’m not entirely sure how you would even prove that — SQL still lacks type safety, testability, and composability.

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#50

Post by dehrmann »

codr7 wrote: post #33

It definitely played its role in high lighting what most functionish coders already knew: that map/filter/reduce is an awesome model for data processing pipelines.

Huge caveat to this: systems like Hadoop accomplish the parallelization by adding keys, and those FP constructs don't broadly have this. It's better to think of it as parallel divide-and-conquer.
Post reply on HN