Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Does (or why does) anyone use MapReduce anymore?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 70 posts

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#61

Post by esafak »

monero-xmr wrote: post #60
esafak wrote: post #56

Earlier quoted context omitted.

So use a better language, and let the compiler optimize it??

The query planner optimizes it. Why would you want a compiler to optimize SQL? The nature of your data affects how it is optimized! The declarative statement to retrieve data must be interpreted based upon the nature of that data. You can't pre-optimize without knowing something about your data, in which case, you are basically storing some of the information outside of the database.

I mean don't use SQL at all. Use a real programming language like Scala, and let Spark (or Flink etc.) do the translation and optimization: https://www.databricks.com/glossary/catalyst-optimizer

I don't understand why anyone would prefer SQL to that for anything beyond a simple SQL query. And it's not just my opinion: industry at large uses Spark for production with complex queries. SQL is for analysts.

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#62

Post by nathants »

the idea of map reduce remains a good one.

there are a number of interesting innovations in streaming systems that followed, mostly around reducing latency, reducing batch size, and failure strategies.

even hadoop could be hard to debug when hitting a performance ceiling for challenging workloads. the streaming systems took this even further, spark being notorious for fiddle with knobs and pray the next job doesn’t fail after a few hours, again.

i played around with the thinnest possible distributed data stack a while back[1][2]. i wanted to understand the performance ceiling for different workloads without all the impenetrable layers of software bureaucracy. turns out modern network and cpu are really fast when you stop adding random layers like lasagna.

i think the future of data, for serious workloads, is gonna be bespoke. the primitives are just too good now, and the tradeoff for understandability is often worth the cost.

1. https://github.com/nathants/s4

2. https://github.com/nathants/bsv

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#63

Post by monero-xmr »

esafak wrote: post #61
monero-xmr wrote: post #60

Earlier quoted context omitted.

The query planner optimizes it. Why would you want a compiler to optimize SQL? The nature of your data affects how it is optimized! The declarative statement to retrieve data must be interpreted based upon the nature of that data. You can't pre-optimize without knowing something about your data, in which case, you are basically storing some of the information outside of the database.

I mean don't use SQL at all. Use a real programming language like Scala, and let Spark (or Flink etc.) do the translation and optimization: https://www.databricks.com/glossary/catalyst-optimizer I don't understand why anyone would prefer SQL to that for anything beyond a simple SQL query. And it's not just my opinion: industry at large uses Spark for production with complex queries. SQL is for analysts.

Now I’m totally confused. SQL is a syntax for querying data. Spark SQL is SQL. You are talking about a different implementation of a server.

SQL is for analysts? Everyone uses SQL.

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#64

Post by esafak »

monero-xmr wrote: post #63
esafak wrote: post #61

Earlier quoted context omitted.

I mean don't use SQL at all. Use a real programming language like Scala, and let Spark (or Flink etc.) do the translation and optimization: https://www.databricks.com/glossary/catalyst-optimizer I don't understand why anyone would prefer SQL to that for anything beyond a simple SQL query. And it's not just my opinion: industry at large uses Spark for production with complex queries. SQL is for analysts.

Now I’m totally confused. SQL is a syntax for querying data. Spark SQL is SQL. You are talking about a different implementation of a server. SQL is for analysts? Everyone uses SQL.

How familiar are you with Spark and the like? This is what it looks like:

https://spark.apache.org/examples.html

SQL is just a DSL; it is not the only or primary API for Spark, and there's nothing magical about it. If you ditch it you can get your type safety, composability, and testability back, like so:

https://medium.com/@sergey.kotlov/unit-testing-of-spark-appl...

See those case classes that neatly encapsulate business objects? Add to that functional transforms that concisely express typical operations like filtering, mapping, and so on, you get something that is simply superior to SQL.

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#65

Post by monero-xmr »

esafak wrote: post #64
monero-xmr wrote: post #63

Earlier quoted context omitted.

Now I’m totally confused. SQL is a syntax for querying data. Spark SQL is SQL. You are talking about a different implementation of a server. SQL is for analysts? Everyone uses SQL.

How familiar are you with Spark and the like? This is what it looks like: https://spark.apache.org/examples.html SQL is just a DSL; it is not the only or primary API for Spark, and there's nothing magical about it. If you ditch it you can get your type safety, composability, and testability back, like so: https://medium.com/@sergey.kotlov/unit-testing-of-spark-appl... See those case classes that neatly encapsulate bu…

Any ORM provides the same features for any SQL database. There is nothing special going on here. If perhaps the database autogenerated a bunch of classes, maybe that’s interesting? I think some projects have introspected a database and created all the boilerplate language classes before.

There is nothing magical about wrapping database objects in language classes. This has been happening forever.

https://docs.sqlalchemy.org/en/20/orm/quickstart.html#select...

Nothing magical about using a function call rather than raw SQL.

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#66

Post by esafak »

monero-xmr wrote: post #65
esafak wrote: post #64

Earlier quoted context omitted.

How familiar are you with Spark and the like? This is what it looks like: https://spark.apache.org/examples.html SQL is just a DSL; it is not the only or primary API for Spark, and there's nothing magical about it. If you ditch it you can get your type safety, composability, and testability back, like so: https://medium.com/@sergey.kotlov/unit-testing-of-spark-appl... See those case classes that neatly encapsulate bu…

Any ORM provides the same features for any SQL database. There is nothing special going on here. If perhaps the database autogenerated a bunch of classes, maybe that’s interesting? I think some projects have introspected a database and created all the boilerplate language classes before. There is nothing magical about wrapping database objects in language classes. This has been happening forever. https://docs.sqlalch…

No, it does not. Furthermore, if you're using an ORM you're not programming in SQL any more. You're using a poor man's Spark. Spark lets you drop down to SQL in all its APIs too.

I don't understand your argument if you're comfortable with ORMs.

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#67

Post by monero-xmr »

esafak wrote: post #66
monero-xmr wrote: post #65

Earlier quoted context omitted.

Any ORM provides the same features for any SQL database. There is nothing special going on here. If perhaps the database autogenerated a bunch of classes, maybe that’s interesting? I think some projects have introspected a database and created all the boilerplate language classes before. There is nothing magical about wrapping database objects in language classes. This has been happening forever. https://docs.sqlalch…

No, it does not. Furthermore, if you're using an ORM you're not programming in SQL any more. You're using a poor man's Spark. Spark lets you drop down to SQL in all its APIs too. I don't understand your argument if you're comfortable with ORMs.

I’m not even sure what we are debating. Spark is for big data work, it isn’t something you would typically use as a general purpose database backing your application. My original point way back is that expecting customers to learn an entirely new paradigm for storing and querying data is a poor decision, and limits your work to niche use cases. Spark has a large niche but it isn’t comparable to MySQL / Postgres or other newish data stores like Dynamo.

Furthermore any competent engineer knows SQL because ORMs are cumbersome and annoying for anything except basic use.

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#68

Post by H8crilA »

lupire wrote: post #38
H8crilA wrote: post #13

Earlier quoted context omitted.

There really was always only Map and Shuffle (Reduce is just Shuffle+Map; also another name for Shuffle is GroupByKey). And you see those primitives under the hood of most parallel systems.

Reduce is useful for aggregate metrics.

My point is that Reduce is Shuffle+Map, without materializing the intermediate result (the result after Shuffle).

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#69

Post by H8crilA »

refulgentis wrote: post #47
H8crilA wrote: post #13

Earlier quoted context omitted.

There really was always only Map and Shuffle (Reduce is just Shuffle+Map; also another name for Shuffle is GroupByKey). And you see those primitives under the hood of most parallel systems.

Shuffle is interesting, I gotta read up on that. Maybe I've been hearing reduce for too long and have too much of a built-in visual sense of it but...shuffle does not seem like the right name at all, then I picture randomizing some set N, where the input and output counts are the same.

Shuffle is an operation that converts "{k1, v1}, {k1, v2}, {k2, v3}" into "{k1, [v1, v2]}, {k2, [v3]}".

Re: Ask HN: Does (or why does) anyone use MapReduce anymore?

#70

Post by mlhpdx »

api wrote: post #45
revscat wrote: post #44

Earlier quoted context omitted.

20 years ago SQL lacked type safety, testability, and composability. Today the same is true. I doubt it will be different 2 decades from now. SQL is powerful. It is also very old and has very large warts.

I think one of the biggest missed opportunities in language design is the integration of powerful relational database and query models directly into a modern language. Not as a bolt-on or an ORM but as a first class part of the language in the same way as maps and arrays. Make a language that deals with data relationally and where relational queries are a core part of the language and relational query execution is ba…

Like language integrated queries in c#?

https://learn.microsoft.com/en-us/dotnet/csharp/linq/

Post reply on HN