Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accidentally exponential behavior in Spark

heap.io

Post reply on HN
Print view View original

1–10 of 38 posts

Re: Accidentally exponential behavior in Spark

#7

Post by dreyfan »

Spark is this weird ecosystem of people who take absolutely trivial concepts in SQL, bury their heads in the sand and ignore the past 50 years of RDBMS evolution, and then write extremely complicated (or broken) and expensive to run code. But whatever it takes to get Databricks to IPO! Afterwards the hype will die down and everyone will collectively abandon it just like MongoDB except for the unfortunate companies with so much technical debt they can't extricate themselves from it.

Re: Accidentally exponential behavior in Spark

#8

Post by eska »

mlyle wrote: post #5

Why not just... val transformedLeftTemp = transform(tree.left) val transformedLeft = if (transformedLeftTemp.isDefined) { transformedLeftTemp } else None

It boggles my mind that the author wrote an entire long article based on this.

The rhetorical question saying that surely that weird refactor of two different functions into one, followed by calling that new, non-trivial function twice for no reason surely shouldn't affect performance.. He already lost me during the premise of the article.

Re: Accidentally exponential behavior in Spark

#9

Post by scottmcdot »

dreyfan wrote: post #7

Spark is this weird ecosystem of people who take absolutely trivial concepts in SQL, bury their heads in the sand and ignore the past 50 years of RDBMS evolution, and then write extremely complicated (or broken) and expensive to run code. But whatever it takes to get Databricks to IPO! Afterwards the hype will die down and everyone will collectively abandon it just like MongoDB except for the unfortunate companies wi…

Is it best to just use spark.sql?

Re: Accidentally exponential behavior in Spark

#10

Post by renewiltord »

eska wrote: post #8
mlyle wrote: post #5

Why not just... val transformedLeftTemp = transform(tree.left) val transformedLeft = if (transformedLeftTemp.isDefined) { transformedLeftTemp } else None

It boggles my mind that the author wrote an entire long article based on this. The rhetorical question saying that surely that weird refactor of two different functions into one, followed by calling that new, non-trivial function twice for no reason surely shouldn't affect performance.. He already lost me during the premise of the article.

What is so hard to understand here? There is some library code you can't immediately change because it belongs to upstream Spark. To illustrate the problem, ne simplifies the code to represent what the problem is.

Then, ne writes some code that works around the library bug by modifying the input losslessly into something that's more easily processed by the library.

Finally, ne patches the library bug and shares the patch.

All of this is also kinda fucking obvious to not just me, but a lot of people, so I'm having a really hard time grasping if you've mixed up the illustrative simplification with the actual code, or if you think that the best engineering approach is to always patch your environment bugs instead of modifying your input, or if you just don't have a Github account or for some other reason can't read the patch.

Between that patch and https://github.com/apache/spark/pull/24910 you can see why the code is what it is.

Post reply on HN