21–30 of 38 posts
There are two lessons you could learn from this episode: 1. Use shallow trees and the clever workaround presented in the article. 2. Don't use Spark for tasks that require complex logic. People should trace out the line of reasoning that leads them to use tools like Spark. It is convoluted and contingent - it goes back to work done at Google in the early 2000s, when the key to getting good price / performance was usi…
I've hit almost the exact same issue with Hive, with a somewhat temporary workaround (like this post) to build a balanced tree out of this by reading it into a list [1] and rebuilding a binary balanced tree out of it. But we ended up implementing a single level Multi-AND [2] so that this no longer a tree for just AND expressions & can be vectorized neater than the nested structure with a function call for each (this…
p1 AND (p2 AND (p3 AND notp4)) -> (p1 AND p2) AND (p3 AND notp4)
ALL (p1 p2 p3 NOT(p4))
There are two lessons you could learn from this episode: 1. Use shallow trees and the clever workaround presented in the article. 2. Don't use Spark for tasks that require complex logic. People should trace out the line of reasoning that leads them to use tools like Spark. It is convoluted and contingent - it goes back to work done at Google in the early 2000s, when the key to getting good price / performance was usi…
Edit: I don’t mean to suggest that there is no reason to use Spark, but ~95% of the usage in industry is unnecessary now and should be avoided.
There are two lessons you could learn from this episode: 1. Use shallow trees and the clever workaround presented in the article. 2. Don't use Spark for tasks that require complex logic. People should trace out the line of reasoning that leads them to use tools like Spark. It is convoluted and contingent - it goes back to work done at Google in the early 2000s, when the key to getting good price / performance was usi…
Is there an alternative you’d recommend?
1. https://www.usenix.org/system/files/conference/hotos15/hotos...
Post author here. Let me know if you have any questions!
There are two lessons you could learn from this episode: 1. Use shallow trees and the clever workaround presented in the article. 2. Don't use Spark for tasks that require complex logic. People should trace out the line of reasoning that leads them to use tools like Spark. It is convoluted and contingent - it goes back to work done at Google in the early 2000s, when the key to getting good price / performance was usi…
Exactly correct. I’ve got a post in the works called “Elegy for Hadoop” that traces the history back to the early 2000s and arrives at the present day where you can easily get on-demand instances with 500Gb of RAM and use it for only your application’s lifetime. If you want 1000Gb instead of 500gb it does not cost 5x it costs 2x, significantly invalidating the “need to use excess commodity hardware” premise of the di…
I've hit almost the exact same issue with Hive, with a somewhat temporary workaround (like this post) to build a balanced tree out of this by reading it into a list [1] and rebuilding a binary balanced tree out of it. But we ended up implementing a single level Multi-AND [2] so that this no longer a tree for just AND expressions & can be vectorized neater than the nested structure with a function call for each (this…
It seems to me that the basic problem is that binary trees are the wrong data type. For instance, you can transform the tree to balance it: p1 AND (p2 AND (p3 AND notp4)) -> (p1 AND p2) AND (p3 AND notp4) But the abstraction is specifying the order of operations unnecessarily. Using general trees, I think you avoid the need to transform the order of operations and "NOT" doesn't have to be a special case. ALL (p1 p2 p…
That needs an "in SQL", the standard imperative language operator ordering has short-cut operations in there (a is null or a.value == true) etc.
In the code I work with, this actually sorts the conditions based on estimated selectivity[1] and type (long compares to constant are cheaper on a columnar data-set due to the SIMD, but string isn't etc).
> Is there any reason to choose binary trees for this?
The parse-tree does come off as binary because inserting logical parentheses makes it easier to tackle, because there are association rules which neatly go into a BinaryOp structure when dealing with operator precedence in parsing.
So it is easier to handle the parsing when you treat (a + (b + c) ) and (a / (b / c)) in similar fashion.
I won't make the same mistake again if I have to build a SQL engine, but this actually made the logical expression match the parse tree very closely and was a good enough generalization until the traversal time bugs[2] started to pop up.
[1] - https://github.com/apache/hive/blob/master/common/src/java/o... [2] - https://issues.apache.org/jira/browse/HIVE-9166
Earlier quoted context omitted.
It boggles my mind that the author wrote an entire long article based on this. The rhetorical question saying that surely that weird refactor of two different functions into one, followed by calling that new, non-trivial function twice for no reason surely shouldn't affect performance.. He already lost me during the premise of the article.
What is so hard to understand here? There is some library code you can't immediately change because it belongs to upstream Spark. To illustrate the problem, ne simplifies the code to represent what the problem is. Then, ne writes some code that works around the library bug by modifying the input losslessly into something that's more easily processed by the library. Finally, ne patches the library bug and shares the p…
Spark is this weird ecosystem of people who take absolutely trivial concepts in SQL, bury their heads in the sand and ignore the past 50 years of RDBMS evolution, and then write extremely complicated (or broken) and expensive to run code. But whatever it takes to get Databricks to IPO! Afterwards the hype will die down and everyone will collectively abandon it just like MongoDB except for the unfortunate companies wi…
spark is far more testable and composable than sql! and you even get static typing checking. plus i can read data from anywhere - local fs, s3, rdbms, json, parquet, csv... rdbms could not compete
There are two lessons you could learn from this episode: 1. Use shallow trees and the clever workaround presented in the article. 2. Don't use Spark for tasks that require complex logic. People should trace out the line of reasoning that leads them to use tools like Spark. It is convoluted and contingent - it goes back to work done at Google in the early 2000s, when the key to getting good price / performance was usi…