PySpark Style Guide
github.com
PySpark Style Guide
1–10 of 28 posts
Re: PySpark Style Guide
#2There is also a blog post on this: https://medium.com/palantir/a-pyspark-style-guide-for-real-w...
Re: PySpark Style Guide
#3> The preferred option is more complicated, longer, and polluted - and correct.
This is the definition of bad design.
Re: PySpark Style Guide
#4[deleted]
Re: PySpark Style Guide
#5> The preferred option is more complicated, longer, and polluted - and correct. This is the definition of bad design.
Was also rolling my eyes at that one. Furthermore, if you are this concerned with the refactoring limitations of your IDE, then don't use Python, but the Scala or Java API instead.
Re: PySpark Style Guide
#6I worked quite a lot in pandas, dplyr, data.table and pyspark for a few years. And even occasionally some scala spark and sparkR. But after getting a bit fed up with F.lit()-this, F.col()-that, and the umpteenth variation on SQL, nowadays I pretty much just stick with plain SQL. I believe I've found my Enlightenment.
Re: PySpark Style Guide
#7I wonder whether I should read about best practices from Palantir.
Re: PySpark Style Guide
#8I worked quite a lot in pandas, dplyr, data.table and pyspark for a few years. And even occasionally some scala spark and sparkR. But after getting a bit fed up with F.lit()-this, F.col()-that, and the umpteenth variation on SQL, nowadays I pretty much just stick with plain SQL. I believe I've found my Enlightenment.
I have opposite experience. After trying pyspark functional pipelines (so many handy functions) plain SQL seems so hard to read/understand. The main probem is that order of execution is not equal to order of code lines. https://i.stack.imgur.com/6YuwE.jpg
another thing is that python is so cool for data processing, and when working with plain sql I feel lack of
.rdd.map(my_python_processing_function)Re: PySpark Style Guide
#9> The preferred option is more complicated, longer, and polluted - and correct. This is the definition of bad design.
So much eye rolling. It's a mixture of some common sense, some bad suggestions (such as this one), and a couple of mandates such as "It is highly recommended to avoid UDFs in all situations" rather than providing any real guidance.
Re: PySpark Style Guide
#10Spark is a cancer. Sooner or later, 99.9% of the people using Spark will wake up to the fact that "hey, I got 1TB of RAM， why do I need this?"
Spark and PySpark are just PITA to the max.