Spark is a cancer. Sooner or later, 99.9% of the people using Spark will wake up to the fact that "hey, I got 1TB of RAM， why do I need this?" Spark and PySpark are just PITA to the max.
What do you recommend for distributed data processing?
PySpark Style Guide
11–20 of 28 posts
Re: PySpark Style Guide
#12Spark is a cancer. Sooner or later, 99.9% of the people using Spark will wake up to the fact that "hey, I got 1TB of RAM， why do I need this?" Spark and PySpark are just PITA to the max.
If there was something better than Spark for distributed processing, we would be using it. The rest of your comment is a straw man argument, assuming everybody uses it for datasets fitting in memory of a single node.
Re: PySpark Style Guide
#13Spark is a cancer. Sooner or later, 99.9% of the people using Spark will wake up to the fact that "hey, I got 1TB of RAM， why do I need this?" Spark and PySpark are just PITA to the max.
People that take this attitude against distributed processing usually have never had to process an amount of data bigger than would fit on one machine or always have the budget to fit everything on one very expensive large machine. It's lack of experience masquerading as cleverness. The only ones that have a right to make this arguments are the people that spread all their processing out as streams on one machine or using distributed streams but even that has serious limitation.
Re: PySpark Style Guide
#14Why not just use Scala and frameless? No style guide needed :)
Re: PySpark Style Guide
#15Spark is a cancer. Sooner or later, 99.9% of the people using Spark will wake up to the fact that "hey, I got 1TB of RAM， why do I need this?" Spark and PySpark are just PITA to the max.
What do you recommend for distributed data processing?
Dask is a great alternative for distributed computing as well: https://github.com/dask/dask
IMO, Spark is better for some tasks and Dask is better for others.
Re: PySpark Style Guide
#16Here's the Scala Spark style guide: https://github.com/MrPowers/spark-style-guide
The chispa README also provides a lot of useful info on how to properly write PySpark code: https://github.com/MrPowers/chispa
Scala is easier than Python for Spark because it allows functions with multiple argument lists and isn't whitespace sensitive. Both are great & Spark is a lot of fun.
Some specific notes:
> Doing a select at the beginning of a PySpark transform, or before returning, is considered good practice
Manual selects ensure column pruning is performed (column pruning only works for columnar file formats like Parquet). Spark does this automatically and always manually selecting may not be practical. Explicitly pruning columns is required for Pandas and Dask.
> Be careful with joins! If you perform a left join, and the right side has multiple matches for a key, that row will be duplicated as many times as there are matches
When performing joins, the first thing to think about is if a broadcast join is possible. Joins on clusters are hard. Then it's good to think about using a data stores that allows for predicate pushdown aggregations.
Re: PySpark Style Guide
#17I wonder whether I should read about best practices from Palantir.
They'll know whether you read it or not...
Re: PySpark Style Guide
#18I worked quite a lot in pandas, dplyr, data.table and pyspark for a few years. And even occasionally some scala spark and sparkR. But after getting a bit fed up with F.lit()-this, F.col()-that, and the umpteenth variation on SQL, nowadays I pretty much just stick with plain SQL. I believe I've found my Enlightenment.
I have opposite experience. After trying pyspark functional pipelines (so many handy functions) plain SQL seems so hard to read/understand. The main probem is that order of execution is not equal to order of code lines. https://i.stack.imgur.com/6YuwE.jpg another thing is that python is so cool for data processing, and when working with plain sql I feel lack of .rdd.map(my_python_processing_function)
Same for me. Python and Scala let users break up the logic into DataFrame transformations that can be unit tested, packaged into Wheel / JAR files, and easily reused in multiple contexts. Maintaining big, complex SQL codebases isn't easy.
Re: PySpark Style Guide
#19I worked quite a lot in pandas, dplyr, data.table and pyspark for a few years. And even occasionally some scala spark and sparkR. But after getting a bit fed up with F.lit()-this, F.col()-that, and the umpteenth variation on SQL, nowadays I pretty much just stick with plain SQL. I believe I've found my Enlightenment.
Could not agree more. Similar to the The Principle of Least Privilege [1], I prefer to use SQL over pyspark if possible.
[1] https://us-cert.cisa.gov/bsi/articles/knowledge/principles/l...
Re: PySpark Style Guide
#20Spark is a cancer. Sooner or later, 99.9% of the people using Spark will wake up to the fact that "hey, I got 1TB of RAM， why do I need this?" Spark and PySpark are just PITA to the max.
What do you recommend for distributed data processing?
First step is decide if you really need distributed data processing. I think this is the point author is making. I've seen GB sized data considered "BIG DATA" and its unbelievable the architectural patterns used to support this "BIG DATA".