Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PySpark Style Guide

github.com

Post reply on HN
Print view View original

21–28 of 28 posts

Re: PySpark Style Guide

#21

Post by choppaface »

MrPowers wrote: post #16

Here's the Scala Spark style guide: https://github.com/MrPowers/spark-style-guide The chispa README also provides a lot of useful info on how to properly write PySpark code: https://github.com/MrPowers/chispa Scala is easier than Python for Spark because it allows functions with multiple argument lists and isn't whitespace sensitive. Both are great & Spark is a lot of fun. Some specific notes: > Doing a select at the…

Agree that they missed broadcast joins, which can greatly impact how you’d go about a query versus plain SQL for big data. One of the best parts about Spark is how it supports rapid iteration—- you can use it to discover what joins are computationally infeasible.

It’s notable that in Spark 3.x, Koalas is standard, which adopts the Pandas API. Yet this style guide uses the Spark DataFrame API. So the guide might be a little stale anyways.

In my experience, it’s helpful to write queries in plain portable (or mostly portable) SQL, because once a Spark job becomes useful it often gets translated or refactored into something else. Definitely depends on the team / context, but plain SQL is often more widely accessible. For fast-moving data science stuff, it’s important to think about extensibility in terms of not just code (style & syntax) but people (who is going to remix this?).

Re: PySpark Style Guide

#22

Post by slotrans »

I'm so confused.

These examples are all using the SQL-like features of Spark. Not a map() or flatMap() in sight.

So... why not just write SQL?

  df.registerTempTable('some_name')
  new_df = spark.sql("""select ... from some_name ...""")
All of this F.col(...) and .alias(...) and .withColumn(...) nonsense is a million times harder to read than proper SQL. I just don't understand what any of this is intended to accomplish.

Re: PySpark Style Guide

#23

Post by legerdemain »

I think this guide mostly dates from 2017, when Palantir was rolling out Spark and Spark SQL code authoring in their Foundry data platform. It mostly targets their untrained "delta" and "echo" employees, most of whose jobs rotated around writing ETL code for customers. I have no idea why this glorified Quip doc was open-sourced.

Looking at the list of contributors on Github, I think I remember that the main author was actually James Thompson (UK), and not anyone on the contributor list. JTUK was called that because the company had another employee, based in the US, who had the same name. James Thompson (US) is now at Facebook and is a pretty cool designer. His astronomer.io media project from 2011 comes up on HN periodically.

Of the people listed on Github, Andrew Ash (now at Google) is the original evangelist for Spark on Palantir Foundry, and Francisco is the PM for SkyWise, Palantir's lucrative, but ill-fated effort to save the Airbus A380.

Re: PySpark Style Guide

#24

Post by legerdemain »

slotrans wrote: post #22

I'm so confused. These examples are all using the SQL-like features of Spark. Not a map() or flatMap() in sight. So... why not just write SQL? df.registerTempTable('some_name') new_df = spark.sql("""select ... from some_name ...""") All of this F.col(...) and .alias(...) and .withColumn(...) nonsense is a million times harder to read than proper SQL. I just don't understand what any of this is intended to accomplish.

Because the examples use the Dataframe API and not the RDD API? At least as of Spark 2.4.7, both `Dataset.map()` and `Dataset.flatMap()` are still marked experimental.

Re: PySpark Style Guide

#25

Post by legerdemain »

speedgoose wrote: post #7

I wonder whether I should read about best practices from Palantir.

Fun fact: Palantir developers wrote what was, for a time, the most starred TypeScript linter (until it was decided to unify it with ESLint). They were working on what was supposed to be the world's largest TS monorepo (at least according to the MS rep).

Re: PySpark Style Guide

#26

Post by MrPowers »

choppaface wrote: post #21
MrPowers wrote: post #16

Here's the Scala Spark style guide: https://github.com/MrPowers/spark-style-guide The chispa README also provides a lot of useful info on how to properly write PySpark code: https://github.com/MrPowers/chispa Scala is easier than Python for Spark because it allows functions with multiple argument lists and isn't whitespace sensitive. Both are great & Spark is a lot of fun. Some specific notes: > Doing a select at the…

Agree that they missed broadcast joins, which can greatly impact how you’d go about a query versus plain SQL for big data. One of the best parts about Spark is how it supports rapid iteration—- you can use it to discover what joins are computationally infeasible. It’s notable that in Spark 3.x, Koalas is standard, which adopts the Pandas API. Yet this style guide uses the Spark DataFrame API. So the guide might be a…

I'd argue Koalas is an anti-pattern but will have to justify that in a blog post ;)

I've written some popular Scala Spark (https://github.com/MrPowers/spark-daria) and PySpark (https://github.com/MrPowers/quinn) libraries that have been adopted by a variety of teams. Not sure how to make a reusable with pure SQL, but sounds like it's possible. Send me a code snippet or link if you have anything I can take a look at to learn more about your approach.

Re: PySpark Style Guide

#27

Post by MrPowers »

slotrans wrote: post #22

I'm so confused. These examples are all using the SQL-like features of Spark. Not a map() or flatMap() in sight. So... why not just write SQL? df.registerTempTable('some_name') new_df = spark.sql("""select ... from some_name ...""") All of this F.col(...) and .alias(...) and .withColumn(...) nonsense is a million times harder to read than proper SQL. I just don't understand what any of this is intended to accomplish.

The spark.sql("""select ... from some_name ...""") bit is how to write pure SQL from the Scala or Python execution context. If you're in the SQL execution context, this syntax isn't required. I never write Spark code like this.

At first glance, the F.col(), .withColumn() syntax isn't as intuitive as pure SQL, but it has a lot of advantages when you get used to it. You can make abstractions, use programming language features like loops, and use IDEs.

I find the PySpark syntax to be uglier than Scala. Lots of teams are terrified to use Scala and that's the reason PySpark is so popular.

Re: PySpark Style Guide

#28

Post by choppaface »

MrPowers wrote: post #26
choppaface wrote: post #21

Earlier quoted context omitted.

Agree that they missed broadcast joins, which can greatly impact how you’d go about a query versus plain SQL for big data. One of the best parts about Spark is how it supports rapid iteration—- you can use it to discover what joins are computationally infeasible. It’s notable that in Spark 3.x, Koalas is standard, which adopts the Pandas API. Yet this style guide uses the Spark DataFrame API. So the guide might be a…

I'd argue Koalas is an anti-pattern but will have to justify that in a blog post ;) I've written some popular Scala Spark ( https://github.com/MrPowers/spark-daria ) and PySpark ( https://github.com/MrPowers/quinn ) libraries that have been adopted by a variety of teams. Not sure how to make a reusable with pure SQL, but sounds like it's possible. Send me a code snippet or link if you have anything I can take a look…

I’m also not happy with Koalas but at least it’s a step towards API unification.

Pure SQL vs DataFrame— just write any typical join, groupby & count OLAP query as SQL and again using the DataFrame API. I’m saying the SQL query is more accessible to non-Spark users (e.g. a DBA who might need to approve your code) and as-is can be thrown into Hive/Presto or any RDBMS pretty easily. The DataFrame version is definitely more extensible, but in my experience Spark is more often used to inform the design of a larger data pipeline versus serve as the pipeline year after year. Appreciate there are places where the opposite is true.

Post reply on HN