Ask HN: Learning PySpark and Related Tools

Hey HN,

I have been working in the data-science and machine-learning domain for the past 8 years or so. I have not been exposed to tools such as PySpark etc. which are being asked frequently in job descriptions. What resource or certification can I use to get upto par on PySpark?

Thanks!

I thought the _Learning Spark_ book was a pretty good introduction. Databricks offers it for free here [0]

[0] https://pages.databricks.com/rs/094-YMS-629/images/LearningS...

I'm a Data Engineer which uses Spark daily. I guess the only important cert would come from Databricks, but I think it will be more worth your while to read the book mentioned here and try to do a little project ingesting/transforming data

Having used spark for the past 8 years or so, it's definitely a solid basic for data engineering. I use it for generating reports the most, but sometimes we have large projects to get data into different staging databases. I use it a lot with ElasticSearch or a parquet. Basically it helps you write large joins and flatten the result to a database that can more quickly perform aggregations on that flattened result (like Elasticsearch) or a columnar database.