Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Spark Scale: A 60 TB+ production use case

code.facebook.com

Post reply on HN
Print view View original

11–20 of 42 posts

Re: Apache Spark Scale: A 60 TB+ production use case

#11

Post by ap22213 »

"While running on 20 TB of input, we discovered that we were generating too many output files (each sized around 100 MB) due to the large number of tasks."

I could be completely missing something here, but to decrease the number of output files you can coalesce() the RDD before you write. For example, let's say you have a 20 node cluster, each with 10 executors, and the RDD action is split into 20000 tasks. You may end up with 20000 partitions (or more). However, you can coalesce, and reduce the number down to 200 partitions. Or, if necessary, you could even shuffle (across VMs but within the node) down to 20 partitions, if you're really motivated.

What am I missing?

Re: Apache Spark Scale: A 60 TB+ production use case

#12

Post by ap22213 »

ap22213 wrote: post #11

"While running on 20 TB of input, we discovered that we were generating too many output files (each sized around 100 MB) due to the large number of tasks." I could be completely missing something here, but to decrease the number of output files you can coalesce() the RDD before you write. For example, let's say you have a 20 node cluster, each with 10 executors, and the RDD action is split into 20000 tasks. You may e…

"Remove the two temporary tables and combine all three Hive stages into a single Spark job that reads 60 TB of compressed data and performs a 90 TB shuffle and sort."

"As far as we know, this is the largest real-world Spark job attempted in terms of shuffle data size"

I'm far, far from a world class engineer, but I regularly do 90 TiB shuffle sorts. I must seriously be missing something, here.

Re: Apache Spark Scale: A 60 TB+ production use case

#14

Post by mark_l_watson »

Great write up. I used Hadoop a lot for two consulting customers, and Google's map reduce as a contractor. I found map reduce to be a natural way to process data. That said, Spark is a huge leap forward. I like both the developer workflow using a repl and the machine learning library mllib is excellent.

Re: Apache Spark Scale: A 60 TB+ production use case

#15

Post by vgt »

Swinx43 wrote: post #2

This is very intriguing, I cannot help but wonder how this operation would have performed on something like Google Big Query. I know that it is highly unlikely that Facebook would ever load their data to the Google Cloud Platform, but it would be an interesting comparison.

While both Spark and BigQuery do the shuffle step in-memory, there are some differences[1]:

- BigQuery's execution is pipelined (don't wait for step 1 to finish to start step 2)

- BigQuery's in-memory shuffler is not co-tenant to compute

And, of course, it's one thing to have software and hardware. BigQuery provides a fully-managed, fully-encrypted, HA, redundant, and constantly seamlessly maintaned and upgraded service [2].

[1]https://cloud.google.com/blog/big-data/2016/08/in-memory-que...

[2]https://cloud.google.com/blog/big-data/2016/08/google-bigque...

(disc: work on BigQuery)

Re: Apache Spark Scale: A 60 TB+ production use case

#16

Post by sigzero »

ap22213 wrote: post #12
ap22213 wrote: post #11

"While running on 20 TB of input, we discovered that we were generating too many output files (each sized around 100 MB) due to the large number of tasks." I could be completely missing something here, but to decrease the number of output files you can coalesce() the RDD before you write. For example, let's say you have a 20 node cluster, each with 10 executors, and the RDD action is split into 20000 tasks. You may e…

"Remove the two temporary tables and combine all three Hive stages into a single Spark job that reads 60 TB of compressed data and performs a 90 TB shuffle and sort." "As far as we know, this is the largest real-world Spark job attempted in terms of shuffle data size" I'm far, far from a world class engineer, but I regularly do 90 TiB shuffle sorts. I must seriously be missing something, here.

Are you using Spark? That's the context.

Re: Apache Spark Scale: A 60 TB+ production use case

#17

Post by openasocket »

mozumder wrote: post #13

Couldn't they have just used Postgres on one 4/8-socket server with RAID? A 60TB dataset can fit in one server. Isn't Spark intended for massive clusters, like dozens or hundreds of servers, over petabytes of data?

Note that it's 60TB compressed

Mostly it's about speed. With multiple machines you can bring more cores to bear for the processing, and have more RAM to cache partial results. Postgres could certainly do the job, but I'd be surprised if it would run within an order of magnitude of these results.

Re: Apache Spark Scale: A 60 TB+ production use case

#18

Post by paulasmuth »

Swinx43 wrote: post #2

This is very intriguing, I cannot help but wonder how this operation would have performed on something like Google Big Query. I know that it is highly unlikely that Facebook would ever load their data to the Google Cloud Platform, but it would be an interesting comparison.

I can't speak for big query, but for our open-source BigQuery alternative EventQL [0].

On the upside, massively parallel SQL databases allow you to express what probably took facebook a lot of code in a single SQL query.

On the downside, with any database that supports streaming inserts and dynamic resharding, you will have to eventually write each entry more than once. The more manual approach with spark prevents this.

[0] https://eventql.io

Re: Apache Spark Scale: A 60 TB+ production use case

#19

Post by ap22213 »

sigzero wrote: post #16
ap22213 wrote: post #12

Earlier quoted context omitted.

"Remove the two temporary tables and combine all three Hive stages into a single Spark job that reads 60 TB of compressed data and performs a 90 TB shuffle and sort." "As far as we know, this is the largest real-world Spark job attempted in terms of shuffle data size" I'm far, far from a world class engineer, but I regularly do 90 TiB shuffle sorts. I must seriously be missing something, here.

Are you using Spark? That's the context.

Yes

Re: Apache Spark Scale: A 60 TB+ production use case

#20

Post by coredog64 »

threeseed wrote: post #4
zaptheimpaler wrote: post #3

Looks awesome, the work on performance especially is much appreciated :D I would love to be able to get flame graphs on my spark cluster per node too - any ideas on how exactly FB does this?

Looks they are using the same library from Presto. You would just add the Java agent line to your Spark config i.e. spark.executor.extraJavaOptions. https://github.com/prestodb/presto/issues/4004 Although you have to distribute the library across to all of the data nodes (if using YARN) or Spark nodes (if in standalone).

Thanks! I was going to ask where I might find it since Google results were a little thin on the ground.
Post reply on HN