Apache Spark Scale: A 60 TB+ production use case
code.facebook.com
Apache Spark Scale: A 60 TB+ production use case
1–10 of 42 posts
Re: Apache Spark Scale: A 60 TB+ production use case
#2This is very intriguing, I cannot help but wonder how this operation would have performed on something like Google Big Query. I know that it is highly unlikely that Facebook would ever load their data to the Google Cloud Platform, but it would be an interesting comparison.
Re: Apache Spark Scale: A 60 TB+ production use case
#3Looks awesome, the work on performance especially is much appreciated :D I would love to be able to get flame graphs on my spark cluster per node too - any ideas on how exactly FB does this?
Re: Apache Spark Scale: A 60 TB+ production use case
#4Looks awesome, the work on performance especially is much appreciated :D I would love to be able to get flame graphs on my spark cluster per node too - any ideas on how exactly FB does this?
Looks they are using the same library from Presto. You would just add the Java agent line to your Spark config i.e. spark.executor.extraJavaOptions.
https://github.com/prestodb/presto/issues/4004
Although you have to distribute the library across to all of the data nodes (if using YARN) or Spark nodes (if in standalone).
Re: Apache Spark Scale: A 60 TB+ production use case
#5Amazing work and nice article! Congrats to the Facebook and Databrick teams!
Re: Apache Spark Scale: A 60 TB+ production use case
#6they never mention hadoop and yahoo but mention they built hive a dozen times in one paragraph. is this how PR people think open sourced tech should be mentioned/credited? so lame. they can't think beyond their belly buttons.
Re: Apache Spark Scale: A 60 TB+ production use case
#7they never mention hadoop and yahoo but mention they built hive a dozen times in one paragraph. is this how PR people think open sourced tech should be mentioned/credited? so lame. they can't think beyond their belly buttons.
Not sure what you mean. They only mentioned building Hive once.
And why would they mention Yahoo ? They stopped being particularly relevant in the Hadoop space quite a few years ago.
Re: Apache Spark Scale: A 60 TB+ production use case
#8This is very intriguing, I cannot help but wonder how this operation would have performed on something like Google Big Query. I know that it is highly unlikely that Facebook would ever load their data to the Google Cloud Platform, but it would be an interesting comparison.
My guess is that it wouldn't be too bad, but that at the volumes Facebook are doing, I suspect a lot of their speed is coming from custom optimisations they can make for their use-cases, which Big Query wouldn't have.
Re: Apache Spark Scale: A 60 TB+ production use case
#9[deleted]
Re: Apache Spark Scale: A 60 TB+ production use case
#10What I find most exciting here is that they were able to improve performance by combining the three previous steps into one, in other words asking directly for what they wanted rather than trying to trick Hive into executing it in a certain manner.
Presumably the task had been split up in the past to work around performance limitations of Hive; the fact that they got better performance in a single step shows that the tooling is improving and much less complexity is required to implement this kind of job.