Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Spark Scale: A 60 TB+ production use case

code.facebook.com

Post reply on HN
Print view View original

21–30 of 42 posts

Re: Apache Spark Scale: A 60 TB+ production use case

#21

Post by vonmoltke »

mark_l_watson wrote: post #14

Great write up. I used Hadoop a lot for two consulting customers, and Google's map reduce as a contractor. I found map reduce to be a natural way to process data. That said, Spark is a huge leap forward. I like both the developer workflow using a repl and the machine learning library mllib is excellent.

> I found map reduce to be a natural way to process data.

It is a natural way to process certain types of data in certain ways. While there are lots of use cases like that, I have seen too many efforts to force things into Hadoop MapReduce/YARN or similar platforms just because it is the parallel processing hotness of the year.

Re: Apache Spark Scale: A 60 TB+ production use case

#22

Post by chime »

I've been working on Databricks for a month now and it is a very unique and satisfying learning experience. I generated a billion n-grams (1-6 word) from 300k docs and thinking in RDDs and Dataframes reminds me of how I felt when I first learned functional programming and GPU coding. RDD flatMap solves problems I didn't even know I had.

I ran a 'small' cluster of 24 nodes (300GB+ of RAM) and while I know I could have done the same thing with super optimized code on a single machine, not having to worry about each node's performance but instead thinking in pipelines is refreshing. My goal was not to use/make optimal tokenizer or stop-word remover but instead to make sure every part of the chain could be done in parallel.

My biggest complaint? Repeatedly having to restart the cluster because nodes stop responding or throw arcane errors. If these reliability fixes were in place last week, I would have easily been 20-25% more efficient with my time. Can't wait till Databricks team deploys this for their users.

Re: Apache Spark Scale: A 60 TB+ production use case

#23

Post by peterstjohn »

zaptheimpaler wrote: post #3

Looks awesome, the work on performance especially is much appreciated :D I would love to be able to get flame graphs on my spark cluster per node too - any ideas on how exactly FB does this?

I knocked up a quick Ansible script this morning. It is _very_ rough right now and is more like 'here's what you could do' (and you have to compile perf-map-agent beforehand), but I got it working on a small cluster: https://github.com/falloutdurham/spark-flame

Re: Apache Spark Scale: A 60 TB+ production use case

#25

Post by vgt »

paulasmuth wrote: post #18
Swinx43 wrote: post #2

This is very intriguing, I cannot help but wonder how this operation would have performed on something like Google Big Query. I know that it is highly unlikely that Facebook would ever load their data to the Google Cloud Platform, but it would be an interesting comparison.

I can't speak for big query, but for our open-source BigQuery alternative EventQL [0]. On the upside, massively parallel SQL databases allow you to express what probably took facebook a lot of code in a single SQL query. On the downside, with any database that supports streaming inserts and dynamic resharding, you will have to eventually write each entry more than once. The more manual approach with spark prevents th…

It's exciting to see new innovative open source contributions to the big data space. Since BigQuery has moved on in many ways from the ColumnIO and Dremel concepts described in the Dremel paper[0] with new versions of many of these components (like in-memory shuffle, new storage, new ingest, execution engine, etc), I'd love to learn where EventQL places itself in that gradient.

[0]http://static.googleusercontent.com/media/research.google.co...

Re: Apache Spark Scale: A 60 TB+ production use case

#26

Post by lightcatcher »

I tried (and failed after a month of work) earlier this year trying to do ~20TB shuffles with Spark. I felt both relief and frustration reading this post.

Relief: I'm not an idiot, and the problems in the shuffle were likely in Spark and not just me being a beginner user.

Frustration: I wanted to group about 100 billion x 200 byte records into 5 billion groups. This seems like exactly the problem Spark is designed for and is advertised for. I had great difficulty even getting example Spark SQL code (or my own RDD based code) working. I hear so many great things about Spark as a tool for big data, and also "your data isn't big". I considered 20TB on the "low end" of big data, but a seemingly popular and widely used big data tool can't shuffle it without numerous bug fixes and pain on the part of the user. Shuffling 90TB was worth a Facebook blog post! This makes me ask: To all the people using Spark for "big data", how painful is it and how much data are you handling? It appears the answer for >=20TB is "very painful", and for <=5TB I think you're generally in "handle on single node" territory.

Re: Apache Spark Scale: A 60 TB+ production use case

#27

Post by brianwawok »

openasocket wrote: post #17
mozumder wrote: post #13

Couldn't they have just used Postgres on one 4/8-socket server with RAID? A 60TB dataset can fit in one server. Isn't Spark intended for massive clusters, like dozens or hundreds of servers, over petabytes of data?

Note that it's 60TB compressed Mostly it's about speed. With multiple machines you can bring more cores to bear for the processing, and have more RAM to cache partial results. Postgres could certainly do the job, but I'd be surprised if it would run within an order of magnitude of these results.

IO is pretty huge as well. You can spend lots and lots and lots of money to buy 1 machine a hard drive that can read 60 TB fast. Or you can have 100 machines with the cheapest possible hard drive and smoke the total IO.

Re: Apache Spark Scale: A 60 TB+ production use case

#28

Post by [deleted] »

openasocket wrote: post #17
mozumder wrote: post #13

Couldn't they have just used Postgres on one 4/8-socket server with RAID? A 60TB dataset can fit in one server. Isn't Spark intended for massive clusters, like dozens or hundreds of servers, over petabytes of data?

Note that it's 60TB compressed Mostly it's about speed. With multiple machines you can bring more cores to bear for the processing, and have more RAM to cache partial results. Postgres could certainly do the job, but I'd be surprised if it would run within an order of magnitude of these results.

[deleted]

Re: Apache Spark Scale: A 60 TB+ production use case

#29

Post by firasd »

mozumder wrote: post #13

Couldn't they have just used Postgres on one 4/8-socket server with RAID? A 60TB dataset can fit in one server. Isn't Spark intended for massive clusters, like dozens or hundreds of servers, over petabytes of data?

Keep in mind that the systems required depend on the actual tasks, not just whether you can fit the data on a disk. I don't think the FB Newsfeed (show more photos from your cousin because you liked their text post last week...) can be built using only SQL.
Post reply on HN