Facebook's Top Open Data Problems

#1

Post by huangwei_chang » Tue, Nov 04, 2014, 4:33 PM UTC

Facebook's Top Open Data Problems
research.facebook.com

Re: Facebook's Top Open Data Problems

#2

Post by beagle3 » Tue, Nov 04, 2014, 8:37 PM UTC

Something does not add up about hive: They say it has 300 PB, and it generates 4PB per day - which means, at this rate, all data was generated within the last 75 days.

Re: Facebook's Top Open Data Problems

#3

Post by boomzilla » Tue, Nov 04, 2014, 8:43 PM UTC

beagle3 wrote: post #2

Something does not add up about hive: They say it has 300 PB, and it generates 4PB per day - which means, at this rate, all data was generated within the last 75 days.

Most likely that the 300PB are distilled/normalized/compacted data whereas the 4PB per day are raw logs.

Re: Facebook's Top Open Data Problems

#4

Post by bbillings » Tue, Nov 04, 2014, 8:49 PM UTC

boomzilla wrote: post #3

beagle3 wrote: post #2

Something does not add up about hive: They say it has 300 PB, and it generates 4PB per day - which means, at this rate, all data was generated within the last 75 days.

Most likely that the 300PB are distilled/normalized/compacted data whereas the 4PB per day are raw logs.

This is the correct answer. Source: I am a FB employee.

Re: Facebook's Top Open Data Problems

#5

Post by beagle3 » Tue, Nov 04, 2014, 9:04 PM UTC

bbillings wrote: post #4

boomzilla wrote: post #3

Earlier quoted context omitted.

Most likely that the 300PB are distilled/normalized/compacted data whereas the 4PB per day are raw logs.

This is the correct answer. Source: I am a FB employee.

Thanks.

So, how much does the 4PB of data go down to when it goes into hive? My guess would be something like 200TB.

Is it zipped (or lz4/lzo/zopfli/lzma whatever)? Or is it just "distilled"?

Re: Facebook's Top Open Data Problems

#6

Post by EricBurnett » Tue, Nov 04, 2014, 9:07 PM UTC

I strongly dislike Facebook the product, and to lesser extent Facebook the company, but I'm continually impressed with Facebook's approach to engineering in the open. I find this an interesting dichotomy. Would I want to work there? I still don't think so, but my opinion on that front is getting less strong over time.

Re: Facebook's Top Open Data Problems

#7

Post by zeroonetwothree » Tue, Nov 04, 2014, 9:17 PM UTC

beagle3 wrote: post #2

Something does not add up about hive: They say it has 300 PB, and it generates 4PB per day - which means, at this rate, all data was generated within the last 75 days.

Data is not stored forever. If most data is only stored ~30 days, then the numbers make sense.

Re: Facebook's Top Open Data Problems

#8

Post by mandeepj » Tue, Nov 04, 2014, 9:18 PM UTC

Can any body throw some light on how facebook's database is designed? I am sure it will be an interesting read.

I was reading somewhere sometime back that each user at fb has its own database. I think that is not possible.

edit: I am googling now again on this topic. First link found is http://www.quora.com/What-is-Facebooks-database-schema

Re: Facebook's Top Open Data Problems

#9

Post by Thaxll » Tue, Nov 04, 2014, 9:21 PM UTC

Still using Memcache wow.

Re: Facebook's Top Open Data Problems

#10

Post by tristanz » Tue, Nov 04, 2014, 9:28 PM UTC

mandeepj wrote: post #8

Can any body throw some light on how facebook's database is designed? I am sure it will be an interesting read. I was reading somewhere sometime back that each user at fb has its own database. I think that is not possible. edit: I am googling now again on this topic. First link found is http://www.quora.com/What-is-Facebooks-database-schema

https://www.cs.cmu.edu/~pavlo/courses/fall2013/static/papers...