Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook's Top Open Data Problems

research.facebook.com

Post reply on HN
Print view View original

41–50 of 65 posts

Re: Facebook's Top Open Data Problems

#41

Post by nbm »

mmmooo wrote: post #17

So ~650M daily active users..4PB of data warehouse created each day, that means ~7MB of new data on each active user per day. Given that its data warehouse, I'm going to guess its not images, seems like a lot to me. I guess it shouldn't surprise anyone that every interaction on and off the site, is heavily tracked.

A lot of that data is duplicated to allow for efficient querying or transformation. It often is too slow to process the data as it comes in, so an initial process will write the data in a raw form, and some other process might select a subset of the data to process, and then submit it in an "annotated" form (filling in, say, the AS number of the client IP). Another process will run later in a batched fashion and perhaps annotate the full set of information and summarize it into a bunch of easily-queried tables.

A lot of that data is also not tied to individuals either - for example the access logs for the CDN (which, being on a different domain by design, does not share cookies so is not attached to an account) even reasonably heavily sampled is probably tens of gigabytes a day, and is rolled up into efficient forms for queries in various ways. A lot of it isn't even about requests coming through the web site/API - it may just be internal inter-service request information, or inter-datacenter flow analysis, or per-machine service metrics ("Oh, look, process A on machines B through E went from 2GB resident to 24GB in 30 seconds a few seconds before the problem manifested").

(Not that it makes too much of a difference at this scale, but it is closer to 860M daily actives.)

Re: Facebook's Top Open Data Problems

#42

Post by Sven7 »

EricBurnett wrote: post #6

I strongly dislike Facebook the product, and to lesser extent Facebook the company, but I'm continually impressed with Facebook's approach to engineering in the open. I find this an interesting dichotomy. Would I want to work there? I still don't think so, but my opinion on that front is getting less strong over time.

That I am sure is also what they said about the Manhattan project. Anyways the train has left the station.

Re: Facebook's Top Open Data Problems

#43

Post by rattray »

Sven7 wrote: post #42
EricBurnett wrote: post #6

I strongly dislike Facebook the product, and to lesser extent Facebook the company, but I'm continually impressed with Facebook's approach to engineering in the open. I find this an interesting dichotomy. Would I want to work there? I still don't think so, but my opinion on that front is getting less strong over time.

That I am sure is also what they said about the Manhattan project. Anyways the train has left the station.

The Manhattan Project was engineering in the open?

EricBurnett was talking about appreciating (some of) FB's values, not relishing the scale of their challenges (another reason people join FB). It's the latter reason that I imagine engineers joined The Manhattan Project (other than those who viewed it as a way to protect against nefarious forces in the world, valid or not).

Re: Facebook's Top Open Data Problems

#44

Post by ransom1538 »

I had a really great time talking to the Facebook engineers during my interviews there. The main pattern I noticed was Harvard (i was applying in management). Even more so, the guys interviewing me were extremely talented and smart. What always weirded me out was... the problems they work on are not that difficult. Once you grasp sharding and operations, you are pretty much set. These guys are not the Manhattan project. The true hard problems in their space: developing their own mobile hardware, keeping teens engaged, pushing the boundaries of design, losing tracking systems in mobile, etc; they don't face head on. Moving petabytes around or caching lots of things in memcache - my roomate and I could do with an aws account and a few beers. Memcache for god sakes is what 300 lines of C?

Re: Facebook's Top Open Data Problems

#45

Post by ransom1538 »

I had a really great time talking to the Facebook engineers during my interviews there. The main pattern I noticed was Harvard (i was applying in management). Even more so, the guys interviewing me were extremely talented and smart. What always weirded me out was... the problems they work on are not that difficult. Once you grasp sharding and operations, you are pretty much set. These guys are not the Manhattan project. The true hard problems in their space: developing their own mobile hardware, keeping teens engaged, pushing the boundaries of design, losing tracking systems in mobile, etc; they don't face head on. Moving petabytes around or caching lots of things in memcache - my roomate and I could do with an aws account and a few beers. Memcache for god sakes is what 300 lines of C?

Re: Facebook's Top Open Data Problems

#46

Post by IndianAstronaut »

EricBurnett wrote: post #6

I strongly dislike Facebook the product, and to lesser extent Facebook the company, but I'm continually impressed with Facebook's approach to engineering in the open. I find this an interesting dichotomy. Would I want to work there? I still don't think so, but my opinion on that front is getting less strong over time.

I was interviewing with them some time ago and just gave up half way. Those guys are absolute assholes and are hugely arrogant. Not a place I would want to work.

Re: Facebook's Top Open Data Problems

#47

Post by srcmap »

coolsunglasses wrote: post #24
crazypyro wrote: post #23

This is slightly off topic, but has any experienced an increase in "fake" toasts from facebook mobile? It seems if I haven't used facebook mobile in a few days or I don't respond to their toasts about very minor people in my life uploading a photo, I tend to start getting toasts that say "You have 5 notifications, 3 pokes and 2 messages.", then I open the app and it takes me to an unknown error page. Am I being too c…

It's been doing this for me via email lately and it's really annoying.

I use gmail's filter to dump them and 99% of my emails to various labels. Once every 2-4 weeks, when I am in the mood, I check them out then delete them all. :-)

Re: Facebook's Top Open Data Problems

#48

Post by srcmap »

mmmooo wrote: post #17

So ~650M daily active users..4PB of data warehouse created each day, that means ~7MB of new data on each active user per day. Given that its data warehouse, I'm going to guess its not images, seems like a lot to me. I guess it shouldn't surprise anyone that every interaction on and off the site, is heavily tracked.

FB and Google can clone what your thinking, maybe predict what your will be thinking? :-)

I wonder if they can predict with some percentage accuracy on what any particular active US user might vote for today base on the user's graph data?

Re: Facebook's Top Open Data Problems

#49

Post by Cakez0r »

nbm wrote: post #40
Cakez0r wrote: post #37

Earlier quoted context omitted.

Yes, that's the mechanism I'm wondering about. Let's say I have: Application -> TAO -> Cache -> Database I have a photo node (P1) and 125 comments nodes [C1, C2, ..., Cn] attached to P1 by the edges [(P1,C1), (P1,C2), ..., (P1,Cn)]. I'll ignore the fact that there can be different edge types for simplicity. Lets say my page size is 50 and I want to view 3 pages of comments for the photo from my application. My applic…

There is no memcache - there is just TAO, and it talks to the database. TAO is a read/write-through cache, so the only way the data changes in the database is through TAO. TAO contains the indices necessary to answer queries like that efficiently (although there may be other systems for doing similar but more specialized or slightly different queries), as well as a cache of the data. So, when you add a comment, TAO u…

Thank you!

Re: Facebook's Top Open Data Problems

#50

Post by e12e »

ransom1538 wrote: post #44

I had a really great time talking to the Facebook engineers during my interviews there. The main pattern I noticed was Harvard (i was applying in management). Even more so, the guys interviewing me were extremely talented and smart. What always weirded me out was... the problems they work on are not that difficult. Once you grasp sharding and operations, you are pretty much set. These guys are not the Manhattan proje…

No idea how much you're trolling, but the single file:

https://github.com/memcached/memcached/blob/master/memcached...

is more like 4000 lines of C (by guesstimating the amount of comments etc).

Post reply on HN