Data Science of the Facebook World • Hacker News Board

Re: Data Science of the Facebook World

#81

Post by taliesinb » Thu, Apr 25, 2013, 11:50 PM UTC

clebio wrote: post #73

taliesinb wrote: post #18

Earlier quoted context omitted.

Data wrangling. So I wrote my own "DataFrame" -- we have an official one coming to Mathematica 10, too. Also, binning. There is a nice theory for multidimensional binning and aggregation [that I haven't seen anyone describe explicitly so far]. So I wrote primitives. They play nicely with plotting, statistics, etc. That'll also be in Mathematica 10. Lots of Go for data egress. It's perfect for it.

How long did the whole piece take to put together, and what's the rough break-down of time spent on each component (data wrangling, finding useful sorts, visualizations, write-up)? Thanks!

Fulltime, around 6 weeks. Breakdown is hard to say.

I wasted a lot of time trying to do things the "traditional" way by loading into SQL, querying, etc, but it was actually much faster to process things in memory (I have a 16 gig machine). Intensive stuff was parallelized in Go and used ordinary filesystem with directory prefix tries for performance.

Writeup was mostly SW. He's worked on it maybe an afternoon a week for the last month.

I really enjoy visualizations and can iterate extremely fast (e.g. ChordPlot took half an hour). Don't know why M is not the defacto standard for dataviz people. Tweaking takes a long time, and design iterated with me on getting things looking really nice.

All in all, most of my time was spent building tools to easily create multidimensional histograms. The nice thing is that those tools are clearly useful enough we'll integrate them into Mathematica, so the cost is somewhat amortized.

NLP took a few weeks of Etienne's time... once again, amortized. Most of that is wrangling, really, and building tools to understand the deficiencies of your training set. Naive Bayes works surprisingly well, the magic is in the tooling and "human intelligence" you iterate with.

Re: Data Science of the Facebook World

#82

Post by taliesinb » Thu, Apr 25, 2013, 11:52 PM UTC

szhorvat wrote: post #75

taliesinb wrote: post #43

Earlier quoted context omitted.

Interesting points. 1. Dynamics You're right, that would be very interesting. The most obvious way we could have done this is by looking at the spread of our app itself as people started to use it. Unfortunately, we only started recording anonymized stats for the second release, so we've somewhat missed the boat there. To do it with links and general "memes" would be technically much harder, because we'd have to peri…

About the "fictive power law" thing: this is THE paper to read: http://arxiv.org/abs/0706.1062 (it's an easy read, explaining what the maximum likelihood method is, etc.). Despite what they say, fitting the log-log CDF usually gives pretty good results when done right (fitting the PDF does not)

Thanks! What do you know? Shalizi!

Re: Data Science of the Facebook World

#83

Post by taliesinb » Thu, Apr 25, 2013, 11:56 PM UTC

xk_id wrote: post #68

The traditional way to plot the assortativity by age is using a scatter plot / heatmap. This is similar to what they did for country homophily on p12 of the Facebook anatomy paper. The result would be a plot with a prominent diagonal, illustrating that "same attracts same". That aside, imo, Facebook is an incredibly idiosyncratic "app", which makes almost no sense. And yet, it gave us so many opportunities for intere…

Yeah, we tried a couple. Those heatmaps I think are quite hard to read.. because it is natural to want to take marginals, but you can't easily do that visually.

I think this whole "octile plot" thing turned out quite nicely. It's in a sense a way of 'slicing' the CDF into 8 even strips and projecting them onto a single axis. It's quite intuitive to read, too. Facebook seems to use it too for some of their papers.

Re: Data Science of the Facebook World

#84

Post by taliesinb » Thu, Apr 25, 2013, 11:58 PM UTC

pbnjay wrote: post #72

taliesinb wrote: post #33

Earlier quoted context omitted.

Actually, NONE of the people in our dataset had zero friends. The x-axis starts at 1, not 0. The point is that resampling to remove the friendship paradox shows that there are many more people with single-digit friends than we expected.

Given mcintyre1994's comment, I think this still explains the same situation. People with single-digit friends are simply people who have friends blocked to apps but have multiple friends who've donated data.

No, we can tell when someone's privacy settings have given them a zero friend count, and they're not included in that aggregation.

Re: Data Science of the Facebook World

#85

Post by carlob » Fri, Apr 26, 2013, 8:40 AM UTC

taliesinb wrote: post #82

szhorvat wrote: post #75

Earlier quoted context omitted.

About the "fictive power law" thing: this is THE paper to read: http://arxiv.org/abs/0706.1062 (it's an easy read, explaining what the maximum likelihood method is, etc.). Despite what they say, fitting the log-log CDF usually gives pretty good results when done right (fitting the PDF does not)

Thanks! What do you know? Shalizi!

Also from Shalizi, if you want a TL;DR

http://vserver1.cscs.lsa.umich.edu/~crshalizi/weblog/491.htm...

Re: Data Science of the Facebook World

#86

Post by stared » Fri, Apr 26, 2013, 4:51 PM UTC

reeses wrote: post #76

stared wrote: post #61

Earlier quoted context omitted.

Well, it is always "people how clicked on FB that they like X", regardless whether it is shallow or deep interest, genuine or ironic, or random, or "because my friends like it and I want to be as cool as them" etc. If more fine grained, I wouldn't be surprised to see ties between seemingly exclusive things... e.g. in this http://meta.stackoverflow.com/questions/157976/map-of-all-se... (not interesting, but participat…

That's not precisely correct. In the old days, we typed in our favorite bands/books/music. The attempted canonicalization created some amusing interpretations of song as band, book as author, band as book, etc. It has also made it exceptionally annoying to get feed updates from the 900 bands, actors, movies, songs, etc. that I said I liked and now have to "unlike". I think you could correlate people within a certain…

Sure, I expect data to be extremely noisy, but I am thinking about looking at very robust things (e.g. this guy is in rock-like perhaps-indie music), not making it too fine-grained.