71–80 of 86 posts
How much of the friends with zero friends is simply because that information is blocked? If my friends "donated" their data, I would show as having 0 friends if I've blocked that information to apps.
All, since they're friends with the person who donated the data.
How much of the friends with zero friends is simply because that information is blocked? If my friends "donated" their data, I would show as having 0 friends if I've blocked that information to apps.
Actually, NONE of the people in our dataset had zero friends. The x-axis starts at 1, not 0. The point is that resampling to remove the friendship paradox shows that there are many more people with single-digit friends than we expected.
Earlier quoted context omitted.
What took the most work? What was the most painful or repetitive bit? What new primitives in Mathematica would you like to see?
Data wrangling. So I wrote my own "DataFrame" -- we have an official one coming to Mathematica 10, too. Also, binning. There is a nice theory for multidimensional binning and aggregation [that I haven't seen anyone describe explicitly so far]. So I wrote primitives. They play nicely with plotting, statistics, etc. That'll also be in Mathematica 10. Lots of Go for data egress. It's perfect for it.
I did the analysis and worked with Stephen on the science side of it. If anyone would like to ask questions about what we did, I'd be happy to answer them. There's still lots more interesting stuff to do, but it was enough for a blog post. Suggest away if you think we missed something obvious!
1. I noticed the interactive slider, embedded into the webpage. That's not what vanilla Mma 9 can do. Is there a simple way we can do the same without the CDF plugin (e.g. a package I'm not aware of) or is this future functionality?
2. The graph with the migration data looks nice. Mma 9 can't do an edge layout like this (curved edges) by default. Is this again custom code (custom Graphics or custom EdgeRenderingFunction) or is it future functionality?
3. There's the part with the frequency of various graph motives (the number of edges, triangles, and other weird shapes in graphs). How did you count these? Was it done using Mma? Some of these motifs are easy to count (there are simple expressions in therms of the adjacency matrix), but some others like the (1-2, 2-3, 1-3, 3-4) subgraph are not so easy.
4. How did you make the word clouds? Is the algorithm written in Mma? Here's a nice but slow one (the worse answer by myself): http://mathematica.stackexchange.com/questions/2334/how-to-c...
I found this interesting. What I would love to have seen, however, is a probe into the dynamics. You did a nice abstraction over time as you measured property X as age was varied. I would have loved to have seen the manner in which topics and ideas spread over your network. For instance: If an event occurred in New York, say, how long would it have taken to spread to San Francisco? If there were no progression, topic…
Interesting points. 1. Dynamics You're right, that would be very interesting. The most obvious way we could have done this is by looking at the spread of our app itself as people started to use it. Unfortunately, we only started recording anonymized stats for the second release, so we've somewhat missed the boat there. To do it with links and general "memes" would be technically much harder, because we'd have to peri…
Earlier quoted context omitted.
That's a good idea! Though if I remember correctly, interests aren't canonicalized, so it might be pretty messy. And I'm not sure if people fill them in non-ironically any more.
Well, it is always "people how clicked on FB that they like X", regardless whether it is shallow or deep interest, genuine or ironic, or random, or "because my friends like it and I want to be as cool as them" etc. If more fine grained, I wouldn't be surprised to see ties between seemingly exclusive things... e.g. in this http://meta.stackoverflow.com/questions/157976/map-of-all-se... (not interesting, but participat…
It has also made it exceptionally annoying to get feed updates from the 900 bands, actors, movies, songs, etc. that I said I liked and now have to "unlike".
I think you could correlate people within a certain confidence, but because of the nature of the data, you would have to expect a surprisingly large dissimilarity of interests within a clique over a certain account age based on this noise. Not 90%, but higher than the real value.
Earlier quoted context omitted.
how would you effectively compete with distributed computing frameworks such as Pregel, MapReduce, and Dremel, when Mathematica is primarily used as a desktop application for in-RAM datasets? I know that Mathematica supports various parallelism options (such as multicore and grid), but frankly to gather real information requires much deeper probing, which far higher numbers of people, graph clustering/centrality on b…
There is already HadoopLink. LibraryLink allows you to write C or C++ that gets dynamically linked into the kernel at runtime (no restart required), which gives you freedom to create your own threads and do your own thing [and crash the kernel]. A lot of kernel development happens that way now. You can even synthesize C code from Mathematica (there is a symbolic subset of C in it already) and have Mathematica run the…
What you're saying about distributing a computation on a cluster sounds very interesting. I used Mathematica for a hybrid Mathematica/C++ calculation (LibraryLink) where the complexity was handled by Mathematica and the (simple) heavy lifting by C++. I used the standard parallel tools to run it on a cluster, which means that communication was done through MathLink.
I never went above ~70 CPUs, but people say that problems start to appear above that (too many MathLink connection): http://mathematica.stackexchange.com/questions/20356/mathema...
Another possible problem with my solution (LibraryLink, then Mma parallelization) was that it required a Mma license for as many kernels as I was running, even though most of them were only running the C++ code. But that's easy to fix on WRI's side.
People donate their data to support Wolfram's closed-source, paid-license, for-profit program?
I did the analysis and worked with Stephen on the science side of it. If anyone would like to ask questions about what we did, I'd be happy to answer them. There's still lots more interesting stuff to do, but it was enough for a blog post. Suggest away if you think we missed something obvious!
Several questions :-) 1. I noticed the interactive slider, embedded into the webpage. That's not what vanilla Mma 9 can do. Is there a simple way we can do the same without the CDF plugin (e.g. a package I'm not aware of) or is this future functionality? 2. The graph with the migration data looks nice. Mma 9 can't do an edge layout like this (curved edges) by default. Is this again custom code (custom Graphics or cus…
2. Custom Graphics. We don't (yet) support doing anything interesting with graph weights, which this relies heavily on. I think this is a good candidate for M10 -- name would probably be ChordPlot. I came up with a "GraphForm" construct that allows you to patch various graph properties into various visual parameters (size, color, edge weight, etc). That turns out to be quite useful.
3. Tally[list, IsomorphicGraphQ] . Isn't that cool?
4. Awesome! Nice code. We plan to create ImageCloud and WordCloud functions for M10. WordCloud will be specialized for representing word frequencies and so on. ImageCloud will be the general case: accept a list of images [potentially with transparency], and then find a nice layout given desired sizes. So much cool dataviz will be possible with this! Like country flags...
Earlier quoted context omitted.
There is already HadoopLink. LibraryLink allows you to write C or C++ that gets dynamically linked into the kernel at runtime (no restart required), which gives you freedom to create your own threads and do your own thing [and crash the kernel]. A lot of kernel development happens that way now. You can even synthesize C code from Mathematica (there is a symbolic subset of C in it already) and have Mathematica run the…
The out of code data processing is something that was sorely needed, and I was wishing for this for a long time. One of the big drawbacks of Mathematica for data processing was that it's only convenient to use if all the data can be read into memory. What you're saying about distributing a computation on a cluster sounds very interesting. I used Mathematica for a hybrid Mathematica/C++ calculation (LibraryLink) where…