21–30 of 86 posts
I did the analysis and worked with Stephen on the science side of it. If anyone would like to ask questions about what we did, I'd be happy to answer them. There's still lots more interesting stuff to do, but it was enough for a blog post. Suggest away if you think we missed something obvious!
Really nice work.
Earlier quoted context omitted.
What took the most work? What was the most painful or repetitive bit? What new primitives in Mathematica would you like to see?
Data wrangling. So I wrote my own "DataFrame" -- we have an official one coming to Mathematica 10, too. Also, binning. There is a nice theory for multidimensional binning and aggregation [that I haven't seen anyone describe explicitly so far]. So I wrote primitives. They play nicely with plotting, statistics, etc. That'll also be in Mathematica 10. Lots of Go for data egress. It's perfect for it.
Also: why Go instead of a JVM lang that can interop directly with Mathematica via JLink?
Finally: Will Mathematica directly support doing the full stack of this kind of work, including the egress?
Wolfram Connection would like to access your public profile, friend list, email address, custom friends lists, News Feed, relationships, birthday, status updates, checkins, education history, hometown, current city, photos, religious and political views, videos, likes and your friends' relationships, birthdays, education histories, hometowns, current cities, photos, religious and political views and videos.
I did the analysis and worked with Stephen on the science side of it. If anyone would like to ask questions about what we did, I'd be happy to answer them. There's still lots more interesting stuff to do, but it was enough for a blog post. Suggest away if you think we missed something obvious!
how would you effectively compete with distributed computing frameworks such as Pregel, MapReduce, and Dremel, when Mathematica is primarily used as a desktop application for in-RAM datasets? I know that Mathematica supports various parallelism options (such as multicore and grid), but frankly to gather real information requires much deeper probing, which far higher numbers of people, graph clustering/centrality on b…
You can even synthesize C code from Mathematica (there is a symbolic subset of C in it already) and have Mathematica run the appropriate build process for you, so things can get pretty interesting with that alone.
Out-of-core processing of large datasets is already on the roadmap for Mathematica 10. We plan to have a domain-specific language to describe and work with external [or in-memory] datasets in an efficient way, translating as appropriate to the native database query languages. Our 'native' format will be HDF5.
Ultimately, though, I think we'll rely on code generation to compile Mathematica to LLVM or transpile it to Go, so that we can distribute chunks of computation out to a cluster using M as command-and-control.
The idea would be that you can create and test large processing pipelines from inside Mathematica and then distribute them across a cluster in an ad-hoc way, then visualize the progress, track errors, and analyze the results. Notebooks are really good for that kind of lightweight UI.
This isn't a new idea, but in a language as dynamic as Mathematica, I think it could be especially powerful. Of course, it is also tricky because type inference would be a big part of making this idea possible in a dynamically typed, symbolic language like Mathematica. But not impossible, I don't think. And functional languages already have demonstrated advantages in this type of situation -- take stream fusion in Haskell.
Very nice looking graphs, but running "Wolfram Alpha Personal Analytics for Facebook" for my own profile comes with a rather nerve-wracking warning: Wolfram Connection would like to access your public profile, friend list, email address, custom friends lists, News Feed, relationships, birthday, status updates, checkins, education history, hometown, current city, photos, religious and political views, videos, likes an…
Earlier quoted context omitted.
Data wrangling. So I wrote my own "DataFrame" -- we have an official one coming to Mathematica 10, too. Also, binning. There is a nice theory for multidimensional binning and aggregation [that I haven't seen anyone describe explicitly so far]. So I wrote primitives. They play nicely with plotting, statistics, etc. That'll also be in Mathematica 10. Lots of Go for data egress. It's perfect for it.
What can Mathematica bring to the DataFrame concept that hasn't been done before? Also: why Go instead of a JVM lang that can interop directly with Mathematica via JLink? Finally: Will Mathematica directly support doing the full stack of this kind of work, including the egress?
2. I used Go because I am very productive in Go and like a lot of things about it. Goroutines are neat. Java is fine, it's just very boilerplatey, and I'm not practiced enough at it to get past that. And I don't see why we can't develop a GoLink as well.
3. Probably not the whole stack, at least in the beginning. But we'll get there. We want to make it really easy to spider websites and so on.
How is it possible that Facebook, which owns the data, does not give tools like these but others tap this using their data?
Hence there are vastly more resources dedicated to assimilating eg photos and games into their ecosystem, than into something computationally innovative.
This is IMHO a huge mistake, since they could instead be introducing simple forms of programming that takes you on a continuous curve from using the product, to developing for it. There is a huge hunger in the masses for better forms of programming.
This point will probably become obvious if the Wolfram Language is successful.
People donate their data to support Wolfram's closed-source, paid-license, for-profit program?
Very nice looking graphs, but running "Wolfram Alpha Personal Analytics for Facebook" for my own profile comes with a rather nerve-wracking warning: Wolfram Connection would like to access your public profile, friend list, email address, custom friends lists, News Feed, relationships, birthday, status updates, checkins, education history, hometown, current city, photos, religious and political views, videos, likes an…
How do you think they are going to get all the data to analyze your facebook account?
You have to opt in to being a data donor for us to store any of it.
Otherwise we just record basic anonymized statistics -- like number of friends, sex, age, etc... and throw all the detailed stuff away. Our privacy policy has more: http://www.wolframalpha.com/fbfaqs.html
We also encrypt with public keys like there's no tomorrow.
I did the analysis and worked with Stephen on the science side of it. If anyone would like to ask questions about what we did, I'd be happy to answer them. There's still lots more interesting stuff to do, but it was enough for a blog post. Suggest away if you think we missed something obvious!
https://thestrangeloop.com/news/strange-loop-2012-video-sche...