Glad they added .query() to Pandas in .13. In general I find the methods in Pandas/NumPy much more consistent with general programming constructs than most of what I see in R. No doubt Hadley has bolted on a lot of great functionality to R, but the Rcpp dependency/GPL license is a turn off.
Dataframes – Julia, R, Python
21–30 of 39 posts
Re: Dataframes – Julia, R, Python
#22There was a very interesting design discussion by JMW on the julia-dev forum about nullable arrays, and column dtypes: https://groups.google.com/forum/#!topic/julia-dev/hS1DAUciv3... Even so - right now Pandas is miles ahead of the Julia equivalent. Pandas was the brain child of Wes McKinney - an amazing coder, who really, really cared about speed (who recently also made a lot of money selling his start up to Clouder…
I am learning J for about a year now, and I will try out the examples in it to see how it goes. Memory-mapped files, and quick array operations. It is APL based, and J has been around since the 80s, and is open source. Wes McKinney seems to think it is a good way to go:
Re: Dataframes – Julia, R, Python
#23For dataframe-like operations, I've started wondering why more languages don't take the dplyr approach and simply default to using something like SQLite under the hood. Granted, I are no super data genius, but every time I start cracking a little into the internals of a dataframe implementation, I get the sinking feeling that SQL databases have already done the hard work of indexes and efficient data structures.
Check out python blaze:
Pandas and linq style frontend to lots of different backends.
Soon there will be OOC array ops. http://blaze.pydata.org/docs/dev/index.html http://nbviewer.ipython.org/url/blaze.pydata.org/notebooks/t...
Re: Dataframes – Julia, R, Python
#24Earlier quoted context omitted.
I thought that before writing dplyr, but now I see that there a big differences. Relational databases are designed to work with large datasets on disk, and to accept changes very rapidly. The demands for in memory data analytics are quite differnt. Columnar data stores are a better fit, but it's pretty easy to bang out efficient code for in memory data; it's much harder to work with out of memory data.
> large datasets on disk I saw this benchmark a while back comparing Pandas to SQLite in-memory databases. While Pandas did edge out SQLite in several areas, it was by well under an order of magnitude: http://wesmckinney.com/blog/?p=414 Pretty solid performance plus the ability to work with large datasets on disk seemed like a pretty big win to me. I could imagine a set of SQLite extensions (a la spatialite) that cou…
That benchmark is only for joins? That's a pretty small part of analytic workflows in my experience.
Re: Dataframes – Julia, R, Python
#25Glad they added .query() to Pandas in .13. In general I find the methods in Pandas/NumPy much more consistent with general programming constructs than most of what I see in R. No doubt Hadley has bolted on a lot of great functionality to R, but the Rcpp dependency/GPL license is a turn off.
What's wrong with Rcpp?
And what's wrong with GPL? Unless you're planning on distributing your code, you shouldn't even need to think about it.
Re: Dataframes – Julia, R, Python
#26Earlier quoted context omitted.
> large datasets on disk I saw this benchmark a while back comparing Pandas to SQLite in-memory databases. While Pandas did edge out SQLite in several areas, it was by well under an order of magnitude: http://wesmckinney.com/blog/?p=414 Pretty solid performance plus the ability to work with large datasets on disk seemed like a pretty big win to me. I could imagine a set of SQLite extensions (a la spatialite) that cou…
That benchmark is only for joins? That's a pretty small part of analytic workflows in my experience.
That's fair. Now I'm curious as to how a more complete set of benchmarks would look using in memory sqlite, and what the opportunity for extension would be.
Re: Dataframes – Julia, R, Python
#27I've seen a lot about Julia on the last months, it seems like a good language (performance and kind of a nice syntax), For me, what makes R a very good choice is because of RStudio. Being able to play there with your data and save it all for later is one of the biggest reasons to use RLang. The Python equivalent would be emacs org-mode, which is great, but not as graphical as RStudio. Julia seems like a good language…
If you're looking for a nice graphical way to play with data in Python, may I suggest IPython Notebook [0]? It's not always easy to configure, but it's maturing fast and lets you have Python code, Markdown, and graphs in one place, not to mention the 21 other languages available natively or as add-ons[1]. [0]: http://ipython.org/notebook.html . [1]: https://github.com/ipython/ipython/wiki/IPython%20kernels%20... (I'm…
I just checked ipython notebook. Looks like Matlab Notebook, but in the browser instead of MS Word (I don't know if Matlab Notebook still exists, it's been a long time since the last time I used Windows), anyway, you should try org mode on emacs for python, is way more versatile, compiles to latex, html, and has almost all the features of the ipython notebook.... still Rstudio has a lot of graphical incentives, like data edition, environment variables, easy to follow documentation. Both org-mode and RStudio are very powerful tools, I was mostly ranting about how many options we have for making almost the same thing.
Re: Dataframes – Julia, R, Python
#28Earlier quoted context omitted.
If you're looking for a nice graphical way to play with data in Python, may I suggest IPython Notebook [0]? It's not always easy to configure, but it's maturing fast and lets you have Python code, Markdown, and graphs in one place, not to mention the 21 other languages available natively or as add-ons[1]. [0]: http://ipython.org/notebook.html . [1]: https://github.com/ipython/ipython/wiki/IPython%20kernels%20... (I'm…
There is a version of iPython that works with Julia too
Yeah you can run IPython Notebook with a Julia kernel.
Re: Dataframes – Julia, R, Python
#29Anyone doing R comparisons should use data.table instead of data.frame. More so for benchmarks. data.table is the best data structure/query language I have found in my career. It's leading the way in The R world, and in my way, in all the data-focused languages.
Re: Dataframes – Julia, R, Python
#30I've seen a lot about Julia on the last months, it seems like a good language (performance and kind of a nice syntax), For me, what makes R a very good choice is because of RStudio. Being able to play there with your data and save it all for later is one of the biggest reasons to use RLang. The Python equivalent would be emacs org-mode, which is great, but not as graphical as RStudio. Julia seems like a good language…
If you're looking for a nice graphical way to play with data in Python, may I suggest IPython Notebook [0]? It's not always easy to configure, but it's maturing fast and lets you have Python code, Markdown, and graphs in one place, not to mention the 21 other languages available natively or as add-ons[1]. [0]: http://ipython.org/notebook.html . [1]: https://github.com/ipython/ipython/wiki/IPython%20kernels%20... (I'm…
Another great graphical way to play with data in Python is using Spyder. It has an Rstudio/matlab sort of interface