R and pandas and what I've learned about each • Hacker News Board

Re: R and pandas and what I've learned about each

#21

Post by takluyver » Tue, Mar 05, 2013, 5:52 PM UTC

oddthink wrote: post #13

wesm wrote: post #10

Earlier quoted context omitted.

Comparisons with data.table on performance are much more favorable than with vanilla R or plyr; a lot of progress has been made last couple years, too. I personally find the data.table syntax to be a bit obtuse at times but it's a great library.

Aside from the performance differences, data.table makes it very easy to do interactive manipulation, at the cost of making it hard to program. Pandas currently goes in the opposite direction. I'd rather have R/data.table at the prompt and python/pandas in my script, but if you have to err on one side, the python/pandas "low magic" is the side to err on. Pandas does have its own strange corners, though. For example,…

I'd guess the benefits are related to performance - Wes is known as something of a speed junkie (see also his vbench project). I know there's quite a bit of code in pandas that makes it much faster than a naive implementation of a similar interface.

That said, if it causes unexpected behaviour, check to see whether it's a bug.

Re: R and pandas and what I've learned about each

#22

Post by pav3l » Tue, Mar 05, 2013, 5:54 PM UTC

I am in the process of migrating all of my analyses from Matlab and R to Python. I have been meaning to do this for quite some time and finally pandas is mature enough to be able to completely replace both Matlab and R for straightforward tasks. If I need something Python doesn't offer, it's still fairly simple to do isolated tasks elsewhere. For me the biggest reasons for change are easy integration with the web and better language features (for Matlab, R language is great, just terribly slow for intensive tasks).

What I miss the most:

- Matlab Excel link (on Windows) - an excel add-on that lets you send back and forth arrays very easily. You need a spreadsheet when you work with datasets, and interchanging data through files just isn't that convenient.

- Matlab's IDE features (debugging, documentation, publishing, variable inspection).

- ggplot2

Re: R and pandas and what I've learned about each

#23

Post by hernamesbarbara » Tue, Mar 05, 2013, 6:46 PM UTC

pav3l wrote: post #22

I am in the process of migrating all of my analyses from Matlab and R to Python. I have been meaning to do this for quite some time and finally pandas is mature enough to be able to completely replace both Matlab and R for straightforward tasks. If I need something Python doesn't offer, it's still fairly simple to do isolated tasks elsewhere. For me the biggest reasons for change are easy integration with the web and…

Thanks for the comment! We've actually been thinking about some of these ideas too. There's a Yhat Excel plugin-in in the works, so stay tuned! Should be available shortly.

Re: R and pandas and what I've learned about each

#24

Post by jmmcd » Wed, Mar 06, 2013, 11:17 AM UTC

takluyver wrote: post #19

teoruiz wrote: post #11

To install the whole set of Python modules needed and iPython in a virtualenv (trick: there is no "pylab" module to install): % virtualenv --distribute --no-site-packages pandas_venv [blahblah] % . pandas_venv/bin/activate (pandas_venv) % easy_install readline # Probably only needed in Mac OS X for iPython to behave [blahblah] (pandas_venv) % pip install ipython [blah blah] (pandas_venv) % pip install numpy [lots of…

Installing numpy with pip isn't recommended: it might not work (if you don't have the necessary development headers to compile it), the resulting numpy might be slower (if it hasn't managed to compile against properly optimised libraries) and compiling from source isn't a very quick way to install it. For most users, the easiest way to get set up is a complete Python distribution, like Anaconda, EPD or Python(x,y). S…

I have wasted so many hours failing to get everything I need to work on OSX 10.6. I tried pip, easy_install, install from source, brew, Enthought. Now Anaconda has just saved my life, thank you! It was the best moment of my day when I was able to type all this without error:

    >>> import numpy as np
    >>> import scipy as sp
    >>> import statsmodels as sm
    >>> import matplotlib as mpl
    >>> import pandas as pd
    >>> import networkx as nx
    >>> import sklearn as sk
    >>> import nltk