Live data from Hacker News

How the R-project is taking over statistical analysis software

sites.google.com

21–30 of 61 posts

Re: How the R-project is taking over statistical analysis software

#22
post #9

I use R as my primary data-analysis tool for almost all of my work, with occasional recourse to SAS for certain specialized models (e.g., PROC GLIMMIX for generalized mixed models). My only complaint is the awful default IDE, which can be mitigated to a large extent by scripting elsewhere and source()ing the script, and some odd edge behaviors including the mystifying row names of dataframes, the difficulty of droppi…

I love R--but I end up using Stata more often because it is easier to produce vector graphics that can be imported to Illustrator. I wish that the R community would start to focus on graphics.

Re: How the R-project is taking over statistical analysis software

#23
post #9

I use R as my primary data-analysis tool for almost all of my work, with occasional recourse to SAS for certain specialized models (e.g., PROC GLIMMIX for generalized mixed models). My only complaint is the awful default IDE, which can be mitigated to a large extent by scripting elsewhere and source()ing the script, and some odd edge behaviors including the mystifying row names of dataframes, the difficulty of droppi…

I love R--but I end up using Stata more often because it is easier to produce vector graphics that can be imported to Illustrator. I wish that the R community would start to focus on graphics.

I've had some success with output from lattice using Cairo's SVG option, although you're right that it's never easy. Self-citing, the plots in these pubs were generated as above (JoCN may be behind a paywall):

http://www.frontiersin.org/human_neuroscience/10.3389/fnhum.... http://www.mitpressjournals.org/doi/abs/10.1162/jocn_a_00089

I'll need to give Stata a look, too.

Re: How the R-project is taking over statistical analysis software

#24
post #15

Earlier quoted context omitted.

Surely that would still be the case under any license.

He is probably comparing R to SAS (which are the two most popular statistical programming languages). SAS doesn't really have libraries, instead you buy additional packages from SAS, which are very reliable and well supported, but expensive. My company shuns R (although I personally like it), primarily because of this issue. If we need to run a rare or uncommon statistical procedure, it is a lot easier to trust the S…

True, Though if you need to run a rare or uncommon stat procedure, SAS is not likely to have it in the core, and then you are back to using what "some grad student wrote".

Re: How the R-project is taking over statistical analysis software

#25
post #18
post #16

Unfortunately, it's almost impossible to work with a very large datasets in R, because of the speed limitations. Many researchers I know use Matlab because of this.

What about Octave? Other than my use in the Stanford Machine Learning class, I've never really used either, so I don't have any basis for comparison.

Octave is Matlab clone, in fact Octave developers openly say that except for some special cases, any difference between Octave and Matlab is a bug.

The biggest difference between Matlab and Octave is JIT compiler in Matlab, which does incredibly good job at vectorizing simple (or sometimes even not-so simple) loops.

I think it's fair to say that Octave performance is very close to a Matlab in a pre-JIT time.

There's also a huge difference in toolboxes, profiling, sparse matrix operations, parallel computing and many-many more. In these areas I'm afraid Octave is light-years behind Matlab.

However, you still can do a lot of useful simple stuff with Octave and it's free! Matlab-like syntax is really, really cool then it comes to vectorized operations. So probably these two reasons determined Andrew Ng's choice of Octave as a main environment for ml-class. Huge win for Octave I guess. This might spur some interest in the development, attract new people to the product. I think it's a well-deserved success for John W Eaton and other people who develop(ed) Octave all these years.

Re: How the R-project is taking over statistical analysis software

#26
post #25
post #18

Earlier quoted context omitted.

What about Octave? Other than my use in the Stanford Machine Learning class, I've never really used either, so I don't have any basis for comparison.

Octave is Matlab clone, in fact Octave developers openly say that except for some special cases, any difference between Octave and Matlab is a bug. The biggest difference between Matlab and Octave is JIT compiler in Matlab, which does incredibly good job at vectorizing simple (or sometimes even not-so simple) loops. I think it's fair to say that Octave performance is very close to a Matlab in a pre-JIT time. There's…

I agree with your take on Octave performance relative to Matlab. The Matlab parallel toolbox is getting more and more useful in a multicore world.

As you note, the Matlab profiler is very nice. You can zero in on the 80% of the 80/20 tradeoff very fast, during your usual development cycle. It's as simple as:

>> profile on >> do_something >> profile report

and you get a nice graphical/textual report on time usage in everything do_something called.

Re: How the R-project is taking over statistical analysis software

#27
I think open-source eventually replaces commercial products, in the same way that proprietary products become commoditized. The response for commercial products is also the same: continual differentiation, adding new features, benefits, support, documentation etc. Exceptions are also the same: natural monopolies (e.g. strong network effects).

Open-source is great at hill-climbing, where there are clear directions for improvement and especially for features that are obviously needed by users (provided the structure of the project is sufficiently modular to facilitate it), by tapping the collective intelligence of users.

It's not great at "hill-hopping": originating radically different products.

Re: How the R-project is taking over statistical analysis software

#29
post #27

I think open-source eventually replaces commercial products, in the same way that proprietary products become commoditized. The response for commercial products is also the same: continual differentiation, adding new features, benefits, support, documentation etc. Exceptions are also the same: natural monopolies (e.g. strong network effects). Open-source is great at hill-climbing, where there are clear directions for…

Counter-examples abound. Can you name even one open source app that has displaced a mature, user-facing desktop app with a non-trivial UI, other than a web browser?

Open source only seems to win in domains in which it makes sense for companies to share work in order to compete at a higher tier of functionality.

Post reply on HN