How the R-project is taking over statistical analysis software • Hacker News Board

Re: How the R-project is taking over statistical analysis software

#41

Post by linhir » Sat, Oct 22, 2011, 4:17 AM UTC

roxtar wrote: post #10

dewarrn1 wrote: post #9

I use R as my primary data-analysis tool for almost all of my work, with occasional recourse to SAS for certain specialized models (e.g., PROC GLIMMIX for generalized mixed models). My only complaint is the awful default IDE, which can be mitigated to a large extent by scripting elsewhere and source()ing the script, and some odd edge behaviors including the mystifying row names of dataframes, the difficulty of droppi…

Default IDE? Do you mean the R interpreter REPL? If you are looking for a nice IDE for R, I would suggest RStudio: http://rstudio.org/

Totally agree, R studio has made my life much easier.

Re: How the R-project is taking over statistical analysis software

#42

Post by steve_cronin » Sat, Oct 22, 2011, 4:19 AM UTC

earl wrote: post #33

6ren wrote: post #27

I think open-source eventually replaces commercial products, in the same way that proprietary products become commoditized. The response for commercial products is also the same: continual differentiation, adding new features, benefits, support, documentation etc. Exceptions are also the same: natural monopolies (e.g. strong network effects). Open-source is great at hill-climbing, where there are clear directions for…

I don't think it's obvious that open source displaces commercial for scientific computing. For every example like R which has in many places displaced S-Plus, there are counterexamples like matlab, for which the open source clone Octave is a bad joke, at least the last time I tried using it: missing functions, slowness, extreme difficulty installing; or Mathematica, or eviews, or gauss, or Maple. One other potential…

Cant even compare R, a statistical engine to Matrix Laboratory, a numerical matrix manipulation engine. they are different software packages designed for different purposes. eg. plugging R into a high grade NMR magnet and processing signals is probably not a good idea.

R is sick though and I am alwalys pleasently surprised at what clever people are doing with it. Octave on the other hand shouldn't be used until someone writes a proper interface and decent graphing. MATLAB is a million light years ahead of Octave in that regard

Re: How the R-project is taking over statistical analysis software

#43

Post by sandGorgon » Sat, Oct 22, 2011, 4:44 AM UTC

cageface wrote: post #29

6ren wrote: post #27

I think open-source eventually replaces commercial products, in the same way that proprietary products become commoditized. The response for commercial products is also the same: continual differentiation, adding new features, benefits, support, documentation etc. Exceptions are also the same: natural monopolies (e.g. strong network effects). Open-source is great at hill-climbing, where there are clear directions for…

Counter-examples abound. Can you name even one open source app that has displaced a mature, user-facing desktop app with a non-trivial UI, other than a web browser? Open source only seems to win in domains in which it makes sense for companies to share work in order to compete at a higher tier of functionality.

VLC

Re: How the R-project is taking over statistical analysis software

#44

Post by jeffmk » Sat, Oct 22, 2011, 4:57 AM UTC

cageface wrote: post #29

6ren wrote: post #27

I think open-source eventually replaces commercial products, in the same way that proprietary products become commoditized. The response for commercial products is also the same: continual differentiation, adding new features, benefits, support, documentation etc. Exceptions are also the same: natural monopolies (e.g. strong network effects). Open-source is great at hill-climbing, where there are clear directions for…

Counter-examples abound. Can you name even one open source app that has displaced a mature, user-facing desktop app with a non-trivial UI, other than a web browser? Open source only seems to win in domains in which it makes sense for companies to share work in order to compete at a higher tier of functionality.

It has by no means "displaced" its proprietary equivalent, but Inkscape is one of the most user-friendly open source apps I've ever used. I find it far more intuitive than Illustrator. An incredible amount of power and complexity is presented in a way that makes it quite intuitive and a joy to use. It's also easily extensible if you're a programmer.

Re: How the R-project is taking over statistical analysis software

#45

Post by zzleeper » Sat, Oct 22, 2011, 5:49 AM UTC

jcdreads wrote: post #32

Probably worth noting about the author: > Robert A. Muenchen is the author of R for SAS and SPSS Users and, with Joseph M. Hilbe, R for Stata Users. He is also the creator of r4stats.com, a popular web site devoted to helping people learn R. Bob is a consulting statistician with 30 years of experience Disclaimer: I hate R's syntax, but my company's analytics group uses R for just about everything.

I started learning SAS (I mostly use stata/matlab/python for my daily needs) but also ended up abhorring some parts of the syntax..

Re: How the R-project is taking over statistical analysis software

#46

Post by muuh-gnu » Sat, Oct 22, 2011, 6:51 AM UTC

earl wrote: post #33

6ren wrote: post #27

I think open-source eventually replaces commercial products, in the same way that proprietary products become commoditized. The response for commercial products is also the same: continual differentiation, adding new features, benefits, support, documentation etc. Exceptions are also the same: natural monopolies (e.g. strong network effects). Open-source is great at hill-climbing, where there are clear directions for…

I don't think it's obvious that open source displaces commercial for scientific computing. For every example like R which has in many places displaced S-Plus, there are counterexamples like matlab, for which the open source clone Octave is a bad joke, at least the last time I tried using it: missing functions, slowness, extreme difficulty installing; or Mathematica, or eviews, or gauss, or Maple. One other potential…

> extreme difficulty installing

Today's Octave installs do not need more than click-click-ok-done or apt-get install octave.

> for which the open source clone Octave is a bad joke

I think you're not giving Octave enough credit, considering that they have only a few part-time developers, and that nobody does sponsor them, they have accomplished a respectable amount of functionality over the last 20 years, it is extermely unfair calling them a "bad joke".

Of course with those limited resources they are not able to match the output of Mathworks, but what they can do as of yet is usually more than universities teach, and _still_ many departments are kind of married to Matlab, only mention Matlab to students, give only matlab examples, matlab labs, matlab exercises, etc. Also very respectable people like Gilbert Strang, who gave the MIT basic Linear Algebra and Computational Science and Engineering classes, seem to have enough vested interests in Mathworks to not even mention Octave to students briefly as something they can download and work at home. Octave is extremely powerful and capable for what you pay for it, and deserves at least a mention.

It is probably not different at other universities and other departments. Several professors I had to deal with were similar, either not even aware that open source packages like Octave, Scilab, Maxima, Scipy exist at all, or extremely faithfully married to companies behind proprietary packages like Matlab/Maple/Mathematica.

Re: How the R-project is taking over statistical analysis software

#47

Post by muuh-gnu » Sat, Oct 22, 2011, 7:42 AM UTC

migiale wrote: post #25

hvs wrote: post #18

Earlier quoted context omitted.

What about Octave? Other than my use in the Stanford Machine Learning class, I've never really used either, so I don't have any basis for comparison.

Octave is Matlab clone, in fact Octave developers openly say that except for some special cases, any difference between Octave and Matlab is a bug. The biggest difference between Matlab and Octave is JIT compiler in Matlab, which does incredibly good job at vectorizing simple (or sometimes even not-so simple) loops. I think it's fair to say that Octave performance is very close to a Matlab in a pre-JIT time. There's…

> in fact Octave developers openly say that

This is not true. They strive for Matlab language compatibility, but none of them refers to Octave as a "Matlab clone", nor are they working on cloning Matlab, nor was the project started to become a matlab clone. It is like calling Linux a "Unix clone".

Re: How the R-project is taking over statistical analysis software

#48

Post by xtracto » Sat, Oct 22, 2011, 9:21 AM UTC

migiale wrote: post #16

Unfortunately, it's almost impossible to work with a very large datasets in R, because of the speed limitations. Many researchers I know use Matlab because of this.

I use R every day for my research (doing social simulations sometimes based on sample surveys). An additional R limitation is the memory limit. R cannot use virtual memory and the maximum amount of data is limited.

There are two ways to deal with that, one is to load datasets through SQL database (using a SQL library) which IMHO is a "dirty hack". The other (what I usually do) is to load the huge datasets in STATA (or any other stats package) and filter the data to get a set that is small enough to work with R.

Other than that, the available libraries in R are crazy good. for example stuff like Approximate Bayesian Computation or survey analysis (considering weight factors) is straightforward with available libraries.

Re: How the R-project is taking over statistical analysis software

#49

Post by hboon » Sat, Oct 22, 2011, 9:28 AM UTC

cageface wrote: post #29

6ren wrote: post #27

I think open-source eventually replaces commercial products, in the same way that proprietary products become commoditized. The response for commercial products is also the same: continual differentiation, adding new features, benefits, support, documentation etc. Exceptions are also the same: natural monopolies (e.g. strong network effects). Open-source is great at hill-climbing, where there are clear directions for…

Counter-examples abound. Can you name even one open source app that has displaced a mature, user-facing desktop app with a non-trivial UI, other than a web browser? Open source only seems to win in domains in which it makes sense for companies to share work in order to compete at a higher tier of functionality.

Does OS X apps like iTerm and CyberDuck count?

But I agree with you and think that in the long run, open source is a big winner in infrastructure software, and generally just that.

Re: How the R-project is taking over statistical analysis software

#50

Post by Create » Sat, Oct 22, 2011, 12:20 PM UTC

muuh-gnu wrote: post #46

earl wrote: post #33

Earlier quoted context omitted.

I don't think it's obvious that open source displaces commercial for scientific computing. For every example like R which has in many places displaced S-Plus, there are counterexamples like matlab, for which the open source clone Octave is a bad joke, at least the last time I tried using it: missing functions, slowness, extreme difficulty installing; or Mathematica, or eviews, or gauss, or Maple. One other potential…

> extreme difficulty installing Today's Octave installs do not need more than click-click-ok-done or apt-get install octave. > for which the open source clone Octave is a bad joke I think you're not giving Octave enough credit, considering that they have only a few part-time developers, and that nobody does sponsor them, they have accomplished a respectable amount of functionality over the last 20 years, it is exterm…

This is not a new issue. Institutionalised education is most of the time producing "knowledge workers", as MS put it a while back. But without the knowledge, of course.

http://www.cs.utexas.edu/~EWD/transcriptions/EWD12xx/EWD1283...