Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The future of R - pessimistic thoughts by R founder Ross Ihaka

r-bloggers.com

Post reply on HN
Print view View original

61–70 of 78 posts

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#61

Post by cdavid »

zheaton wrote: post #59
earl wrote: post #46

Earlier quoted context omitted.

Here's my code -- I would have contacted you directly, but I can't find your email. It's as simple as this: http://gist.github.com/578226#file_gistfile1.py (sorry, that's an editable link, so please be nice) Anyway -- the machine is not swapping -- python grabs ~20GB of ram, there is another ~100GB available. It pegs one of the cores. Nothing else was running during this test so there was no competition for the fsb.…

Is there any chance that SciPy was compiled without the ATLAS/BLAS/LAPACK libraries? (You can verify this using with the "numpy.show_config()" and "scipy.show_config()" functions in an interactive python session.) In that case, all the numerical math would be done in straight Python rather than optimized C/FORTRAN.

ATLAS/BLAS/LAPACK is mostly useless for basic sparse operations, at least in scipy. It becomes useful for more advanced operations like SVD, etc... Unless you can use dot (scalar product), which is very fast in numpy if you use atlas (e.g. time the difference between np.sum(x * x) vs np.dot(x, x) - sum does not use atlas, dot does).

For sparse SVD as I implemented in scipy, even fast dot does not seem to make that much of a difference (would be interesting to do real benchmarks, though).

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#62

Post by cdavid »

earl wrote: post #47
cdavid wrote: post #43

Earlier quoted context omitted.

To be fair, the sparse matrix package is still very rough. I am highly skeptical of the 1sec in java vs 8 hours in scipy, though. In general, numpy/scipy is quite faster than R: it does not have the pass by value semantics for once. I am also skeptical about writing a "new" R: the main value of R is in the R packages. Any new language would threw that away.

cdavid -- see http://news.ycombinator.com/item?id=1688694

thanks, I will take a look at it at home (where I have enough memory to deal with that kind of matrix). As I said, the scipys.sparse is pretty rough and too low-level, but 3 sec vs 8 hours is way too long to be normal.

(I forked your gist so you know how to contact me)

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#63

Post by regularfry »

piggybox wrote: post #51
Quiark wrote: post #27

Earlier quoted context omitted.

For JVM, there is Incanter which is a statistics library written in Clojure. It is backed by Parallel Colt for the heavy number lifting. Note that I'm not trying to say that Clojure would be a scientist-friendly language :)

A lot of people blame Lisp for too many brackets, but I generally feel they are not the issue after reading Lisp for a while. What may make it scientist-unfriendly is to write math formulas in infix grammar.

Incanter has the $= macro for writing infix expressions, so I don't think that need be an issue.

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#64

Post by equark »

cdavid wrote: post #60
equark wrote: post #57

Earlier quoted context omitted.

How important is the distinction between NaN versus NA? I agree this is a subtle issue.

It is important - you cannot know whether NaN is coming from a computation or is really a missing value otherwise. In Numpy, we have the MaskedArray implementation to do this.

What does a NA value become when you extract it to a float? i.e. What is the behavior of X[0]?

It is somewhat confusing that python base types and numpy differ in behavior, for instance when dealing with inf or divide by zero exceptions. I think this gets to hadley's point that it will be hard to bolt on R to an existing language.

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#65

Post by nanairo »

earl wrote: post #48
nanairo wrote: post #6

To be honest I've been using R a bit lately for my work and while I like it I don't find it at all innovative. That's not a criticism of R: the libraries it has are amazing, as well as the mindshare among people who care of statistics. But I wonder why R actually needs to exist as its own language. It seems it could be recast in Ruby for example or one of the latest functional languages. So I am kind of pleased my th…

R needs it's own language because one of the most important factors in it's widespread use is the language itself. First, it is very similar to an older language that is one of the first widely used statistical packages. Second, the syntax is very easy to use at a simple level. For example, to read a csv file and compute a linear regression with betas, p-values, the works, all you have to do is: # read a csv file wit…

Could you tell me a bit more about why you like data.frame? Is it not just a 2D matrix?

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#66

Post by Quiark »

nanairo wrote: post #34
Quiark wrote: post #27

Earlier quoted context omitted.

For JVM, there is Incanter which is a statistics library written in Clojure. It is backed by Parallel Colt for the heavy number lifting. Note that I'm not trying to say that Clojure would be a scientist-friendly language :)

How does Incanter work on HPC? R is pretty awful from that point of view, and if there ever will be room for a specialised statistical language it's got to be able to do massive number crunching. I've heard bad things of JVM for tightly coupled jobs on HPC (though I know there's been some improvement: e.g. a lot of work done by EPCC in Edinburgh). Does Clojure manages to offer a good parallel implementation on top of…

According to the website, Parallel Colt supports multicore machines. There's no mention of MPI though. As far as I know, Incanter only wraps it, so it does not influence the performance that much..

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#67

Post by Quiark »

earl wrote: post #48
nanairo wrote: post #6

To be honest I've been using R a bit lately for my work and while I like it I don't find it at all innovative. That's not a criticism of R: the libraries it has are amazing, as well as the mindshare among people who care of statistics. But I wonder why R actually needs to exist as its own language. It seems it could be recast in Ruby for example or one of the latest functional languages. So I am kind of pleased my th…

R needs it's own language because one of the most important factors in it's widespread use is the language itself. First, it is very similar to an older language that is one of the first widely used statistical packages. Second, the syntax is very easy to use at a simple level. For example, to read a csv file and compute a linear regression with betas, p-values, the works, all you have to do is: # read a csv file wit…

I'm pretty sure you could do a lot of this in Python as well.

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#68

Post by cdavid »

equark wrote: post #64
cdavid wrote: post #60

Earlier quoted context omitted.

It is important - you cannot know whether NaN is coming from a computation or is really a missing value otherwise. In Numpy, we have the MaskedArray implementation to do this.

What does a NA value become when you extract it to a float? i.e. What is the behavior of X[0]? It is somewhat confusing that python base types and numpy differ in behavior, for instance when dealing with inf or divide by zero exceptions. I think this gets to hadley's point that it will be hard to bolt on R to an existing language.

If X[0] is masked, it will return the value mask, of type MaskedConstant.

As for python vs numpy differences: yes, those can be confusing, and that's inherent to the fact that we use a "real" language with a library on top of it instead of the language designed around the domain. If you want to do numerical computation, you do want the behavior of numpy in most cases, I think. There is the issue of "easiness" vs what scientists need. You regularly have people who complain about various float issues, and people with little numerical computation knowledge advising to use decimal, etc... unaware of the issues. Also, python will want to "hide" the complexities, whereas numpy is much less forgiving.

As for the special case of divide by 0 or inf, note that you can get a behavior quite similar to python float. You can control how FPU exceptions are raised with numpy.seterr:

import numpy as np a = np.random.randn(4) a / 0 # puts a warnings, gives an array of +/- inf np.seterr(divide="raise") a / 0 # raise divide by 0 exception

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#69

Post by earl »

earl wrote: post #46
studer wrote: post #36

Earlier quoted context omitted.

Given that numpy/scipy is basically a collection of C/C++/Fortran primitives, chances are that you managed to write a program that spent very little time actually computing things, and a lot of time doing something else. Not sure what that could be, though; even low-level algorithms usually run 10-100x slower than C speed if naively coded in plain Python, so a 30000x slowdown using a specialized library sounds rather…

Here's my code -- I would have contacted you directly, but I can't find your email. It's as simple as this: http://gist.github.com/578226#file_gistfile1.py (sorry, that's an editable link, so please be nice) Anyway -- the machine is not swapping -- python grabs ~20GB of ram, there is another ~100GB available. It pegs one of the cores. Nothing else was running during this test so there was no competition for the fsb.…

Typo in above -- it should be 1.3e7 columns. Sorry.

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#70

Post by pajarito »

a translator from R to Maxima would be another idea. Maxima is Lisp based, use an algol like language, is used in education and is free. But developer are scare so more hands are needed. Maxima can use BLAS and other libraries for numeric computation, and it support many implementations of Lisp (sbcl, gcl, cll, ecl ...), some of their member are working in a java based lisp (Arms beast?). With so many tools available it is a something to consider.
Post reply on HN