Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The future of R - pessimistic thoughts by R founder Ross Ihaka

r-bloggers.com

Post reply on HN
Print view View original

71–78 of 78 posts

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#71

Post by pvxc »

earl wrote: post #46
studer wrote: post #36

Earlier quoted context omitted.

Given that numpy/scipy is basically a collection of C/C++/Fortran primitives, chances are that you managed to write a program that spent very little time actually computing things, and a lot of time doing something else. Not sure what that could be, though; even low-level algorithms usually run 10-100x slower than C speed if naively coded in plain Python, so a 30000x slowdown using a specialized library sounds rather…

Here's my code -- I would have contacted you directly, but I can't find your email. It's as simple as this: http://gist.github.com/578226#file_gistfile1.py (sorry, that's an editable link, so please be nice) Anyway -- the machine is not swapping -- python grabs ~20GB of ram, there is another ~100GB available. It pegs one of the cores. Nothing else was running during this test so there was no competition for the fsb.…

Such large differences imply a difference in the algorithm

Indeed, you write

c = mat2.getcol(j)

norms[0, j] = scipy.linalg.norm(c.A)

which means (i) extract a sparse column vector, (ii) convert it to a dense vector, and (iii) compute the norm. Now, this should explain the speed difference. Looking at the nnz, a dense norm can take up to a factor 5e5/(1.2e8/1.3e7) ~ 54000 longer :)

The main issue here is that the linear algebra stuff under `scipy.linalg` doesn't know about sparse matrices, and tends to convert everything to dense first. You'd need to muck around with `m2.data` to go faster.

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#72

Post by respiratoryclub »

equark wrote: post #64
cdavid wrote: post #60

Earlier quoted context omitted.

It is important - you cannot know whether NaN is coming from a computation or is really a missing value otherwise. In Numpy, we have the MaskedArray implementation to do this.

What does a NA value become when you extract it to a float? i.e. What is the behavior of X[0]? It is somewhat confusing that python base types and numpy differ in behavior, for instance when dealing with inf or divide by zero exceptions. I think this gets to hadley's point that it will be hard to bolt on R to an existing language.

I agree. It's pretty important. Missing values can be coded as funny things like minus infinity in some languages. So then, if you want to assign a subset of a vector to a new group (say the number of people who have had a an event e.g. "heart attack") and you used the argument x Of course if you're always careful, if your NA values in other languages are stored as numbers, you can avoid this error. But it's made easy by R's approach to NA.

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#73

Post by earl »

pvxc wrote: post #71
earl wrote: post #46

Earlier quoted context omitted.

Here's my code -- I would have contacted you directly, but I can't find your email. It's as simple as this: http://gist.github.com/578226#file_gistfile1.py (sorry, that's an editable link, so please be nice) Anyway -- the machine is not swapping -- python grabs ~20GB of ram, there is another ~100GB available. It pegs one of the cores. Nothing else was running during this test so there was no competition for the fsb.…

Such large differences imply a difference in the algorithm Indeed, you write c = mat2.getcol(j) norms[0, j] = scipy.linalg.norm(c.A) which means (i) extract a sparse column vector, (ii) convert it to a dense vector, and (iii) compute the norm. Now, this should explain the speed difference. Looking at the nnz, a dense norm can take up to a factor 5e5/(1.2e8/1.3e7) ~ 54000 longer :) The main issue here is that the line…

Thanks a bunch for the help. It would be nice if this were documented somewhere besides reading the source code though :( And none of my googling turned up m2.data.

I'd actually guessed that it might be making columns full, but I'd expected to see a step-ladder up and down memory pattern as fectors were allocated, gc was triggered, vectors were allocated, etc. I didn't observe such a pattern; memory usage was almost constant.

Anyway, thanks again for your help -- I'd offer via email to buy you a beer if you're ever in SF, but no email, so...

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#74

Post by hogu »

python

has, RPy2 so you can still access R's statistics libraries http://scikits.appspot.com/statsmodels, and http://pandas.sourceforge.net/ (dataframes)

plus, scipy has a decent stats library as well (random variables, etc)

still rough around the edges, but a good solution in my opinion

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#75

Post by filobloomz »

equark wrote: post #49
phren0logy wrote: post #45

Earlier quoted context omitted.

Would you be willing to be more specific? That's not very helpful.

I can't be yet, unfortunately. But there are definitely several different groups of very talented people working on this problem (think MIT and Harvard PhDs). I suspect some very interesting work will come out in the next year or two. Whether they are open source remains another question.

Austin Heap? Is that you?

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#76

Post by hadley »

respiratoryclub wrote: post #72
equark wrote: post #64

Earlier quoted context omitted.

What does a NA value become when you extract it to a float? i.e. What is the behavior of X[0]? It is somewhat confusing that python base types and numpy differ in behavior, for instance when dealing with inf or divide by zero exceptions. I think this gets to hadley's point that it will be hard to bolt on R to an existing language.

I agree. It's pretty important. Missing values can be coded as funny things like minus infinity in some languages. So then, if you want to assign a subset of a vector to a new group (say the number of people who have had a an event e.g. "heart attack") and you used the argument x Of course if you're always careful, if your NA values in other languages are stored as numbers, you can avoid this error. But it's made eas…

And this isn't just a hypothetical example - it has happened and has affected the results of important medical studies.

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#77

Post by hadley »

Quiark wrote: post #67
earl wrote: post #48

Earlier quoted context omitted.

R needs it's own language because one of the most important factors in it's widespread use is the language itself. First, it is very similar to an older language that is one of the first widely used statistical packages. Second, the syntax is very easy to use at a simple level. For example, to read a csv file and compute a linear regression with betas, p-values, the works, all you have to do is: # read a csv file wit…

I'm pretty sure you could do a lot of this in Python as well.

It seems that way on the surface, but their are a lot of subtleties to R (lazy evaluation of function arguments, access to both static and dynamic scope) that would make implementation in another language difficult. R has many features that are not that desirable from a programming language stand point (and that do make optimisation difficult), but do make it a very expressive, flexible language for statistical computing.
Post reply on HN