Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The future of R - pessimistic thoughts by R founder Ross Ihaka

r-bloggers.com

Post reply on HN
Print view View original

41–50 of 78 posts

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#41

Post by fhars »

_delirium wrote: post #19
nanairo wrote: post #6

To be honest I've been using R a bit lately for my work and while I like it I don't find it at all innovative. That's not a criticism of R: the libraries it has are amazing, as well as the mindshare among people who care of statistics. But I wonder why R actually needs to exist as its own language. It seems it could be recast in Ruby for example or one of the latest functional languages. So I am kind of pleased my th…

But I wonder why R actually needs to exist as its own language. It seems it could be recast in Ruby for example or one of the latest functional languages. I agree if we're talking about defining a new R (like the blog post discusses), but the existing R makes sense to me to exist as its own language. It wasn't really invented from scratch gratuitously, but began as an open-source reimplementation of the Bell Labs "S"…

R and S already have completely different scoping rules (S is dynamically scoped, while R has static scope, although quite broken according to the article). So it might be totally possible to develop a third language with mostly the same syntax (so snippets from papers that don't explore the darker corners of the existing semantics can be reused or easily adapted) and this time consistent scoping rules. If it would be worthwhile, I don't know.

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#43

Post by cdavid »

studer wrote: post #36
earl wrote: post #26

Earlier quoted context omitted.

Yes, but (to my shock), their speed makes even R look fast. Which I didn't think as possible. My problem was taking a matrix market formatted matrix, loading it, turning it into a sparse column vector representation, then computing norms of the columns. It was running for roughly 8 hours on 12MM columns. R of all things was running faster. I reimplemented in java and it takes < 1 second.

Given that numpy/scipy is basically a collection of C/C++/Fortran primitives, chances are that you managed to write a program that spent very little time actually computing things, and a lot of time doing something else. Not sure what that could be, though; even low-level algorithms usually run 10-100x slower than C speed if naively coded in plain Python, so a 30000x slowdown using a specialized library sounds rather…

To be fair, the sparse matrix package is still very rough. I am highly skeptical of the 1sec in java vs 8 hours in scipy, though.

In general, numpy/scipy is quite faster than R: it does not have the pass by value semantics for once. I am also skeptical about writing a "new" R: the main value of R is in the R packages. Any new language would threw that away.

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#44

Post by messel »

_delirium wrote: post #19
nanairo wrote: post #6

To be honest I've been using R a bit lately for my work and while I like it I don't find it at all innovative. That's not a criticism of R: the libraries it has are amazing, as well as the mindshare among people who care of statistics. But I wonder why R actually needs to exist as its own language. It seems it could be recast in Ruby for example or one of the latest functional languages. So I am kind of pleased my th…

But I wonder why R actually needs to exist as its own language. It seems it could be recast in Ruby for example or one of the latest functional languages. I agree if we're talking about defining a new R (like the blog post discusses), but the existing R makes sense to me to exist as its own language. It wasn't really invented from scratch gratuitously, but began as an open-source reimplementation of the Bell Labs "S"…

"One in-between possibility could be to retain the standard syntax/semantics but target an existing VM with a bigger development community"

I think that's a great move for a number of languages as they lose popularity over time. The Scheme on lisp machines was outpaced by version on compiled machines, and is the version we use today.

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#46

Post by earl »

studer wrote: post #36
earl wrote: post #26

Earlier quoted context omitted.

Yes, but (to my shock), their speed makes even R look fast. Which I didn't think as possible. My problem was taking a matrix market formatted matrix, loading it, turning it into a sparse column vector representation, then computing norms of the columns. It was running for roughly 8 hours on 12MM columns. R of all things was running faster. I reimplemented in java and it takes < 1 second.

Given that numpy/scipy is basically a collection of C/C++/Fortran primitives, chances are that you managed to write a program that spent very little time actually computing things, and a lot of time doing something else. Not sure what that could be, though; even low-level algorithms usually run 10-100x slower than C speed if naively coded in plain Python, so a 30000x slowdown using a specialized library sounds rather…

Here's my code -- I would have contacted you directly, but I can't find your email. It's as simple as this:

http://gist.github.com/578226#file_gistfile1.py (sorry, that's an editable link, so please be nice)

Anyway -- the machine is not swapping -- python grabs ~20GB of ram, there is another ~100GB available. It pegs one of the cores. Nothing else was running during this test so there was no competition for the fsb. This is on a recent-gen 16 core xeon server. The bit with the pipe and the popen is just me writing the header for the matrix market file in a separate file because it's simpler to dump it from hadoop that way. My time measurements above did not include file reading time -- just the experience of running the norms code (which is just a dot product of each column with itself, essentially). The matrix file is 2GB uncompressed on disk; dimensions are [5e5, 13e7] with 1.2e8 nz.

I rewrote as a sparse, column major matrix in java, running on the same machine, with a sparse dot product implementation. I was off a little before -- the time to stripe the entire dataset is ~2.7 seconds, averaged over 1000 tries. I was confused because I'm spawning 5 threads which gets the time to compute a dot product of one column against every other column down to an average of 1 pass through the data per second. If you contact me, I'm happy to share the code, but it's rather more lengthy.

One more edit -- the java code is not particularly optimized and doesn't batch columns. It just spawns 5 threads that bang on my giant array. Where the data is stored as a column major array of sparse vectors, and each sparse vector is an array of integer indicies and an array of integer values.

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#47

Post by earl »

cdavid wrote: post #43
studer wrote: post #36

Earlier quoted context omitted.

Given that numpy/scipy is basically a collection of C/C++/Fortran primitives, chances are that you managed to write a program that spent very little time actually computing things, and a lot of time doing something else. Not sure what that could be, though; even low-level algorithms usually run 10-100x slower than C speed if naively coded in plain Python, so a 30000x slowdown using a specialized library sounds rather…

To be fair, the sparse matrix package is still very rough. I am highly skeptical of the 1sec in java vs 8 hours in scipy, though. In general, numpy/scipy is quite faster than R: it does not have the pass by value semantics for once. I am also skeptical about writing a "new" R: the main value of R is in the R packages. Any new language would threw that away.

cdavid -- see http://news.ycombinator.com/item?id=1688694

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#48

Post by earl »

nanairo wrote: post #6

To be honest I've been using R a bit lately for my work and while I like it I don't find it at all innovative. That's not a criticism of R: the libraries it has are amazing, as well as the mindshare among people who care of statistics. But I wonder why R actually needs to exist as its own language. It seems it could be recast in Ruby for example or one of the latest functional languages. So I am kind of pleased my th…

R needs it's own language because one of the most important factors in it's widespread use is the language itself. First, it is very similar to an older language that is one of the first widely used statistical packages. Second, the syntax is very easy to use at a simple level. For example, to read a csv file and compute a linear regression with betas, p-values, the works, all you have to do is:

  # read a csv file with headers into ram
  data
That's it. So you can see why stats people love it -- it's easy to pick up and use to get productive work done. It's easy to show students. For what it's meant to do originally -- desktop statistics -- the language works quite well. It lacks as a programming language in many regards, don't get me wrong, but simplicity and ease of use, particularly in the beginning, are critical.

Also, the data frame is the single best data structure I've ever used for manipulating tabular data. Finally, the excellent repl makes working in R and exploratory analysis absolutely awesome.

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#49

Post by equark »

phren0logy wrote: post #45
equark wrote: post #42

Earlier quoted context omitted.

Building a next generation of statistical computing.

Would you be willing to be more specific? That's not very helpful.

I can't be yet, unfortunately. But there are definitely several different groups of very talented people working on this problem (think MIT and Harvard PhDs). I suspect some very interesting work will come out in the next year or two. Whether they are open source remains another question.

Re: The future of R - pessimistic thoughts by R founder Ross Ihaka

#50

Post by hadley »

haberman wrote: post #32

To me the problem with R isn't performance problems, which I've never run into myself, but rather the complicated and confusing semantics of its data types. R's aggregate data types are: vector, matrix, array, dataframe, and list. The semantics of these types and the relationships between them are extremely confusing. I wish I had gathered examples of this so I could be more specific, but I have basically come to the…

The semantics are fairly straightforward once you figure out the two main differences.

Vectors, matrices and arrays are atomic/homogeneous objects, and only differ in their dimensionality. Vectors are 1d, matrices 2d, and arrays are 3d or higher. Calling a 2d homogenous structure is a matrix is just a convention: a matrix is identical to a 2d array in every important way.

Lists and data frames are heterogenous/recursive. Lists are 1d, and data frames are (essentially) 2d (each row is homogenous, but each column can be a different type).

Post reply on HN