Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's Next for R?

qz.com

Post reply on HN
Print view View original

31–40 of 72 posts

Re: What's Next for R?

#31

Post by jpalomaki »

zosima wrote: post #8
pickdenis wrote: post #3

I know this is a dead horse, but I think R seriously shot itself in the foot with its data structures[1]. I don't really see a solution for this, as fixing it would never be backward compatible. I'll always pick Python over R because the data structures actually make sense to me as a programmer (objects that look like lists, dicts, matrices, etc. or any combination of the above, and they all behave in very predictabl…

True, the default semantics of R's data structures are somewhat arcane (of course as they're based on S [1] from the 70's). And the current support for e.g. 64bit integers leaves something to be desired. But behind the scenes, R is just a lisp with some data structures that are adapted to statistics and data science. All base data structures are by default immutable. And e.g. the vector type is extremely performant a…

This 5 minute video by Wickham was eye opening for me regarding the lispiness or R.

https://youtu.be/nERXS3ssntw

Re: What's Next for R?

#32

Post by closed »

latte wrote: post #18

Cannot comment from my personal impressions, as I have almost zero knowledge of R, compared to several years of using Python for writing apps and working with data. I like R's focus on functional programming, though. However, a couple of years ago, my wife tried to transition from business consulting to a data analytics / data science role. She started with taking an R course. She was put off by R's complexity and th…

We ran into this problem often teaching R when I was in grad school. However, in the years since, the tidyverse has put strong focus on getting users manipulating data even if they don't know how to define a function, etc..!

Here's a useful post, comparing the classic approach you mention to an alternative

http://varianceexplained.org/r/teach-tidyverse/

Re: What's Next for R?

#33

Post by CreRecombinase »

thegginthesky wrote: post #30

When I used R in University (majored in Applied Mathematics and Statistics) I was always awestruck at how every sort of novel modeling technique from GLM, to Beta Regressions, to GARCH, is all easily accessible for free, with proper academic paper and documentation, and with a cohesive standard support. It was really useful to be able to apply most theory I was learning to actual research datasets. This is what I mis…

What didn't you like about the packaging system? Even if you hate R the language, R has among the most user-friendly, cross-plaform packaging systems I'm aware of.

Re: What's Next for R?

#34

Post by Quasimoto3000 »

thegginthesky wrote: post #30

When I used R in University (majored in Applied Mathematics and Statistics) I was always awestruck at how every sort of novel modeling technique from GLM, to Beta Regressions, to GARCH, is all easily accessible for free, with proper academic paper and documentation, and with a cohesive standard support. It was really useful to be able to apply most theory I was learning to actual research datasets. This is what I mis…

R has actually come a long way on the environments front. Check out “renv” from the good people at RStudio.

Link: https://rstudio.github.io/renv/articles/renv.html

Re: What's Next for R?

#35

Post by meztez »

609venezia wrote: post #27
meztez wrote: post #11

I would highly recommend the use of the package data.table over tibble or the basic data.frame if you are doing any type of modeling in R with larger datasets. Yes R has many data structures but knowing how to use data.table will blow your mind in term of efficiency. Matt and other contributors have built something extremely fast and flexible. I get that R is not for everyone but used correctly it is a beast. Now thi…

> I believe Julia to be the future but so far the adoption rate in house has been low. Why do you believe it will be the future, and what do you see as the barriers to roll-out? I ask as someone who is curious about when/whether to start investing in Julia competence

Debugging is something I could not do as easily in Julia vs debugonce and trace in R. Compiling takes time.

But so far we have seen great development. Flux is a truly beautiful ml library. Being a compiled language remove a lot of headache when building production images. The syntax, the full utf support in variable name. Package management is great. Having that abstraction layer between CPU, GPU so you don't have to rewrite code. Dispatch based on signature, type management. I don't see it going away soon. It took me 13 years to make them transition out of SAS, good thing cloud computing come around and someone realised the clusterfuck of having to manage SAS licence in the cloud.

Re: What's Next for R?

#36

Post by ACow_Adonis »

meztez wrote: post #11

I would highly recommend the use of the package data.table over tibble or the basic data.frame if you are doing any type of modeling in R with larger datasets. Yes R has many data structures but knowing how to use data.table will blow your mind in term of efficiency. Matt and other contributors have built something extremely fast and flexible. I get that R is not for everyone but used correctly it is a beast. Now thi…

As someone "fully fluent" in both, for many workflows that can be properly implemented in SAS, you would expect on a technical level the SAS program could be faster. It's a fully compiled language, it's a "simple" compilation model (compared to R), and the interaction between incremental compilation and the macro system allows you to do some really good blurring between run-time and compilation when performance matters. Plus, by abusing the fact you can define both sql and data step views to further minimise disk read/write, database pass through on certain procedures, and allowimg for in-memory operations (like R) with the sasfile command, from a purely technical point of view, an experienced user of both should be able to beat R in SAS.

But... and here's the big but...I almost never actually meet anyone capable of putting all these steps together in SAS these days that actually understands the SAS computation model end to end.

And SAS's strength, a computation model not being limited by memory by default, becomes a performance weakness when everyone reads/writes every step out to disk and programs without understanding all those little intricacies. SAS hasn't helped any of this by trying to move its eco system away from "programmer" to "application users", so now "programmers" can pick up an interpreted language like R with in-memory default vectorised operations and beat SAS.

Course, I'd still recommend places move to python/R these days because of the broader ecosystems, university talent pool, and avoiding the extensive lock in of proprietary software, but I still feel I have to reflexively respond to "R faster than SAS" claims :p

Re: What's Next for R?

#37

Post by meztez »

ACow_Adonis wrote: post #36
meztez wrote: post #11

I would highly recommend the use of the package data.table over tibble or the basic data.frame if you are doing any type of modeling in R with larger datasets. Yes R has many data structures but knowing how to use data.table will blow your mind in term of efficiency. Matt and other contributors have built something extremely fast and flexible. I get that R is not for everyone but used correctly it is a beast. Now thi…

As someone "fully fluent" in both, for many workflows that can be properly implemented in SAS, you would expect on a technical level the SAS program could be faster. It's a fully compiled language, it's a "simple" compilation model (compared to R), and the interaction between incremental compilation and the macro system allows you to do some really good blurring between run-time and compilation when performance matte…

Believe me, I know. The code just becomes unreadable when you put all execution inside the same data step and use hash table to do fast small to big merging. And not to mention debugging that mess when you have a macro layer on top of it. Not having access to function source code, installation process being what it was. I do not miss it.

And yes technically SAS is faster than R but part of the equation is how many people can make SAS code faster than R/python. I had maybe, 1-2 people that could write efficient SAS code.

One version we had was a bunch of macro producing hash merge plus the whole how can I do something without having to get out of the data step. Just horrible. Number of characters in a line of code? You forgot your quote somewhere and now you have to run the magic line.

I hope I'm not too emotional when I say I hope SAS disappears from my industry and we embrace less adversarial licensing.

Re: What's Next for R?

#39

Post by jrumbut »

ACow_Adonis wrote: post #36
meztez wrote: post #11

I would highly recommend the use of the package data.table over tibble or the basic data.frame if you are doing any type of modeling in R with larger datasets. Yes R has many data structures but knowing how to use data.table will blow your mind in term of efficiency. Matt and other contributors have built something extremely fast and flexible. I get that R is not for everyone but used correctly it is a beast. Now thi…

As someone "fully fluent" in both, for many workflows that can be properly implemented in SAS, you would expect on a technical level the SAS program could be faster. It's a fully compiled language, it's a "simple" compilation model (compared to R), and the interaction between incremental compilation and the macro system allows you to do some really good blurring between run-time and compilation when performance matte…

Do you have any resources that help explain these SAS performance measures? A book perhaps?

I have been trying to help with exactly this (and your breadcrumbs help) but it is tricky for me since I am used to open source/*nix environment where you can use much different tools and also information and tutorials are distributed much more widely.

Re: What's Next for R?

#40

Post by xiaodai »

ACow_Adonis wrote: post #36
meztez wrote: post #11

I would highly recommend the use of the package data.table over tibble or the basic data.frame if you are doing any type of modeling in R with larger datasets. Yes R has many data structures but knowing how to use data.table will blow your mind in term of efficiency. Matt and other contributors have built something extremely fast and flexible. I get that R is not for everyone but used correctly it is a beast. Now thi…

As someone "fully fluent" in both, for many workflows that can be properly implemented in SAS, you would expect on a technical level the SAS program could be faster. It's a fully compiled language, it's a "simple" compilation model (compared to R), and the interaction between incremental compilation and the macro system allows you to do some really good blurring between run-time and compilation when performance matte…

Just use https://diskframe.com and you will not limited by memory!!
Post reply on HN