R Passes SAS in Scholarly Use • Hacker News Board

Re: R Passes SAS in Scholarly Use

#81

Post by cwyers » Thu, Jun 09, 2016, 2:51 PM UTC

jnbiche wrote: post #67

blahi wrote: post #52

Earlier quoted context omitted.

How pandas solves statistical problems or the thousand utility functions that are missing? Please, explain.

Have you even looked at pandas? I get the impression you haven't. Pandas has most of the statistical utility functions that R does, and for those few that it lacks, Python/Pandas also has available an easy-to-use FFI interface to R via rpy2. Take a look: http://pandas.pydata.org/pandas-docs/stable/ EDIT: Specified name of interface (rpy2).

> Pandas has most of the statistical utility functions that R does

Pandas is not shipping with most of the stuff you can get on CRAN.

Re: R Passes SAS in Scholarly Use

#82

Post by dekhn » Thu, Jun 09, 2016, 2:52 PM UTC

hadley wrote: post #76

dekhn wrote: post #72

EDIT: I am corrected in regards to the SAS routines statement; see the reply. A few comments. I worked in pharma and the FDA specifically requires a number of SAS routines- specific function calls- to be used when doing drug studies/clinical trials. R can't replace SAS in those cases without massive effort because the FDA is slow and conservative and people like to have validated results. I think the writing was on t…

That is not true. The FDA uses R internally, and there is no requirement that you must use any specific software tool. See https://www.r-project.org/doc/R-FDA.pdf for more details

OK, so my employers were wrong! http://blog.revolutionanalytics.com/2012/06/fda-r-ok.html

"""Despite some mistaken conceptions in the pharmaceutical industry, SAS is not required to be used for clinical trials. This origin of this fallacy is probably related to the fact that data must be submitted in the XPT "transport format" (which was originally created by SAS). This data format is now an open standard: XPT files can be read into R with the standard read.xport function, and exported from R with the write.xport function in the SASxport package. (And if you have legacy data in other SAS formats, there's a handy SAS macro to export XPT files.)"""

Thanks for clearning that up.

Re: R Passes SAS in Scholarly Use

#83

Post by pedrosorio » Thu, Jun 09, 2016, 2:52 PM UTC

return0 wrote: post #44

aabajian wrote: post #15

I think Python is the biggest hidden gem in statistics. It's had a tremendous impact on machine learning and algorithm development, yet traditional statisticians still rely on SAS/R/Stata/MATLAB. All of these languages have libraries that produce the same results, the difficulty is mangling the data into the correct input format. Python's list comprehensions are much, much easier to use than MATLAB matrices, R's data…

Whats the status of matpootlib nowadays? Graphs are a major major attractir/detractor

https://web.stanford.edu/~mwaskom/software/seaborn/

Re: R Passes SAS in Scholarly Use

#84

Post by micro_cam » Thu, Jun 09, 2016, 2:53 PM UTC

aabajian wrote: post #15

I think Python is the biggest hidden gem in statistics. It's had a tremendous impact on machine learning and algorithm development, yet traditional statisticians still rely on SAS/R/Stata/MATLAB. All of these languages have libraries that produce the same results, the difficulty is mangling the data into the correct input format. Python's list comprehensions are much, much easier to use than MATLAB matrices, R's data…

I'm primarily a python user and agree with most of your post but I do find myself going back to R for many of the more esoteric statistical methods. Ie if i want a specific sort of penalized regression it may not yet be implemented in python.

Re: R Passes SAS in Scholarly Use

#85

Post by pedrosorio » Thu, Jun 09, 2016, 2:53 PM UTC

ldp01 wrote: post #66

hadley wrote: post #60

Earlier quoted context omitted.

I'd recommend reading http://r4ds.had.co.nz for an R workflow that eliminates a lot of those pain points. (Except for infix string concatenation - I've never really understood why people prefer that to paste(). Maybe if you're not thinking in vectors?)

A reply from the man himself! Thanks for the link. I'll have a go. I do like the look of the dplyr library a lot. Combining functions like select and group_by with the pipe operators creates code that is reminiscent of SQL- very nice for readability.

You're going to love Spark if you haven't tried it yet.

Re: R Passes SAS in Scholarly Use

#86

Post by micro_cam » Thu, Jun 09, 2016, 2:59 PM UTC

minimaxir wrote: post #25

jacobolus wrote: post #21

Earlier quoted context omitted.

The even bigger advantage of python is everything else : web scraping, interfacing with weird APIs, hitting the database, consuming and emitting obscure formats, parsing text, calling operating system services, and a million other things. That and much much better abstractions for building larger systems out of reusable components and much better tooling for serious software engineering. R or Matlab can be fine at ex…

If you haven't looked at the R ecosystem in awhile, there is a package for each of those use cases. (scraping/API: rvest; database: dplyr; parsing text: stringr, etc). Yes, Hadley Wickham is primarily responsible for the popularity of R.

Wickham has done a ton but but R had great popularity even before those projects, especially with projects like bioconductor.

Re: R Passes SAS in Scholarly Use

#87

Post by ececconi » Thu, Jun 09, 2016, 3:10 PM UTC

Malarkey73 wrote: post #68

I'm not totally sure whether this analysis captures the true extent which R vs SAS vs SPSS is used. If I use R for a plot, or a simple bit of regression, or anova, or even cross-validation. I don't reference it in a paper. I only cite it if there is a package designed for a particular type of data (e.g. a Bioconductor package) or something a bit more esoteric (e.g. apcluster). About 95% of the work is data munging an…

This is especially true in undergrad papers.

Re: R Passes SAS in Scholarly Use

#88

Post by cwyers » Thu, Jun 09, 2016, 3:25 PM UTC

jhbadger wrote: post #55

pollitos wrote: post #41

R is really LISP with syntactic sugar and bindings to well respected high-performance FORTRAN matrix and math optimization codes. http://librestats.com/2011/08/27/how-much-of-r-is-written-in... It's great for bleeding edge scientific research. The results of many languages don't always match for advanced algorithms, but the open source nature of R, makes it easier to identify the problem areas. The R-core interpreter…

People say that, but I'd prefer the actual LISP syntax then (being a fan of xlispstat back in the day). I'm surprised nobody has created a "Lisp-flavored R" analogous to Erlang's LFE or Python's Hy.

I'd prefer something more like TypeScript for R, where you can gradually move over but you get better tooling. I'd also ask for a new standard library but I think Hadley is basically doing that.

Re: R Passes SAS in Scholarly Use

#89

Post by jsprogrammer » Thu, Jun 09, 2016, 3:29 PM UTC

Malarkey73 wrote: post #68

I'm not totally sure whether this analysis captures the true extent which R vs SAS vs SPSS is used. If I use R for a plot, or a simple bit of regression, or anova, or even cross-validation. I don't reference it in a paper. I only cite it if there is a package designed for a particular type of data (e.g. a Bioconductor package) or something a bit more esoteric (e.g. apcluster). About 95% of the work is data munging an…

Shouldn't your full source be available; which would explicitly record your dependencies?

Re: R Passes SAS in Scholarly Use

#90

Post by jsprogrammer » Thu, Jun 09, 2016, 3:32 PM UTC

aabajian wrote: post #15

I think Python is the biggest hidden gem in statistics. It's had a tremendous impact on machine learning and algorithm development, yet traditional statisticians still rely on SAS/R/Stata/MATLAB. All of these languages have libraries that produce the same results, the difficulty is mangling the data into the correct input format. Python's list comprehensions are much, much easier to use than MATLAB matrices, R's data…

I have heard of people that could not get the same results in Python as they get in R.

Have you looked at R's source? Some common functions contain dozens of unsourced magic numbers (and the comments indicate they have been modified over time).