Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

R Passes SAS in Scholarly Use

r4stats.com

Post reply on HN
Print view View original

111–120 of 126 posts

Re: R Passes SAS in Scholarly Use

#111

Post by baldfat »

compactmani wrote: post #40
aabajian wrote: post #15

I think Python is the biggest hidden gem in statistics. It's had a tremendous impact on machine learning and algorithm development, yet traditional statisticians still rely on SAS/R/Stata/MATLAB. All of these languages have libraries that produce the same results, the difficulty is mangling the data into the correct input format. Python's list comprehensions are much, much easier to use than MATLAB matrices, R's data…

Can you give some concrete examples of the benefits of wrangling in python over R and a general sense of how much time this would save a user?

I really would be interested because I started in Python and R and now use R exclusively due to manipulation. R (dplyr) really is the best manipulation system have ever used. Perhaps people try to do loops in R and other non-R like ways and give up. R is fundamentally a Functional Language and people try to band a square into a round hole and give up?

Re: R Passes SAS in Scholarly Use

#112

Post by hadley »

noelsusman wrote: post #100
hooloovoo_zoo wrote: post #92

Earlier quoted context omitted.

Why don't you cite the packages you use?

Nobody cites every package they use, it's not feasible. I use a lot of packages, and some journals have a limit on the number of citations you can have. I only cite packages when it provides specialized statistical functionality. For example, I do a lot of work with data from complex surveys, and I always cite Lumley's survey package because without it I wouldn't be able to do the work. On the flip side, I use Hadley…

To me, citing readr feels like citing the company that made your pipettes. It's just useful infrastructure.

Re: R Passes SAS in Scholarly Use

#113

Post by hadley »

pollitos wrote: post #95
hadley wrote: post #58

Earlier quoted context omitted.

I would love to know what you mean by data wrangling because I think R has a lot of good tools for it.

For example, reshaping JSON to the format an intricate R function expects. Appreciate the great work with (d)plyr and similar packages, but it's still work and overhead. Combined with some inefficiencies/quirks in base r functions (does ifelse() still evaluate twice?) it's easier to go with a widely used and respected package in a general purpose language; Nokogiri for example. For data engineering, consider there is…

JSON is often a pain because it's so hierarchical and un-dataframe like. I have a few notes on working with it here: http://r4ds.had.co.nz/hierarchy.html.

`ifelse()` is a nightmare of a function but I don't think double-evaluation is ever a problem.

There are two maintained web-clients: curl (low-level) and httr (high-level). And I think rvest does everything that nokogiri does.

Re: R Passes SAS in Scholarly Use

#114

Post by Houshalter »

hadley wrote: post #77
Houshalter wrote: post #75

Earlier quoted context omitted.

I suspect that the utility of more packages increases only logarithmically. Having 10x more packages doesn't mean it's 10 times more useful. If your obscure need isn't in the first 8,000 packages, it probably won't be in the next 80,000. That's just how power laws work. And any common task you can think of will probably be in the top 8,000.

I totally buy that argument, but there's probably not a huge overlap in the set of "common tasks" for R and python.

Well then all the more reason to use R, if those are the set of tasks you want to do.

Re: R Passes SAS in Scholarly Use

#115

Post by dredmorbius »

thebelal wrote: post #8
dredmorbius wrote: post #3

1. This is a good place for use of log or semi-log plots. 2. How do the authors unambiguously search for 'R'? Monocharacter language names are difficult search keys. (C, B, S, R)

1. There is a log plot at the end of the article 2. The author posted the exact search terms used for all languages in an earlier post [0] [0]: http://r4stats.com/articles/how-to-search-for-data-science-a...

Thanks, on both points.

Re: R Passes SAS in Scholarly Use

#116

Post by yoplait_ »

jhbadger wrote: post #98
LionessLover wrote: post #49

Earlier quoted context omitted.

To 2: When I look for just "R" even in an anonymous window (so it should not use my history) I get as the first suggestion a link to https://www.r-project.org/ - the home of R. What else is there for that letter - that is equally popular? "R" is "hip" and trending. Microsoft not too long ago started a big push into the R space and now regularly generates headlines around the system, accelerating the trend even more.

Even weirder, a Google search for "xlispstat" seems to bring up more R hits that don't even mention xlispstat than actual xlispstat ones. Some weird algorithm is associating R and xlispstat as relating to statistics and because R is much more popular these days, prioritizing R over xlispstat.

it's not the search engine's fault, it's the contents. R is associated to xlispstat as the credible alternative. E.g. Jan de Leeuw JStat's paper. Also the author of xlispstat is one main contributor to R.

Re: R Passes SAS in Scholarly Use

#117

Post by jiiam »

noelsusman wrote: post #103
jiiam wrote: post #73

Good. * Rant mode: On Maybe in 30 years they will also learn a true programming language and stop producing undocumented, unusable, unportable, underdeveloped libraries for research level tools and technologies. Outside the world of Neural Network it is a complete disaster, and the NN landscape is at an acceptable level only because of big companies, surely not thanks to the researchers. And the reason, of course, is…

I mean, I won't argue against having better code and documentation, but it's not really our job to ship a real product. Shipping well documented, easily usable, ultra portable, well developed libraries takes a fuckload of time, resources, and expertise that we don't have. Our primary job is to ship ideas. It would be awesome if every project I did ended up with a nice, polished piece of software, but that's not what…

Fair enough. But, speaking as a researcher, I often find myself reading through hundreds of lines of code and rebuilding routines from scratch in order to reproduce and expand on what others did in their works.

However, I was very harsh, and of course I wouldn't find viable to expect production ready code, but something moderately portable could come handy. Of course, as you said, a researcher doesn't have the time to build a well developed library. As a solution, my University is considering the idea of hiring a dedicated developer whose job would be to maintain libraries. I really hope this to happen.

Re: R Passes SAS in Scholarly Use

#118

Post by pollitos »

hadley wrote: post #113
pollitos wrote: post #95

Earlier quoted context omitted.

For example, reshaping JSON to the format an intricate R function expects. Appreciate the great work with (d)plyr and similar packages, but it's still work and overhead. Combined with some inefficiencies/quirks in base r functions (does ifelse() still evaluate twice?) it's easier to go with a widely used and respected package in a general purpose language; Nokogiri for example. For data engineering, consider there is…

JSON is often a pain because it's so hierarchical and un-dataframe like. I have a few notes on working with it here: http://r4ds.had.co.nz/hierarchy.html . `ifelse()` is a nightmare of a function but I don't think double-evaluation is ever a problem. There are two maintained web-clients: curl (low-level) and httr (high-level). And I think rvest does everything that nokogiri does.

Thanks for the link on JSON and your packages, great work as always. I should clarify when I said we-client, I meant websockets client to consume feed. The last time I tried, the only R package (r-websockets) just crashed my Linux box and not maintained for several years. httr doesn't do websockets, as I understand. Seems likely a fundamental way to engineer/wrangle data into R.

Re: R Passes SAS in Scholarly Use

#119

Post by hadley »

pollitos wrote: post #118
hadley wrote: post #113

Earlier quoted context omitted.

JSON is often a pain because it's so hierarchical and un-dataframe like. I have a few notes on working with it here: http://r4ds.had.co.nz/hierarchy.html . `ifelse()` is a nightmare of a function but I don't think double-evaluation is ever a problem. There are two maintained web-clients: curl (low-level) and httr (high-level). And I think rvest does everything that nokogiri does.

Thanks for the link on JSON and your packages, great work as always. I should clarify when I said we-client, I meant websockets client to consume feed. The last time I tried, the only R package (r-websockets) just crashed my Linux box and not maintained for several years. httr doesn't do websockets, as I understand. Seems likely a fundamental way to engineer/wrangle data into R.

You can do it with httpuv, but it might a bit clunky. I think better websockets support, and better async generally, is on the roadmap for the next year.

Re: R Passes SAS in Scholarly Use

#120

Post by Myrmornis »

jnbiche wrote: post #69
Myrmornis wrote: post #39

Earlier quoted context omitted.

I think you got downvoted because people didn't know what you meant. You're right though. Although R is laughably inferior to python as a programming language, it is vastly more work to try to do statistical data analysis in python than in R. I recommend using both languages and using csv or whatever format to exchange data sets.

> using csv or whatever format to exchange data sets Have you used Pandas in the past year or two? I'm curious why you would exchange data sets using csv or another format between Python and R when you could easily call the R function of interest from within Pandas (using rpy2) and not even worry about data interchange. It's definitely not vastly more work to do statistical analysis using Python Pandas than in R anym…

> It's definitely not vastly more work to do statistical analysis using Python Pandas than in R anymore, perhaps it was several years ago.

Haha, what? What statistics can you do in pandas? You can do some statistics in python by cobbling together stuff from scipy and statsmodels (maybe I'm out of date, is there more?). I see a few modules for regression and stuff in pandas but they are marked as deprecated. I think perhaps you and I mean different things by "statistics". R provides a vast ecosystem covering, for example

- Gold standard implementations of simulation, PDFs, quantiles of any probability distribution you can mention (in python you can find some of this in scipy; not pandas. But scipy is a real mess compared to R and not as comprehensive.)

- Gold standard implementations of any classical hypothesis test you can mention

- Gold standard implementations of computational methods for fitting generalized linear models, mixed models, frameworks for MCMC samplers, graphical models, HMMs, and a vast amount of other stuff I'm not clever enough to name right now let alone understand.

Really any statistical procedure -- whether "classical" or "modern"/"computational statistics" -- in R you will find it, and furthermore it will be basically the reference implementation / gold standard.

That's not mentioning the plotting tools and the numerical computing and clean linear algebra syntax. But that's it, no more: the people who go further and suggest using R for building a web server or web scraping or something mostly haven't used real programming languages.

You're missing the point. The python ecosystem can't compete with R on the statistics front -- it would be crazy to try. That's certainly not the aim of pandas.

> you definitely don't need to use intermediary csv files anymore to move data back and forth between R and Python.

Perhaps not, but doesn't it please you to have a well-defined interface (a serialization format) between the two languages? I haven't tried Rpy2 for years. I don't like to have two different languages get their tentacles into each other like that if I can avoid it, but I'm sure it's a good project which has its use cases.

EDIT: thanks, I hadn't seen feather. That looks like the thing to use.

Post reply on HN