How R Took the World of Statistics by Storm • Hacker News Board

Re: How R Took the World of Statistics by Storm

#31

Post by grayclhn » Sun, Nov 22, 2015, 3:29 PM UTC

PlotlyThrowaway wrote: post #22

B1gred wrote: post #16

Earlier quoted context omitted.

Python has a steeper learning curve and is not as curtailed to simple data analysis. Many use Rstudio (an ide) and use the import data, and other tools to make then skill entry even lower. Also, mathematicians and statisticians think functionally and the general attitude in python is to do object oriented programming while R is strictly functional programming with a little bit of object programming.

I'm a little at odds on this--for production quality analyses, (and only for analyses) R is excellent. However, in my experience, for the data munging required as a preliminary to the analyses, R is worse than bad. It's as if satan himself designed a language. I find that what then happens is this: data scientists/statisticians/[your favorite word here] become reliant on programmers to clean/format the data to do the…

I might be misrembering, but I think that it was assumed for a while that perl would be used for data munging, so there wasn't much effort put into that part of the language. That's being addressed now by packages, and a lot of the uptake in R seems to coincide with work by R developers to make the language less hostile to new users. (Have you used the bundled IDE? Satan's work again.)

BUT perl + R was a really nice combination for a while.

Re: How R Took the World of Statistics by Storm

#32

Post by B1gred » Sun, Nov 22, 2015, 3:33 PM UTC

lottin wrote: post #26

PlotlyThrowaway wrote: post #22

Earlier quoted context omitted.

I'm a little at odds on this--for production quality analyses, (and only for analyses) R is excellent. However, in my experience, for the data munging required as a preliminary to the analyses, R is worse than bad. It's as if satan himself designed a language. I find that what then happens is this: data scientists/statisticians/[your favorite word here] become reliant on programmers to clean/format the data to do the…

What do you mean by data munging? Things like extracting data from XML files for instance?

It can mean anything, and that is why it is hard. Many old school statisticians can not work with anything other than csv, excel spreadsheets or basic sql queries. Munging is the conversion to a nice format that can then be used for analysis.

Re: How R Took the World of Statistics by Storm

#33

Post by jasonpbecker » Sun, Nov 22, 2015, 3:47 PM UTC

B1gred wrote: post #30

Simorgh wrote: post #20

Earlier quoted context omitted.

I agree with your assertion that R is slow, yet quick to develop in. I recently had to loop through 1.3Gb of data (5000 files) and merge just one column from each file into a new dataset. It did so in ~2 hours. Yet the loop was just ~5 lines of code.

It is slow. And it is ok. Very few times will R ever beat any other language. Usually it is not off by much, but especially if coded by a novice using for loops vs apply functions can make is 100 -1000 x slower. Another example is the immutable structure that causes R to be a memory hog. Creating copies of data everywhere. But, again if you plan well and execute the 'best' solutions you can avoid the giant pitfalls b…

Post R 3.1 there are far fewer deep copies (e.g. modifying a list or adding a column to a data.frame no longer copies the whole thing like it used to).

Re: How R Took the World of Statistics by Storm

#34

Post by lottin » Sun, Nov 22, 2015, 3:57 PM UTC

B1gred wrote: post #32

lottin wrote: post #26

Earlier quoted context omitted.

What do you mean by data munging? Things like extracting data from XML files for instance?

It can mean anything, and that is why it is hard. Many old school statisticians can not work with anything other than csv, excel spreadsheets or basic sql queries. Munging is the conversion to a nice format that can then be used for analysis.

Understood. Yeah, this sounds like a job for awk or some such specialised tool.

Re: How R Took the World of Statistics by Storm

#35

Post by TheLogothete » Sun, Nov 22, 2015, 4:11 PM UTC

Simorgh wrote: post #21

anotheryou wrote: post #15

What would be a good start to learn this? I have some programming background and really would like to get into statistics. Should I do some R tutorial and throw my weblogs at it to see what I can do? Or is there some awesome learning resource you could share?

There is a nice book by Brett Lantz called Machine Learning. The first edition (which I have) built machine learners in R, I assume the new second edition does the same.

Statistics does not mean machine learning just fyi.

Re: How R Took the World of Statistics by Storm

#36

Post by jordigh » Sun, Nov 22, 2015, 4:12 PM UTC

cwyers wrote: post #4

jordigh wrote: post #2

I've regretted that Octave hasn't done for Matlab what R did for S. I understand some of the cirumstances that made this happen, but I'm deeply saddened of the entrenchment that Matlab has in scientific computing. It's getting chipped away little by little at the edges by Python, and to a lesser extent by Julia, but Matlab still is strong. And yes, some uses of Matlab can be replaced by R, but overall the two package…

> I understand some of the cirumstances that made this happen What are they?

The story has fallen off the front page, so there's no point in writing a detailed response to this. If you're still interested, email me at jordigh@octave.org for my analysis.

Re: How R Took the World of Statistics by Storm

#37

Post by grayclhn » Sun, Nov 22, 2015, 4:48 PM UTC

frik wrote: post #11

R replaced SPSS. Octave replaced Mathlab. Python based libraries are somewhere in between. Julia with Jupyter will probably replace Mathematica, LabVIEW and Mathcad (and unify all of the above) with a powerful native language and environment.

I have high hopes for Julia, but it's not really competing with Mathematica AFAIK, so I'm not really sure what's behind these claims.

Re: How R Took the World of Statistics by Storm

#38

Post by PlotlyThrowaway » Sun, Nov 22, 2015, 7:41 PM UTC

jasonpbecker wrote: post #24

PlotlyThrowaway wrote: post #22

Earlier quoted context omitted.

I'm a little at odds on this--for production quality analyses, (and only for analyses) R is excellent. However, in my experience, for the data munging required as a preliminary to the analyses, R is worse than bad. It's as if satan himself designed a language. I find that what then happens is this: data scientists/statisticians/[your favorite word here] become reliant on programmers to clean/format the data to do the…

I couldn't disagree more. R is great at munging pretty much everything but unstructured textual data. The tools are definitely behind Python if you're dealing with literal written documents. I don't know anyone who considers themselves a "data scientist" of any sort that doesn't view their job as 80% or more data wrangling/munging/cleaning. I write production ETL processes in R at my current job. AMA.

May I ask what tools you favor in the R environment? I just haven't found anything as performant for operations on irregular and poorly formatted time series as the pandas library, and in fact I just finished an ETL in pandas for my current job.

I'm always interested in learning a new tool, though.

Re: How R Took the World of Statistics by Storm

#39

Post by frik » Sun, Nov 22, 2015, 8:37 PM UTC

grayclhn wrote: post #37

frik wrote: post #11

R replaced SPSS. Octave replaced Mathlab. Python based libraries are somewhere in between. Julia with Jupyter will probably replace Mathematica, LabVIEW and Mathcad (and unify all of the above) with a powerful native language and environment.

I have high hopes for Julia, but it's not really competing with Mathematica AFAIK, so I'm not really sure what's behind these claims.

Not Julia alone, but Julia in combination with Jupyter notebook technology - at least for common usage. (Mathematica has of courses a lot of features)

Re: How R Took the World of Statistics by Storm

#40

Post by jasonpbecker » Sun, Nov 22, 2015, 9:42 PM UTC

PlotlyThrowaway wrote: post #38

jasonpbecker wrote: post #24

Earlier quoted context omitted.

I couldn't disagree more. R is great at munging pretty much everything but unstructured textual data. The tools are definitely behind Python if you're dealing with literal written documents. I don't know anyone who considers themselves a "data scientist" of any sort that doesn't view their job as 80% or more data wrangling/munging/cleaning. I write production ETL processes in R at my current job. AMA.

May I ask what tools you favor in the R environment? I just haven't found anything as performant for operations on irregular and poorly formatted time series as the pandas library, and in fact I just finished an ETL in pandas for my current job. I'm always interested in learning a new tool, though.

I don't work much with data that would benefit from being very tight about datetimes as a dimension. I'd have to know a bit more about what was challenging before I could confidently recommend for your particularly case. My email is on my profile and I'd be happy to chat there if it's something that would be helpful.

I have largely avoided ts, zoo, etc where possible. Time series stuff seems to have a lot of specialized tooling all of which tends to be much more strict about data structure than I'm comfortable with for my flow.