21–30 of 49 posts
What would be a good start to learn this? I have some programming background and really would like to get into statistics. Should I do some R tutorial and throw my weblogs at it to see what I can do? Or is there some awesome learning resource you could share?
Earlier quoted context omitted.
what are the problem domains ? Do you think something like Python Pandas... and especially Jupyter/IPython can replace it ?
Python has a steeper learning curve and is not as curtailed to simple data analysis. Many use Rstudio (an ide) and use the import data, and other tools to make then skill entry even lower. Also, mathematicians and statisticians think functionally and the general attitude in python is to do object oriented programming while R is strictly functional programming with a little bit of object programming.
However, in my experience, for the data munging required as a preliminary to the analyses, R is worse than bad. It's as if satan himself designed a language.
I find that what then happens is this: data scientists/statisticians/[your favorite word here] become reliant on programmers to clean/format the data to do the analyses.
This is all fine, but those same scientists are then put off learning python, where they could do all of their own munging, and probably 95% of the analysis they need to do, and where they could further add value by writing programs that are easier to production-alize.
Job security for those who know how to write production code, I guess.
What would be a good start to learn this? I have some programming background and really would like to get into statistics. Should I do some R tutorial and throw my weblogs at it to see what I can do? Or is there some awesome learning resource you could share?
http://www.meetup.com/Dallas-R-Users-Group/pages/R_Helpful_L...
Earlier quoted context omitted.
Python has a steeper learning curve and is not as curtailed to simple data analysis. Many use Rstudio (an ide) and use the import data, and other tools to make then skill entry even lower. Also, mathematicians and statisticians think functionally and the general attitude in python is to do object oriented programming while R is strictly functional programming with a little bit of object programming.
I'm a little at odds on this--for production quality analyses, (and only for analyses) R is excellent. However, in my experience, for the data munging required as a preliminary to the analyses, R is worse than bad. It's as if satan himself designed a language. I find that what then happens is this: data scientists/statisticians/[your favorite word here] become reliant on programmers to clean/format the data to do the…
I don't know anyone who considers themselves a "data scientist" of any sort that doesn't view their job as 80% or more data wrangling/munging/cleaning.
I write production ETL processes in R at my current job. AMA.
Earlier quoted context omitted.
Python has a steeper learning curve and is not as curtailed to simple data analysis. Many use Rstudio (an ide) and use the import data, and other tools to make then skill entry even lower. Also, mathematicians and statisticians think functionally and the general attitude in python is to do object oriented programming while R is strictly functional programming with a little bit of object programming.
I'm a little at odds on this--for production quality analyses, (and only for analyses) R is excellent. However, in my experience, for the data munging required as a preliminary to the analyses, R is worse than bad. It's as if satan himself designed a language. I find that what then happens is this: data scientists/statisticians/[your favorite word here] become reliant on programmers to clean/format the data to do the…
[1] http://opiateforthemass.es/articles/james-bond-film-ratings/
Earlier quoted context omitted.
Python has a steeper learning curve and is not as curtailed to simple data analysis. Many use Rstudio (an ide) and use the import data, and other tools to make then skill entry even lower. Also, mathematicians and statisticians think functionally and the general attitude in python is to do object oriented programming while R is strictly functional programming with a little bit of object programming.
I'm a little at odds on this--for production quality analyses, (and only for analyses) R is excellent. However, in my experience, for the data munging required as a preliminary to the analyses, R is worse than bad. It's as if satan himself designed a language. I find that what then happens is this: data scientists/statisticians/[your favorite word here] become reliant on programmers to clean/format the data to do the…
I program in R 80% of my day. I have experiences in all the major alternatives but keep returning to R. It has one huge flaw, being slow but otherwise is fantastic to work with and has a vibrant community. The bigger issue is that while R is liked by statisticians it lacks many of the features for the software development. We run across difficulties with logging, version control of packages, speed, size of docker ima…
I agree with your assertion that R is slow, yet quick to develop in. I recently had to loop through 1.3Gb of data (5000 files) and merge just one column from each file into a new dataset. It did so in ~2 hours. Yet the loop was just ~5 lines of code.
I wonder if you tried doing things like:
* preallocate a list, then do.call(cbind, your_data) * Same as above, but with some of the faster alternatives to cbind like dplyr::bind_cols or data.table::cbind * Use data.table, which has far faster joins than base R (so does dplyr) if you were doing a true merge/join
If it was truly just adding a column rom each file together into a file, these kinds of tasks are much better using UNIX tools, in my experience.
Earlier quoted context omitted.
Python has a steeper learning curve and is not as curtailed to simple data analysis. Many use Rstudio (an ide) and use the import data, and other tools to make then skill entry even lower. Also, mathematicians and statisticians think functionally and the general attitude in python is to do object oriented programming while R is strictly functional programming with a little bit of object programming.
I'm a little at odds on this--for production quality analyses, (and only for analyses) R is excellent. However, in my experience, for the data munging required as a preliminary to the analyses, R is worse than bad. It's as if satan himself designed a language. I find that what then happens is this: data scientists/statisticians/[your favorite word here] become reliant on programmers to clean/format the data to do the…
I program in R 80% of my day. I have experiences in all the major alternatives but keep returning to R. It has one huge flaw, being slow but otherwise is fantastic to work with and has a vibrant community. The bigger issue is that while R is liked by statisticians it lacks many of the features for the software development. We run across difficulties with logging, version control of packages, speed, size of docker ima…
packrat has helped a lot with version control of packages, but it still doesn't quite feel like the right solution.
I've been really impressed in the last 5 years how far R has come in these areas though, so like you I keep coming back. By the time I start getting over the learning curve other places, R seems to have developed better tooling for what I want to accomplish anyway and I can come back and right cleaner, clearer, better software faster in R.
I program in R 80% of my day. I have experiences in all the major alternatives but keep returning to R. It has one huge flaw, being slow but otherwise is fantastic to work with and has a vibrant community. The bigger issue is that while R is liked by statisticians it lacks many of the features for the software development. We run across difficulties with logging, version control of packages, speed, size of docker ima…
I agree with your assertion that R is slow, yet quick to develop in. I recently had to loop through 1.3Gb of data (5000 files) and merge just one column from each file into a new dataset. It did so in ~2 hours. Yet the loop was just ~5 lines of code.
Another example is the immutable structure that causes R to be a memory hog. Creating copies of data everywhere. But, again if you plan well and execute the 'best' solutions you can avoid the giant pitfalls but will rarely ever beat a equally well written python equivalent.