Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

R: Introduction to Data Science (2019)

rafalab.dfci.harvard.edu

Post reply on HN
Print view View original

81–90 of 139 posts

Re: R: Introduction to Data Science (2019)

#81

Post by minimaxir »

chollida1 wrote: post #3

i'm an old R user, now migrated fully to python. For those of you who us R still what is your use case? We found R has a really hard time integrating into data pipelines and was best used as a standalone tool by individuals, which doesn't really work in our particular professional setup where everyone works collaboratively together. What we found was that R had alot of packages but most haven't been touched in years…

There is currently no Python equivalent for both the ease of use and output quality of ggplot2 for data visualization. Many have tried over the past decade, but none have gotten close. (Plotnine was the closest: per Hadley in another comment his company hired the maintainer)

Re: R: Introduction to Data Science (2019)

#82

Post by downrightmike »

jslakro wrote: post #15
chollida1 wrote: post #3

i'm an old R user, now migrated fully to python. For those of you who us R still what is your use case? We found R has a really hard time integrating into data pipelines and was best used as a standalone tool by individuals, which doesn't really work in our particular professional setup where everyone works collaboratively together. What we found was that R had alot of packages but most haven't been touched in years…

What python learning material you recommend focused on data science?

I like this and they have a video course on o'reilly https://www.amazon.com/Python-Programmers-Artificial-Intelli...

Re: R: Introduction to Data Science (2019)

#83

Post by mint2 »

SoftTalker wrote: post #61

I took a two or three day on-site intro to R class that my employer put together. Perhaps it was not a great class, but as a seasoned software developer familiar with a number of imperative and functional languages I was baffled by R. It felt like a bunch of little functions that had been developed by different people with no consistent framework, and thrown together in some kind of big wrapper. I know it's popular a…

I get a similar impression but to contextualize, in terms of statistical programming what you’re saying is even more so true of what came before R, but a thousand fold worse. In that context R is fantastic.

For example SAS makes R look beautiful and consistent. And that’s more a comment on SAS than R. And this isn’t to say python is perfect either, but I prefer it.

Re: R: Introduction to Data Science (2019)

#84

Post by mslip1 »

hadley wrote: post #20
chollida1 wrote: post #3

i'm an old R user, now migrated fully to python. For those of you who us R still what is your use case? We found R has a really hard time integrating into data pipelines and was best used as a standalone tool by individuals, which doesn't really work in our particular professional setup where everyone works collaboratively together. What we found was that R had alot of packages but most haven't been touched in years…

If you tell me what makes R hard to integrate into data pipelines I will do my best to fix it :)

Hey Hadley!! Personally only issues for me with integrating R is making renv play nice in multistage docker builds. I found that I need to have my other pipeline software built in the same stage as my R env setup (building specific version from archive, system dependencies, then r package dependencies via renv)

Re: R: Introduction to Data Science (2019)

#85

Post by civilized »

hadley wrote: post #71
civilized wrote: post #23

Earlier quoted context omitted.

> We found R has a really hard time integrating into data pipelines and was best used as a standalone tool by individuals In my org we have several 100% R teams (including mine) that have been developing and maintaining business-critical, data-intensive applications for a decade now. We don't find R difficult to integrate into data pipelines. We write our data pipelines in R, and we find it very efficient to do so. T…

If you have specific issues around error messages and tracebacks please feel free to let me know directly or to file issues on Github. We really do care about the legibility of errors and tracebacks and me and my team have put a lot of effort into them in the last few years. But there's always room to do better and I'd love to know where the pain points are. (The intersection of tidyverse and shiny tracbacks are a kn…

There's clearly been a ton of progress in this area; the only issue is that feature development is even faster :) I'll keep an eye out for specific issues that seem helpful to raise.

The biggest one I have right now is a little niche, but probably useful to address. Moderately complex dbplyr pipelines on wide tables have a tendency to generate very long queries, and if there's an error, the generated SQL returned tends to overflow some text or line limit allotted to show the error at the command prompt. My workaround is to use sink() to dump the error to a file, which is a little painful as the sink() API and documentation are not the most straightforward or intuitive. (Hmm, I wonder if a withr wrapper would help me make something simpler to use...)

Re: R: Introduction to Data Science (2019)

#86

Post by armchairhacker »

AndyMcConachie wrote: post #55

I dipped my feet into R a few years back, but eventually stopped it because of the way it handles integers. At the time it treated all integers internally as signed 32-bit and if the number is too large for that it converted it to a float. I don't know what R does now, but this was a deal breaker for me at the time because I was dealing with really large integers that regularly broke this limit.

Integers are still only 32 bits. There’s a class which effectively represents 64-bit integers (https://www.rdocumentation.org/packages/csvread/versions/1.2...) as well as arbitrary-sized (https://cran.r-project.org/web/packages/gmp/index.html, https://www.rdocumentation.org/packages/gmp/versions/0.7-4/t...). I will say there are a few pitfalls where the integer bits are unexpectedly converted to something else, but it’s workable.

Re: R: Introduction to Data Science (2019)

#88

Post by fastaguy88 »

hadley wrote: post #20
chollida1 wrote: post #3

i'm an old R user, now migrated fully to python. For those of you who us R still what is your use case? We found R has a really hard time integrating into data pipelines and was best used as a standalone tool by individuals, which doesn't really work in our particular professional setup where everyone works collaboratively together. What we found was that R had alot of packages but most haven't been touched in years…

If you tell me what makes R hard to integrate into data pipelines I will do my best to fix it :)

(1) The big problem I have is transitioning from RStudio to a pipeline (so I end up not using RStudio). A traditional pipeline is going to be a script with some set of arguments -- parameter values, fitting functions, and data file names, that I put into a shell script and say:

my_plot_script.R --plot_col=g_max --output_type=pub_quality data_file1 data_file2 data_file3

It's possible to use optparse/OptionParser() to get that information (but you have an option for every argument, no --param1 X --param2 Y file1 file2 file3) but it is much more difficult to fit those arguments into the RStudio environment. I want an RStudio to be able emulate reading command line arguments (since they do not exist in RStudio). Right now, I have to check to see if there are commandArgs(), and, if not, do something else to get the information to the RStudio script.

(2) There needs to be an option that says STOP if something doesn't make sense. I have dozens of beautiful data plots that look great, but in fact do not in fact plot what I think they do, because factors have not been properly assigned to colors, shapes, or linetypes. (And it can be really hard to recognize that the data has not been plotted properly.) Give me an option that says, if I did not explicitly declare a column a factor, and I did not specifically associate colors/shapes/lines with factors, then the data will not be plotted.

Re: R: Introduction to Data Science (2019)

#89

Post by trts »

chollida1 wrote: post #3

i'm an old R user, now migrated fully to python. For those of you who us R still what is your use case? We found R has a really hard time integrating into data pipelines and was best used as a standalone tool by individuals, which doesn't really work in our particular professional setup where everyone works collaboratively together. What we found was that R had alot of packages but most haven't been touched in years…

two reasons for me

1) tidyverse makes prodding and plotting my data faster and more enjoyable. when I am prototyping a model I'll sometimes do the groundwork in R and then migrate the production version to python

2) I can't seem to write data wrangling code in py that is as aesthetically pleasing and easy to reinterpret later. could just be that I started in R, but while the methods in pandas "work" I don't always totally understand why they work the way they do. with tidy it works the way I expect and feels easier to read back and iterate on

Post reply on HN