Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

R: Introduction to Data Science (2019)

rafalab.dfci.harvard.edu

Post reply on HN
Print view View original

111–120 of 139 posts

Re: R: Introduction to Data Science (2019)

#111

Post by omaranto »

guccigav wrote: post #5
chollida1 wrote: post #3

i'm an old R user, now migrated fully to python. For those of you who us R still what is your use case? We found R has a really hard time integrating into data pipelines and was best used as a standalone tool by individuals, which doesn't really work in our particular professional setup where everyone works collaboratively together. What we found was that R had alot of packages but most haven't been touched in years…

Exactly what you said, R is easy to get started for individuals in social science fields. Most people I know who want to dive deeper end up learning Python anyway.

And people wishing to delve even deeper switch back to R? I don't use it, but I understood that most advanced techniques in statistics are implemented first, and sometimes only, as R packages, no?

Re: R: Introduction to Data Science (2019)

#112

Post by hadley »

mushufasa wrote: post #110
hadley wrote: post #67

Earlier quoted context omitted.

A lighterweight alternative to renv is to use Posit Public Package Manage ( https://packagemanager.posit.co/ ) with a pinned date. That doesn't help if you're installing packages from a mix of places, but if you're only using CRAN packages it lets you get everything as of a fixed date. And of course on the web side you have shiny ( https://shiny.posit.co ), which now also comes in a python flavour.

shiny is nice for one-off data dashboards and single-purpose mini-apps. I see the python equivalents are like dash/plotly. Shiny is not a full fledged web framework, and isn't a viable replacement for e.g. Django. Aside -- we tried using dash in our production app and then had to remove it after a month, because these types of frameworks that spit out front-end code are almost never flexible enough to do what you act…

I'd highly encourage you to look into shiny more. No, it's not django, but it's a much richer framework than dash, and you can always bring your own HTML if what it generates for you isn't sufficient.

Re: R: Introduction to Data Science (2019)

#113

Post by hadley »

fastaguy88 wrote: post #88
hadley wrote: post #20

Earlier quoted context omitted.

If you tell me what makes R hard to integrate into data pipelines I will do my best to fix it :)

(1) The big problem I have is transitioning from RStudio to a pipeline (so I end up not using RStudio). A traditional pipeline is going to be a script with some set of arguments -- parameter values, fitting functions, and data file names, that I put into a shell script and say: my_plot_script.R --plot_col=g_max --output_type=pub_quality data_file1 data_file2 data_file3 It's possible to use optparse/OptionParser() to…

(1) You might want to check out https://github.com/t-kalinowski/Rapp by my colleague Tomasz

(2) I think part of that is in scope for strict (https://github.com/hadley/strict). You might also be well served by adopting some more data validation tooling, e.g. pointblank (https://rstudio.github.io/pointblank/).

Re: R: Introduction to Data Science (2019)

#114

Post by hadley »

laylower wrote: post #78
hadley wrote: post #70

Earlier quoted context omitted.

What are the premium packages you're talking about? As far as I know all of our R packages are 100% open source. I'd love to hear more why you're using webshot etc to talk screenshots of your shiny app. A more typical workflow would be to generate a separate HTML/PDF with quarto/RMarkdown.

Thanks for responding and your amazing work with the tidyverse. I am the "R-guy" in my finservices company and we have a paid rconnect dev/qa/prod and rserver pro licences for a few hundred users. The packages I think are the dependencies of some DB connectivity libraries. https://www.rstudio.com/tags/databases/ - these are the ones I was referring to. Re webshot my use case is: I have a heavily modified DT table in…

Oh, you mean the pro drivers? Unfortunately we can't give those away because we have to pay several $100k a year just to get access for our customers. Most of the pro drivers do have equivalent open source versions that you should be able to use instead.

Hmmm, I'd still try generating the table with quarto (since you can output word documents), or try gt (https://gt.rstudio.com), which I know has much greater control over output, and supports RTF output (https://gt.rstudio.com/reference/as_rtf.html) which should import cleanly into word.

Re: R: Introduction to Data Science (2019)

#115

Post by hadley »

nutshell42 wrote: post #106
doodledoodahs wrote: post #42

Earlier quoted context omitted.

OK, since you're here! (this all prefaced with a massive thank you for tidyverse, without which R is very crusty). I love R for interactive work and quick analyses, but I'm currently trying to integrate various bits of R code into a large document-building pipeline and wishing I could use Python for it: - Exception handling and error processing seem a pain in R. Maybe I'm doing it wrong, but if feels like a mess and…

> The distinction between warning, stop etc seems odd. The option to stop on warnings isn't useful because older packages seem to abuse warnings as messages. Use suppressWarnings() to silence misbehaving functions or withCallingHandlers() to stop or handle specific conditions. > Passing variable names as strings to dynamically generate things seems clunky compared with python. Can you give me an elegant example in Py…

For the most common cases, the tidyverse now only requires {{ }}. This allows you to tell tidyeval functions that you have the name of a df-var stored in an env-var. Do you have specific cases that you find frustrating?

Re: R: Introduction to Data Science (2019)

#116

Post by hadley »

civilized wrote: post #105
ramblenode wrote: post #101

Earlier quoted context omitted.

> My fondest hope is that in the coming decades there arises a new, thoughtfully designed language with the Lispy flexibility of R, but also optional type safety and static analysis affordances. I think many of us saw Julia as the successor to R. Unfortunately, the package ecosystem---one of R's strongest points---still has a long way to go.

I was excited about Julia too but it now seems to be a relatively niche HPC language. It's about saving CPU time more than user time. My sniff test for a successor language to R is whether it can replicate the tidyverse API with 100% fidelity. The API is already optimal for tabular data analysis, especially the dplyr core. It can be thought of as a specification for other languages to implement. There is a great deal…

I love this framing :)

Re: R: Introduction to Data Science (2019)

#117

Post by hadley »

civilized wrote: post #85
hadley wrote: post #71

Earlier quoted context omitted.

If you have specific issues around error messages and tracebacks please feel free to let me know directly or to file issues on Github. We really do care about the legibility of errors and tracebacks and me and my team have put a lot of effort into them in the last few years. But there's always room to do better and I'd love to know where the pain points are. (The intersection of tidyverse and shiny tracbacks are a kn…

There's clearly been a ton of progress in this area; the only issue is that feature development is even faster :) I'll keep an eye out for specific issues that seem helpful to raise. The biggest one I have right now is a little niche, but probably useful to address. Moderately complex dbplyr pipelines on wide tables have a tendency to generate very long queries, and if there's an error, the generated SQL returned ten…

Hmmmm, I think that's something we could probably help with in dbplyr by providing something like `last_sql()` that would return the most recent SQL sent to the database. (By analogy to ggplot2::last_plot() and httr2::last_request()/last_response()).

I filed an issue so I don't forget about this: https://github.com/tidyverse/dbplyr/issues/1471

Re: R: Introduction to Data Science (2019)

#118

Post by j7ake »

nomilk wrote: post #14
chollida1 wrote: post #3

i'm an old R user, now migrated fully to python. For those of you who us R still what is your use case? We found R has a really hard time integrating into data pipelines and was best used as a standalone tool by individuals, which doesn't really work in our particular professional setup where everyone works collaboratively together. What we found was that R had alot of packages but most haven't been touched in years…

About half our team can wrangle and plot as fast as we can think of ideas. It creates an incredibly tight cycle time between us having ideas and getting answers; sometimes many (e.g. 10-20+) of those cycles in a single meeting. Before we used R, it would require someone jotting down things to investigate and reporting back in the next meeting. But we can do ~80% of whatever people can think of on the spot (more invol…

For exploring data, plotting data, and fitting complex models like GLMs and GAMs, using R is essentially as fast as thought at this point.

So if you have tabular data, it’s a no brainer to use R.

Getting the data into table form is often better suited for python. Fitting models that leverage autograd are also better with python.

Re: R: Introduction to Data Science (2019)

#119

Post by civilized »

civilized wrote: post #105
ramblenode wrote: post #101

Earlier quoted context omitted.

> My fondest hope is that in the coming decades there arises a new, thoughtfully designed language with the Lispy flexibility of R, but also optional type safety and static analysis affordances. I think many of us saw Julia as the successor to R. Unfortunately, the package ecosystem---one of R's strongest points---still has a long way to go.

I was excited about Julia too but it now seems to be a relatively niche HPC language. It's about saving CPU time more than user time. My sniff test for a successor language to R is whether it can replicate the tidyverse API with 100% fidelity. The API is already optimal for tabular data analysis, especially the dplyr core. It can be thought of as a specification for other languages to implement. There is a great deal…

Follow-up: I did a little legwork and found some Julia folks who were also inspired to implement tidyverse as faithfully as possible: https://github.com/TidierOrg

Here's how dplyr-style chains look in their system:

  using TidierData
  using RDatasets

  movies = dataset("ggplot2", "movies");

  @chain movies begin
      @mutate(Budget = Budget / 1_000_000)
      @filter(Budget >= mean(skipmissing(Budget)))
      @select(Title, Budget)
      @slice(1:5)
  end
Not a character-for-character match to dplyr, but gets much closer than most other attempts!

Re: R: Introduction to Data Science (2019)

#120

Post by trts »

guccigav wrote: post #5
chollida1 wrote: post #3

i'm an old R user, now migrated fully to python. For those of you who us R still what is your use case? We found R has a really hard time integrating into data pipelines and was best used as a standalone tool by individuals, which doesn't really work in our particular professional setup where everyone works collaboratively together. What we found was that R had alot of packages but most haven't been touched in years…

Exactly what you said, R is easy to get started for individuals in social science fields. Most people I know who want to dive deeper end up learning Python anyway.

this is kinda funny to me. practically every social scientist I’ve known that had to use R was appalled by how cryptic and unintuitive it was.

I always understood that it was its open source nature being a successor to the S language that gave it traction

Post reply on HN