Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

One Year with R

github.com

Post reply on HN
Print view View original

161–170 of 266 posts

Re: One Year with R

#161

Post by dxbydt »

Lot of useful insights in the comments here. I wanted to address one specific comment -

>can't remember the last time I saw a project someone did in R get very much traction anywhere...the only time people talk about R on the internet is to discuss the language itself which is definitely frustrating

There is a lot of R deployed in industry, even in silicon valley, but you have to be in-the-know. R gets plenty of use in statarb & model checking in finance - speaking from personal experience at GS & BofA/ML. My one non-trivial project at Twitter involved working with this team building a model & I remarked - hey this can be done rather easily if you use this library in R - and the teamlead says, yeah that's how we're doing it! But I thought we are a Scala shop, I said. So he says, yeah but imagine building that entire library in Scala from scratch, it'll take forever! So I enquired how he gets it done - you basically spin up a socket server & the jvm sends R commands plus data as payload over the socket, the server runs R and returns the result of the model back as a string, boom done! I said it was kinda janky & he says - I won't tell if you don't ! So that's R for you - it gets the job done & its fast & somewhat messy, but it is used everywhere, yet people won't openly admit to it because its a 30 year old language & we all want to be using the latest & greatest tool.

I now work at a news startup with a few million users, & all of the news personalization is done in R. So when these millions of viewers watch TV, the piece of code that decides which news clip should be shown ahead of which other news clip & which clip comes after - all of that is decided by a block of R code that I wrote. ~ 300 lines of R, uses quanteda, tidytext & parallel under the hood. Pretty much everything I do involves mcmapply, which parallelizes your compute & uses as many cores as you specify. But that's sort of the thing with R - you have to know which functions/libs to use & which ones to avoid. Just switching from tm to quanteda got us a 200% bump in perf. Switching sapply's to mcmapply was another winner. These things aren't documented cleanly - you have to keep up with cran, experiment & see what works best for you.

Re: One Year with R

#162

Post by _dain_ »

dm319 wrote: post #128
tel wrote: post #61

R, and by R I mean R+tidyverse, is the world's best graphing calculator attached to an OK scheme. To which I mean R is a highly optimized, well-oiled machine if you're using it for its highly-optimized, well-oiled purposes. I tend to have notebooks full of tiny fragments like this dat_min %>% group_by(ymd = make_date(year(date), month(date), day(date))) %>% summarize(vol_btc=sum(vol_btc), vol_usdt=sum(vol_usdt), trad…

This is just a quick example - I would be grateful if people could recreate this brief look at UK COVID figures in another language: library(tidyverse) library(scales) download.file(url = "https://api.coronavirus.data.gov.uk/v2/data?areaType=overview&metric=covidOccupiedMVBeds&metric=newAdmissions&metric=newCasesBySpecimenDate&metric=newDeaths28DaysByDeathDate&metric=newPeopleReceivingFirstDose&format=csv", destfile…

    import pandas as pd
    import matplotlib.pyplot as plt
    import matplotlib.dates as mdates
    import seaborn as sn

    df = (pd.read_csv("/tmp/overview_2022-03-21.csv") # i just used curl beforehand
        .assign(date=lambda x: pd.to_datetime(x["date"]))
        .set_index("date")
        .melt(value_vars=[
                "newCasesBySpecimenDate",
                "covidOccupiedMVBeds",
                "newAdmissions",
                "newDeaths28DaysByDeathDate"],
            var_name="Data", ignore_index=False)
        .assign(Data=lambda x: x["Data"].replace({
            "newCasesBySpecimenDate": "New Cases",
            "newAdmissions": "Admissions",
            "newDeaths28DaysByDeathDate": "Deaths",
            "covidOccupiedMVBeds": "Ventilated"
        }))
    )
    ax = sn.scatterplot(data=df, x=df.index, y=df["value"], hue="Data")
    ax.set(xlabel="Date", ylabel="Daily rate", yscale="log")
    ax.xaxis.set_major_formatter(mdates.DateFormatter("%b"))
    plt.show()
I spend 2 minutes on the pandas part and 20 minutes on the plotting part, which really says it all. Seaborn's support for smoothing is really bad and doesn't play nicely with datetimes for some reason, so if I wanted smoothing I'd need to do it myself. And the other stuff I left out requires going into matplotlib's documentation which I don't want to spend time on.

pandas is as good or better than R's dataframe manipulation, but R's plotting tools are best in class. I hate all the python plotting libraries.

Re: One Year with R

#163

Post by scientism »

_Wintermute wrote: post #94
ggrothendieck wrote: post #49

Earlier quoted context omitted.

1. Before R commercial statistical packages were mainly used. You can, in principle, just use assembler too and develop everything yourself but it isn't practical. Regarding C/C++ and Fortran, many R packages are, in fact, wrappers around code in those or other languages making it easier to access them. From that point of view R can be regarded as a glue language. 2. Regarding keeping versions straight, all past vers…

> Before R commercial statistical packages were mainly used. Maybe in your field, I work in bioinformatics - before R, perl was widely used as a high-level language. > Regarding keeping versions straight, all past versions of packages in the CRAN repository are kept on CRAN... This is woefully inadequate if you need to replicate somebody else's environment. Nobody should think manually guessing and then typing in eac…

Totally agree. I find it frustrating trying to reproduce other people's work in R. How has this situation has been allowed to continue for so long? It's unacceptable, especially when used for science. It's impossible to replicate anything unless you are lucky enough you manage to find which package version introduces breaking changes and even then this is something you have to do repeatedly for every code break. Even with _renv_ it's a library you have to install within your R environment which is pointless. Where is a dependency solver like conda for R? - Not that it's perfect, but I've been happy with its drop-in replacement - mamba recently.

Re: One Year with R

#164

Post by deehouie »

hadley wrote: post #64
indeed30 wrote: post #12

I love R more than any other language I have ever used. Perhaps more than any piece of software I've ever used. All of these points are valid, and yes, it's messy, and if you try to write the same type of code that you would in Python, it will frustrate you. And yet.. it somehow works. It makes data analysis and statistical modelling a pleasure. It somehow gives off a sense of lightness, and makes it easy to investig…

100% this :)

Bravo!

Re: One Year with R

#165

Post by ggrothendieck »

_Wintermute wrote: post #94
ggrothendieck wrote: post #49

Earlier quoted context omitted.

1. Before R commercial statistical packages were mainly used. You can, in principle, just use assembler too and develop everything yourself but it isn't practical. Regarding C/C++ and Fortran, many R packages are, in fact, wrappers around code in those or other languages making it easier to access them. From that point of view R can be regarded as a glue language. 2. Regarding keeping versions straight, all past vers…

> Before R commercial statistical packages were mainly used. Maybe in your field, I work in bioinformatics - before R, perl was widely used as a high-level language. > Regarding keeping versions straight, all past versions of packages in the CRAN repository are kept on CRAN... This is woefully inadequate if you need to replicate somebody else's environment. Nobody should think manually guessing and then typing in eac…

The packages that were used in statistics were SAS, SPSS and Stata. perl is not a statistical package and has nowhere near the depth of statistical capabilities of R.

Don't forget that I also mentioned the checkpoint package in my post. You only need to know the date for that, not the version of each of the packages.

In your last paragraph I think you are referring more to software development practices than what is available through R. Simply using R or any language doesn't guarantee this.

Re: One Year with R

#166

Post by _dain_ »

CapmCrackaWaka wrote: post #78
tel wrote: post #61

R, and by R I mean R+tidyverse, is the world's best graphing calculator attached to an OK scheme. To which I mean R is a highly optimized, well-oiled machine if you're using it for its highly-optimized, well-oiled purposes. I tend to have notebooks full of tiny fragments like this dat_min %>% group_by(ymd = make_date(year(date), month(date), day(date))) %>% summarize(vol_btc=sum(vol_btc), vol_usdt=sum(vol_usdt), trad…

I also use R for any heavy data manipulation, but I primarily use the data.table package. The efficiency that both of these packages unlock is absolutely unparalleled in any other tabular data manipulation library, in any other language that I have used. And R has the top 2!! My skin writhes every time I need to type: table.loc[(table.column > 2) | (table.column2 when I want to subset a table.

.loc lets you supply a callable, so you can write:

  table.loc[lambda df: df["column"].between(2, 3, inclusive="neither")]
this is useful when your dataframe has a long name, or when you have some long method chain and you need to subset at the end: table.foo().bar().baz().loc[lambda df: ...]

it is still more verbose, but I actually prefer always providing column names as strings. it's more explicit. I don't like R's environment-manipulation metaprogramming magic where you can give column names as symbols.

as for resetting the index all the time, this is something of an antipattern. if you set up your index right beforehand it isn't necessary so often.

Re: One Year with R

#167

Post by nuclearnice1 »

_dain_ wrote: post #162
dm319 wrote: post #128

Earlier quoted context omitted.

This is just a quick example - I would be grateful if people could recreate this brief look at UK COVID figures in another language: library(tidyverse) library(scales) download.file(url = "https://api.coronavirus.data.gov.uk/v2/data?areaType=overview&metric=covidOccupiedMVBeds&metric=newAdmissions&metric=newCasesBySpecimenDate&metric=newDeaths28DaysByDeathDate&metric=newPeopleReceivingFirstDose&format=csv", destfile…

import pandas as pd import matplotlib.pyplot as plt import matplotlib.dates as mdates import seaborn as sn df = (pd.read_csv("/tmp/overview_2022-03-21.csv") # i just used curl beforehand .assign(date=lambda x: pd.to_datetime(x["date"])) .set_index("date") .melt(value_vars=[ "newCasesBySpecimenDate", "covidOccupiedMVBeds", "newAdmissions", "newDeaths28DaysByDeathDate"], var_name="Data", ignore_index=False) .assign(Dat…

This is great.

You don't need the curl -- read_csv works with URLs directly.

The lambda can be replaced by passing parse_dates=[“date”]

Re: One Year with R

#168

Post by _dain_ »

nuclearnice1 wrote: post #167
_dain_ wrote: post #162

Earlier quoted context omitted.

import pandas as pd import matplotlib.pyplot as plt import matplotlib.dates as mdates import seaborn as sn df = (pd.read_csv("/tmp/overview_2022-03-21.csv") # i just used curl beforehand .assign(date=lambda x: pd.to_datetime(x["date"])) .set_index("date") .melt(value_vars=[ "newCasesBySpecimenDate", "covidOccupiedMVBeds", "newAdmissions", "newDeaths28DaysByDeathDate"], var_name="Data", ignore_index=False) .assign(Dat…

This is great. You don't need the curl -- read_csv works with URLs directly. The lambda can be replaced by passing parse_dates=[“date”]

Thanks

Re: One Year with R

#169

Post by [deleted] »

magnio wrote: post #60
epistasis wrote: post #53

Earlier quoted context omitted.

The tidyverse docs are the only ones with the super frustrating ... of impenetrable gnostic "documentation" that I know of. In general the tidyverse documentation is horrible, almost as bad as typical Python docs, IMHO. Other parts of base R are wonderfully documented in my opinion.

> almost as bad as typical Python docs I found numpy, scipy, pandas, and plotly docs to be quite clear and extensive. The only docs I have found to be confusing are matplotlib's and the Python standard library's. Not sure what packages you are referring to?

[deleted]

Re: One Year with R

#170

Post by SubiculumCode »

indeed30 wrote: post #12

I love R more than any other language I have ever used. Perhaps more than any piece of software I've ever used. All of these points are valid, and yes, it's messy, and if you try to write the same type of code that you would in Python, it will frustrate you. And yet.. it somehow works. It makes data analysis and statistical modelling a pleasure. It somehow gives off a sense of lightness, and makes it easy to investig…

Thanks for expressing how I feel about R so succinctly.
Post reply on HN