Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most commonly used statistical tests and implementation in R

r-statistics.co

Post reply on HN
Print view View original

21–30 of 34 posts

Re: Most commonly used statistical tests and implementation in R

#21

Post by stared »

minimaxir wrote: post #5
earino wrote: post #3

Earlier quoted context omitted.

It seems like this list is incomplete without mentioning that both RStudio[1] and Jupyter[2] notebooks now have really first class support for R. There are also two upstatrs, Rodeo[3] and Beaker[4] are doing cool stuff as well. The company I work for, Domino Data Lab[5], let's you fire up a lot of these notebooks in a nice hosted environment on big cloud servers with minimal cost and effort. It's a fun way to learn h…

> Jupyter[2] notebooks now have really first class support for R. Jupyter and R is a bit iffy since the R kernel is not native. Although the kernel works fine, setting it up has a ton of manually-installed dependencies, and in-line plots flat-out give unexpected output. (I've had to cheat by embeding charts via Markdown. Although that has the benefit of having the charts be responsive) The important perk is that Jupy…

Manually setting it is hard (on OS X + Homebrew Python I did it after a long fight; main problem: rmzq library). But... it is super easy with Anaconda: https://www.continuum.io/blog/developer/jupyter-and-conda-r

Re: Most commonly used statistical tests and implementation in R

#22

Post by sandGorgon »

phillc73 wrote: post #20
sandGorgon wrote: post #15

Earlier quoted context omitted.

Interesting. Did you post about this on /r/rprogramming There are lots of people who have rolled their own solutions for production deployment. Including nodejs !

I've also used Shiny to construct fairly complicated web apps, with dashboards (using shinydashboard). It's very good, up to a point. I keep banging my head against issues around persistent data storage and app customisation at a user level. Unless one pays for Shiny Server Pro, the free Shiny Server doesn't support user authentication. Hosting on shinyapps.io doesn't really support persistent user data, unless it's…

That's interesting to know. The reason shinydashboard or something else a nice is because the data scientist can work on his own. Expecting him to learn js would be a dead end.

I am willing to bet that he would rather use R api and excel to build a dashboard, rather than anything else.

Re: Most commonly used statistical tests and implementation in R

#23

Post by GFK_of_xmaspast »

ekianjo wrote: post #16

> If the p-Value is less than significance level (ideally 0.05), Erm, no. P=0.05 is borderline meaningless, there could as much as 30% chance you are wrong about the actual difference being there depending on the true probability of the initial hypothesis. P-values should be used with strong caution.

> could as much as 30% chance you are wrong about the actual difference being there depending on the true probability of the initial hypothesis.

I'm having trouble parsing this, are you talking about the power of the test?

Re: Most commonly used statistical tests and implementation in R

#24

Post by phillc73 »

sandGorgon wrote: post #22
phillc73 wrote: post #20

Earlier quoted context omitted.

I've also used Shiny to construct fairly complicated web apps, with dashboards (using shinydashboard). It's very good, up to a point. I keep banging my head against issues around persistent data storage and app customisation at a user level. Unless one pays for Shiny Server Pro, the free Shiny Server doesn't support user authentication. Hosting on shinyapps.io doesn't really support persistent user data, unless it's…

That's interesting to know. The reason shinydashboard or something else a nice is because the data scientist can work on his own. Expecting him to learn js would be a dead end. I am willing to bet that he would rather use R api and excel to build a dashboard, rather than anything else.

If you just want a personal or internal dashboard, then Shiny is a good solution, but you will need to deal with setting up and managing Shiny Server. If you don't care about the world viewing the dashboard, or can afford a small fee each month, then shinyapps.io is a decent solution.

I just wanted to point out that there are limitations with this route, when the apps start to become more complex, with multiple users, various access permissions and personalisation requirements.

Re: Most commonly used statistical tests and implementation in R

#25

Post by TheLogothete »

IndianAstronaut wrote: post #19
minimaxir wrote: post #5

Earlier quoted context omitted.

> Jupyter[2] notebooks now have really first class support for R. Jupyter and R is a bit iffy since the R kernel is not native. Although the kernel works fine, setting it up has a ton of manually-installed dependencies, and in-line plots flat-out give unexpected output. (I've had to cheat by embeding charts via Markdown. Although that has the benefit of having the charts be responsive) The important perk is that Jupy…

Package installation can be a bit of an issue as well, especially if you accidentally install a package twice. But overall, I still prefer notebooks to Rstudio. They are transparent and you can really trace your progress and share the info with others.

And you can't with Rstudio? There's a freaking notebook feature built right in.

Re: Most commonly used statistical tests and implementation in R

#26

Post by capnrefsmmat »

GFK_of_xmaspast wrote: post #23
ekianjo wrote: post #16

> If the p-Value is less than significance level (ideally 0.05), Erm, no. P=0.05 is borderline meaningless, there could as much as 30% chance you are wrong about the actual difference being there depending on the true probability of the initial hypothesis. P-values should be used with strong caution.

> could as much as 30% chance you are wrong about the actual difference being there depending on the true probability of the initial hypothesis. I'm having trouble parsing this, are you talking about the power of the test?

No, the base rate. If you're doing a significance test to detect a rare disease that only occurs in a tiny fraction of people, for example, the majority of statistically significant results you obtain will be false, despite having p This is a common problem in many fields, and you can use false discovery rate control methods to account for it: http://www.statisticsdonewrong.com/p-value.html

Re: Most commonly used statistical tests and implementation in R

#27

Post by IndianAstronaut »

TheLogothete wrote: post #25
IndianAstronaut wrote: post #19

Earlier quoted context omitted.

Package installation can be a bit of an issue as well, especially if you accidentally install a package twice. But overall, I still prefer notebooks to Rstudio. They are transparent and you can really trace your progress and share the info with others.

And you can't with Rstudio? There's a freaking notebook feature built right in.

Much easier to turn the Jupyter noteboks into a clean html format. Also, it's sequential, with Rstudio it's nice, but doesn't have the cell format.

Re: Most commonly used statistical tests and implementation in R

#29

Post by cetacea »

ekianjo wrote: post #16

> If the p-Value is less than significance level (ideally 0.05), Erm, no. P=0.05 is borderline meaningless, there could as much as 30% chance you are wrong about the actual difference being there depending on the true probability of the initial hypothesis. P-values should be used with strong caution.

Even better, p-values should not be used at all. If I have data in hand, I want to use it to find out the probability that my hypothesis is true. But p-value analysis requires me to instead ask a different question that I don't really care about, involving whether my data are consistent with the null hypothesis.

Everything is just so much more sensible if you allow yourself to assign probabilities to hypotheses, rather than assuming a hypothesis from the outset and computing opaque statistics relating to your data.

Re: Most commonly used statistical tests and implementation in R

#30

Post by earino »

sandGorgon wrote: post #15
earino wrote: post #14

Earlier quoted context omitted.

Well, I really don't want to turn this into a sales pitch, but that's the exact use case for Domino's API endpoints. Check out http://support.dominodatalab.com/hc/en-us/articles/204173149... for an explanation. If you're interested, drop me an email and I can work up an example project for you. Exposing R algorithms as REST endpoints is exactly what it does quite well. As for OpenCPU, I know that the guy who wrote it…

Interesting. Did you post about this on /r/rprogramming There are lots of people who have rolled their own solutions for production deployment. Including nodejs !

Greetings!

So I made a small demo for you. If you go to https://app.dominodatalab.com/earino/d3_dashboard_demo/raw/6... you will see a very simplistic, d3 powered, R backed dashboard. It just draws a pie chart, a line chart, and a bar chart. Every 10 seconds, it polls an R API endpoint to get new data.

The code for the R endpoint is https://app.dominodatalab.com/earino/d3_dashboard_demo/view/.... It's a simple R function which generates synthetic data. It could be used to pull much more complex data, generate predictions from an ML model, etc...

Please don't hesitate to reach out if you have any questions!

Post reply on HN