Bayes’ Theorem in the 21st Century (2013) [pdf]

The only sensible "non-informative" prior is Jeffreys' prior. Invariance under reparameterization of the parameter is what I would consider to be a non-negotiable feature of any non-informative prior belief.

To assign a (improper) uniform prior to the variance of a Gaussian distribution is to assign a non-uniform prior to its standard deviation, and vice versa. One can, in certain circumstances, assign priors to be non-inforamative in a particular way, but to be universally non-informative, no, it must be Jeffreys' or nothing at all.

In consideration of the aforementioned, the debate about non-informative Bayesian priors is a relic of 20th century philosophy. The construction of hiearchical causality networks for the purposes of unsupervised learning is the future of Bayesian statistics, and priors in this context are rarely non-informative.

I think this article omits the most important distinction between Bayesian and Frequentist statistics: subjective vs. frequentist interpretations of probability. In my own opinion, neither is "true", they're both just different tools for different purposes.

Bayesian inference is great when you have to make a decision and there are many theorems that illustrate this (for example, the arguments around coherence [1] and the complete class theorems [2]). In fact, Bayesian techniques are often useful for creating estimators with great frequentist properties! However, Bayesian interpretations of probability, and thereby the meaning of Bayesian statements, are inherently tied to the beliefs of an individual. That means that Bayesian statements usually aren't "true" in the objective / non-relative sense that we often expect from science. On the other hand, frequentist statements tend to have more of an objective flavor. The trick is: all our mathematical models have short comings and ways in which they're wrong when applied to any particular situation -- so neither really has a claim to being true.

The frequentist perspective often looks at worst case risk and tends to give a more global understanding of a procedure in terms of "how does this procedure shake out in all reasonably possible scenarios?". So, frequentist methods tend to be a bit more risk-averse which is often useful but can cost you for being to pessimistic. Ultimately, the real win is to know your tools well and to pick the right one for the job.

[1] https://en.wikipedia.org/wiki/Coherence_(philosophical_gambl...

[2] https://projecteuclid.org/euclid.aoms/1177730345

The way I read this, the first two trials did not bring enough evidence towards the drug working, but they did provide some evidence. In the end, all three experiments taken together provide pretty massive evidence that the drug actually works, way below 0.05.

Of course, this all comes crashing down if the first two experiments happen to provide contrary evidence (that is, evidence the drug does not work). This would cancel out the results of the final trial somewhat, and not taking this into account is clearly cheating by publication bias.

It's wrong. Not because the stated probability at the point you stop the experiment is wrong, but because the (stupid) rule is that we'll approve the drug if the probability that the drug is better than the alternative is above some arbitrary threshold value. If you run experiments a fixed number of trials, you'll get a variety of conclusions with different strengths. If you stop as soon as you are in the "barely passing" zone, you'll get a lower number of failing result, a higher number of barely passing results, and none at all that do better than barely passing.

I apologize for repeating myself, I don't think I'm being clear on what I'm trying to say. Let me give an illustrative example: Let's say I've got an 100% fair coin, but I want to trick you into thinking it comes up heads more often than tails. The way I do this is wit a meta-experiment: I will have 1000 trials with up to 1000 flips each, but I stop the experiment whenever I have a majority of heads.

What we expect to find at the end is, I get about the same number of heads and tails in the whole meta-experiment. About 95% of the runs will have more heads than tails, but each of those runs will only have one extra head. The few runs where I did all 1000 flips will be ones where heads never had a majority, so they'll probably have lots of extra tails. Same number of heads and tails over all is the relevant result, 95% of runs had majority heads is bullshit intended to baffle you. Nobody would be fooled by such nonsense, right?