One very important thing to know about the 0.05 threshold, and which I did not found in this thread, is that the ideal p-value for a problem is a function of the number of samples (and the effect size but it as a lesser impact). 0.05 is way to stringent if you have 10 samples and way too lenient if you have 1 million samples. But, by force of convention, everyone is using 0.05 (a value suggested by Fischer when basic…
It is not a function of only the number of samples. It is also a function of how costly are false positives (type I error) and false negatives (type II error). That is (from my understanding) the paper's main point. But then you have to be able to calculate how costly is a type I and a type II error! That's seems a relatively straightforward question for a business (for example in A/B testing), but how do you measure…
The strenght here is that you get rid of an arbitrary decision (p-value) and instead use quantities that can be measured and critiqued by a skeptical reviewer.
But, in my experience with small data, the impact of the size of the dataset dwarfed the impact of the cost/probability.