Earlier quoted context omitted.
> In a randomized controlled trial you either find a significant difference in your metrics, or you don't. There's no other option. This is a poor way of thinking about statistics. Whether you reject or not a sharp null hypothesis doesn't give you much information (See for example: https://www.gwern.net/Everything ). Failing to reject in particular, can be compatible with a wide range of effects. >In all cases where…
> With enough data, there could totally have been a tight range around no effect or a small effect. This is not what we got here though. What the trial showed was a small effect with a wide uncertainty on a big sample. We cannot distinguish this from zero. Again, could the observed effect be significant with a larger trial? Sure. But that's always true for a negative result. The objection carries no information.
Sure, this is true, it's one of the reasons why results being significant or not is not very relevant. At some point you want to move towards whether the effect size is in a clinically relevant range or not.
>The objection carries no information.
Inasmuch as something like a confidence interval provides an idea of the range of the effect size, more data does carry more information. I know it's complicated to do this analysis properly with prediction intervals and such, but you have no choice if you want to be able to make good decisions with your data. A wide range estimate that doesn't allow you to make good clinical decisions is not useful.
For clinical purposes, I would even have been more confortable treating an significant but small effect in support of the "let's not test" scenario, than this wide range where the effect could be large and positive or negative on the other side and we just don't know. Significant doesn't automatically mean "do the test" and vice versa. Effect size matters! A non-significant result because of a wide interval just doesn't tell you much useful information.