Any movement that lacks explicit acknowledgement of statistical significance is worrisome to me. I think for many -- myself included -- it is temptingly easy to attribute causation or project a bias onto data collected like this (eg "I'm in a bad mood today because I only slept for 5 and a half hours last night"). I would be interested in analyzing the data after I had a large pool and a decent idea of shape, center,…
I'm not necessarily too worried about that. Statistical significance is the wrong concept for QS and its use is essentially cargo cult statistics.
Leaving aside the profound conceptual and applied problems with null-hypothesis testing ( http://lesswrong.com/lw/g13/against_nhst/ ), QS is much closer to cost-benefit analysis where effect sizes and costs are the critical variables, not alpha. We don't care about testing some intervention and not making the completely arbitrary cutoff of 0.05 (which doesn't mean anything about the truth of the hypothesis in the first place)! We care whether the intervention make a large impact on the variable in question and how expensive the intervention was; if, say, the intervention is an expensive supplement that costs hundreds of dollars a year, we want a higher burden of proof than if the intervention is something free (like taking your vitamin D supplement in the morning rather than evening) or something we should be doing anyway (like exercise).
Far* more worrisome than QS's failure to run t-tests and ritually chant 'we calculate a p-value of * If you are wondering why anyone would care about my opinion, I've been self-experimenting for years and have a little bit of insight into the matter; see http://www.gwern.net/Zeo http://www.gwern.net/Nootropics and http://www.gwern.net/Weather