31–36 of 36 posts
Building your own bayesian model with something like pymc3 is also a very reasonable approach to take with small data or data with too much variance to detect effects in a timely manner. This also forces you to think about the underlying distributions that generate your data which is an exercise in itself that can yield interesting insights.
Earlier quoted context omitted.
Thanks for factorial design! I'll update the post to the proper nomenclature. The frequentist/bayesian debate is not one I understand well enough to opine - do you have any reading you'd recommend for this topic?
I myself am a rather recent convert to using Bayesian statistic, for the simple reason, that I was trained and have used frequentist statistics extensively in the past and I had no experience using Bayesian statistics. Once you take the time to master the basic tools, it becomes quite straightforward to use. I am currently away from my computer and resources, which makes it difficult to suggest them. As a somewhat sh…
There's an argument to be made that, so long as your testing fully encompasses all visitors to your site, you aren't sampling the population, you're fully observing it, and statistical significance is irrelevant.
That argument is missing that you are using past users’ behaviour as representative of future users’ preferences. You are not sampling marbles in a jar, but making a lot of assumptions, notably about continuity.
There's an argument to be made that, so long as your testing fully encompasses all visitors to your site, you aren't sampling the population, you're fully observing it, and statistical significance is irrelevant.
Sites are always getting new visitors, losing old ones and the ones they’ve observed return irregularly (or commonly, or somewhere in between). So it’s not realistic to assume a given sample of visitors is the population.
It's not unreasonable to assume it's a sample, I just don't think it's worth getting paralyzed by worrying about whether or not you have power, or getting into hacky tricks to try to fix it.
...but most power calculations are also sort of bullshit.
Earlier quoted context omitted.
I myself am a rather recent convert to using Bayesian statistic, for the simple reason, that I was trained and have used frequentist statistics extensively in the past and I had no experience using Bayesian statistics. Once you take the time to master the basic tools, it becomes quite straightforward to use. I am currently away from my computer and resources, which makes it difficult to suggest them. As a somewhat sh…
Any similar GitHub for Python you recommend?
Earlier quoted context omitted.
Sites are always getting new visitors, losing old ones and the ones they’ve observed return irregularly (or commonly, or somewhere in between). So it’s not realistic to assume a given sample of visitors is the population.
If future users are markedly different than past users, a p-value isn't going to help you here. It's not unreasonable to assume it's a sample, I just don't think it's worth getting paralyzed by worrying about whether or not you have power, or getting into hacky tricks to try to fix it. ...but most power calculations are also sort of bullshit.