The Importance of Sequential Testing
austinrochford.com
The Importance of Sequential Testing
1–7 of 7 posts
Re: The Importance of Sequential Testing
#2[I'm assuming the author posted this. MidsizeBlowfish has no profile information.]
Good stuff, and just one of the many reasons that most A/B testing practice is flawed. I haven't seen the SPRT used much in practice. I'm not sure why. A quick Googling indicates non-parametric sequential tests have been developed. What about the case with more than two alternatives?
My field is bandit algorithms, which are a generalisation of hypothesis tests. Here there are algorithms like Empirical Bernstein Stopping (http://icml2008.cs.helsinki.fi/papers/523.pdf) which achieve the same early stopping from a different approach. If this is of interest sign up to the course I'm running (http://noelwelsh.com/data/2013/08/16/free-bandit-course-draf...) -- I plan to cover this algorithm soon.
BTW, you have a very formal writing style. I think if you're trying for a more general audience you should loosen up a bit. This ain't a journal paper. :-)
Re: The Importance of Sequential Testing
#3[I'm assuming the author posted this. MidsizeBlowfish has no profile information.] Good stuff, and just one of the many reasons that most A/B testing practice is flawed. I haven't seen the SPRT used much in practice. I'm not sure why. A quick Googling indicates non-parametric sequential tests have been developed. What about the case with more than two alternatives? My field is bandit algorithms, which are a generalis…
Haha, yes, I am the author. I do have a very formal style, an unfortunate product of too many years in grad school.
The SPRT is a nice toy test, but only useful for point hypotheses. There are generalizations to more realistic composite hypotheses, though.
Re: The Importance of Sequential Testing
#4[I'm assuming the author posted this. MidsizeBlowfish has no profile information.] Good stuff, and just one of the many reasons that most A/B testing practice is flawed. I haven't seen the SPRT used much in practice. I'm not sure why. A quick Googling indicates non-parametric sequential tests have been developed. What about the case with more than two alternatives? My field is bandit algorithms, which are a generalis…
Haha, yes, I am the author. I do have a very formal style, an unfortunate product of too many years in grad school. The SPRT is a nice toy test, but only useful for point hypotheses. There are generalizations to more realistic composite hypotheses, though.
It would be interesting to me if you covered the generalizations. This field seems to have evolved entirely separately from the banditry I'm accustomed to.
Re: The Importance of Sequential Testing
#5Earlier quoted context omitted.
Haha, yes, I am the author. I do have a very formal style, an unfortunate product of too many years in grad school. The SPRT is a nice toy test, but only useful for point hypotheses. There are generalizations to more realistic composite hypotheses, though.
It would be interesting to me if you covered the generalizations. This field seems to have evolved entirely separately from the banditry I'm accustomed to.
I'll put a follow up on some other sequential tests into my list of blog-posts-to-do and let you know when it happens.
Re: The Importance of Sequential Testing
#6A heads up to the author: with narrower browser window sizes (e.g. less than ~1024px), the left side menu overlaps the text.
Re: The Importance of Sequential Testing
#7A heads up to the author: with narrower browser window sizes (e.g. less than ~1024px), the left side menu overlaps the text.
Just noticed that on my iPad, will look into it.