A better alternative to your simple a/b test would be the multi-armed bandit algorithm. This will give you your answer the fastest, though it's slightly harder to wrap your head around check it out here: http://stevehanov.ca/blog/index.php?id=132
A/B Testing, from scratch
11–15 of 15 posts
Re: A/B Testing, from scratch
#12A better alternative to your simple a/b test would be the multi-armed bandit algorithm. This will give you your answer the fastest, though it's slightly harder to wrap your head around check it out here: http://stevehanov.ca/blog/index.php?id=132
I liked the article because it's really fun to stick with something like that, wrap my head around the math, and enjoy the exploration but the multi armed bandit algorithm was supremely satisfying to read. Simple, intuitive, logical, and with easy to tweak parameters. It doesn't get much better than that.
Re: A/B Testing, from scratch
#13Earlier quoted context omitted.
btilly where does this approach come from?
My scratchbook. :-) I keep meaning to write it up. Mathematically it is similar http://www.evanmiller.org/sequential-ab-testing.html but with the difference that Evan produced a stopping rule that is a 95% confidence interval, while mine is stopping at 95% confidence that you won't come to a different conclusion by the end of the test. Otherwise the idea of the analysis is the same. (Note, under the null hypothesis r…
Please do write it up as I would certainly be interested in reading it.
Re: A/B Testing, from scratch
#14There's a possible big language barrier here from "urn". I guessed it was an acronym, wikipedia[1] seemed like no help until I realized that "urn" if for an actual vessel-like object (called an urn ) so I could find the correct article[2]. Which turns out to be such a simple concept, but maybe a note could help future international readers. [1] https://en.wikipedia.org/wiki/Urn_%28disambiguation%29 [2] https://en.wik…
> I guessed it was an acronym
If it were an acronym, it would likely be capitalized as URN, similar to ASAP or GNU.
> until I realized that "urn" if for an actual vessel-like object
A dictionary search seems like it would have helped here instead of Wikipedia.
> Which turns out to be such a simple concept, but maybe a note could help future international readers.
I'm not sure how native speakers in any language could determine whether a given word would be problematic for international readers. There's no inherent ambiguity in the word "urn" in this case for native speakers.
Do you have any ideas, or are you recommending a change for this particular article?
Personally, I'd recommend that the author change "urn" to something more common, like "jar" or "bottle." Since the type of the container itself isn't even germane to the discussion, "box" or "container" might be clearer still.
Re: A/B Testing, from scratch
#15There's a possible big language barrier here from "urn". I guessed it was an acronym, wikipedia[1] seemed like no help until I realized that "urn" if for an actual vessel-like object (called an urn ) so I could find the correct article[2]. Which turns out to be such a simple concept, but maybe a note could help future international readers. [1] https://en.wikipedia.org/wiki/Urn_%28disambiguation%29 [2] https://en.wik…
> I guessed it was an acronym If it were an acronym, it would likely be capitalized as URN, similar to ASAP or GNU. > until I realized that "urn" if for an actual vessel-like object A dictionary search seems like it would have helped here instead of Wikipedia. > Which turns out to be such a simple concept, but maybe a note could help future international readers. I'm not sure how native speakers in any language could…
I've met many people who were learning Spanish and now I get a "feeling" when something could be confusing, but I'm not sure of any general advice for determining it otherwise. It's even more difficult when it's a particular field like Statistics. In retrospective this is something I could have guessed normally. Just some ideas to answer in a more "general" way from my point of view:
- "Statisticians love urns and, guess what, our problem can be modeled as an extraction from two different urns."
Something that might help is noting that it's a typical problem like this:
+ "Statisticians love urns and, guess what, our problem can be modeled as an typical extraction from two different urns."
Note the addition of "typical". Just wording like that would give me a hint that it's something that I'm unaware of. It's just like talking about any other field. Compare these:
- The two prisoners is a case widely known for its... (what two prisoners? were they in the news? it could be anything and seems non-googleable)
+ The typical two prisoners problem is widely known for its... (oh, it's a reference to a specific, famous problem, google gives a quick match)