20 lines of code that beat A/B testing (2012) • Hacker News Board

Re: 20 lines of code that beat A/B testing (2012)

#111

Post by orasis » Wed, Apr 06, 2016, 4:30 PM UTC

kens wrote: post #101

orasis wrote: post #76

Here's what everyone is missing. Don't use bandits to A/B test UI elements, use them to optimize your content / mobile game levels. My app, 7 Second Meditation, is solid 5 stars, 100+ reviews because I use bandits to optimize my content. By having the system automatically separate the wheat from the chaff, I am free to just spew out content regardless of its quality. This allows me to let go of perfectionism and just…

I tried to find the original source of the quality-vs-quantity pottery class story a while back. I think it originates in the book "Art and Fear" but in that book it reads like a parable rather than a factual event. I'm highly suspicious of whether this event actually happened. Anyone have solid evidence?

It was featured in "Thinking Fast and Slow", and those authors seem quite academically rigorous.

Re: 20 lines of code that beat A/B testing (2012)

#112

Post by orasis » Wed, Apr 06, 2016, 4:32 PM UTC

wlesieutre wrote: post #99

orasis wrote: post #76

Here's what everyone is missing. Don't use bandits to A/B test UI elements, use them to optimize your content / mobile game levels. My app, 7 Second Meditation, is solid 5 stars, 100+ reviews because I use bandits to optimize my content. By having the system automatically separate the wheat from the chaff, I am free to just spew out content regardless of its quality. This allows me to let go of perfectionism and just…

> The first group would have the entirety of their grade based on the creativity of a single piece they submit. The second group was graded on only the total number of pounds of clay they threw. I feel like that works partly because an important part of practice the feedback loop between continually practicing and having a sense of whether you did well or not. Your strategy of not evaluating your own work sounds a bi…

I get the same effect using bandits. I practice my craft by spewing out volume rather than focusing on quality. The penalty I pay for the bad content is negligible because the bayesian bandits cull them very quickly. I am learning and getting better, but it is because I am not paralyzed by perfectionism.

Re: 20 lines of code that beat A/B testing (2012)

#113

Post by orasis » Wed, Apr 06, 2016, 4:36 PM UTC

mrspeaker wrote: post #84

orasis wrote: post #76

Here's what everyone is missing. Don't use bandits to A/B test UI elements, use them to optimize your content / mobile game levels. My app, 7 Second Meditation, is solid 5 stars, 100+ reviews because I use bandits to optimize my content. By having the system automatically separate the wheat from the chaff, I am free to just spew out content regardless of its quality. This allows me to let go of perfectionism and just…

I don't understand what you mean by "use them to optimize your content" - how are you doing that with your app? Are you serving different messages to different groups of people? How are you grouping/testing/rating them?

My bandit system generates an ordered list of the content for each individual user. I then will track if the user came back tomorrow or churned. Yes, they may churn due to many other factors, but the signal of the content itself is strong enough.

In the past I have used the share rate to optimize, but I've realized that retention is more important.

Re: 20 lines of code that beat A/B testing (2012)

#114

Post by trjordan » Wed, Apr 06, 2016, 4:36 PM UTC

sixtypoundhound wrote: post #100

Novices also tend to gravitate towards "end-game" business metrics which have a lot more inherent variation than simple operational indicators. For example - optimizing a content site for AdSense; many folks would gravitate to AdSense $$ as the target metric, which is admittedly an intuitive solution (since that's how you're ultimately getting paid). But if you think about it.... AdSense Revenue => (1 - Bounce Rate)…

Isn't that a good argument for using $$ as the metric to optimize for? If you're going to get wiped out by variations in behavior because highly-retargeted legal clicks are worth 500x more than mobile clicks, isn't that an important variable?

The problem I frequently wonder about is that you have to assume independence about the stable variables to be comfortable testing them. In reality, the bounce rate of a people who make you lots of money is probably driven by different factors than the bounce rate of the overall population.

I guess what you should really do is optimize the bounce rate / pages per visit / etc. for just the population of people that could make you money, but you don't typically have access to that information.

Re: 20 lines of code that beat A/B testing (2012)

#115

Post by orasis » Wed, Apr 06, 2016, 4:38 PM UTC

searine wrote: post #97

orasis wrote: post #76

Here's what everyone is missing. Don't use bandits to A/B test UI elements, use them to optimize your content / mobile game levels. My app, 7 Second Meditation, is solid 5 stars, 100+ reviews because I use bandits to optimize my content. By having the system automatically separate the wheat from the chaff, I am free to just spew out content regardless of its quality. This allows me to let go of perfectionism and just…

>I am free to just spew out content regardless of its quality. Oh, that's a great goal to have...

By focusing on volume, I get quality as a side effect. I'm very proud of the the app I've created and the feedback I get from my users:

"This app gives a text reminder to do what everyone wants to do: relax, love one's self and others, and bring peace and light into the world. I smile when the alert message appears and feel grateful to the makers of this app for creating a pleasant, quick way to meditate at the most stressful point of my day. I have recommended it to many people"

https://itunes.apple.com/us/app/7-second-meditation-daily/id...

Re: 20 lines of code that beat A/B testing (2012)

#116

Post by squeaky-clean » Wed, Apr 06, 2016, 4:38 PM UTC

searine wrote: post #97

orasis wrote: post #76

Here's what everyone is missing. Don't use bandits to A/B test UI elements, use them to optimize your content / mobile game levels. My app, 7 Second Meditation, is solid 5 stars, 100+ reviews because I use bandits to optimize my content. By having the system automatically separate the wheat from the chaff, I am free to just spew out content regardless of its quality. This allows me to let go of perfectionism and just…

>I am free to just spew out content regardless of its quality. Oh, that's a great goal to have...

I actually think that's a great idea, because then the users and algorithm decide what's good for you, and it will hide the low quality content automatically.

You have to do that sort of "curation" anyways anytime you make something. You have to continually decide whether it's worth it to keep working on something, and then decide if it's good enough to release. People tend to be pretty bad judges of this (especially for more creative tasks. There are many examples of someone's most famous song/book/painting not being their own favorite, or even being their least favorite!).

So why not relieve some of that stress, and let the users pick what they really like, instead of you guessing for them?

Re: 20 lines of code that beat A/B testing (2012)

#117

Post by mangeletti » Wed, Apr 06, 2016, 4:39 PM UTC

ced wrote: post #94

rco8786 wrote: post #63

Earlier quoted context omitted.

You really should question the statistical significance of those numbers.

I'll add my own Bayesian analysis to the fray. Assuming a binomial, in Julia: using Distributions b_old = Beta(66+1, 6392-66+1) b_yel = Beta(83+1, 6362-83+1) N = 1000000 # Sample from both distributions, count the fraction of samples that are better sum(rand(b_old, N) .> rand(b_yel, N)) / N This yields 7.7% chance that the old one is better than "yellow". It's fascinating to see how we can get such different answers…

Thanks for this analysis.

I did an A/B test on an older framework which didn't automate statistical significance at all, but the website was getting more than 2000-3000 orders per day, so after a single week we had enough data to determine that sales had increased by 36% (reduced the page's load time by almost half, changed the checkout to use Ajax, and a few other small changes.) without the need to quantify things. In fact, at the time, I didn't even know what "statistical significance" was... not that I know too much more about statistics now than I did then.

Anyway, in theory all of the exact p values, etc. matter, but in practice, the bottom line is all that matters, because the p values can change in a moment based on something I haven't factored in. That's where, at least for the time being, intuition still plays a great part in being actually correct, which is why we still have people with repeated successes.

Re: 20 lines of code that beat A/B testing (2012)

#118

Post by orasis » Wed, Apr 06, 2016, 4:43 PM UTC

trjordan wrote: post #114

sixtypoundhound wrote: post #100

Novices also tend to gravitate towards "end-game" business metrics which have a lot more inherent variation than simple operational indicators. For example - optimizing a content site for AdSense; many folks would gravitate to AdSense $$ as the target metric, which is admittedly an intuitive solution (since that's how you're ultimately getting paid). But if you think about it.... AdSense Revenue => (1 - Bounce Rate)…

Isn't that a good argument for using $$ as the metric to optimize for? If you're going to get wiped out by variations in behavior because highly-retargeted legal clicks are worth 500x more than mobile clicks, isn't that an important variable? The problem I frequently wonder about is that you have to assume independence about the stable variables to be comfortable testing them. In reality, the bounce rate of a people…

$$ can be done in a bandit setting, but the key challenge is that your feedback is highly delayed (maybe weeks or months).

As the parent poster says, its best to focus on individual funnel steps that provide fast feedback, at least initially.

Once the whole funnel is optimized (does this ever happen?), you could start feeding in end-to-end $$ metrics.

Re: 20 lines of code that beat A/B testing (2012)

#119

Post by jedberg » Wed, Apr 06, 2016, 4:47 PM UTC

Just a warning, this isn't a magic bullet to replace all A/B testing. This is great for code that has instant feedback and/or the user will only see once, but for things where the feedback loop is longer or the change is more obvious or longer lasting (like a totally different UI experience), it doesn't work so well.

For example, if your metric of success is that someone retains their monthly membership to your site, it will take a month before you start getting any data at all. At that point, in theory almost all of your users should already be allocated to a test because hopefully they visited (and used) their monthly subscription at least once. So it would be a really bad experience to suddenly reallocate them to another test each month.

Re: 20 lines of code that beat A/B testing (2012)

#120

Post by orasis » Wed, Apr 06, 2016, 4:51 PM UTC

tswartz wrote: post #108

orasis wrote: post #76

Here's what everyone is missing. Don't use bandits to A/B test UI elements, use them to optimize your content / mobile game levels. My app, 7 Second Meditation, is solid 5 stars, 100+ reviews because I use bandits to optimize my content. By having the system automatically separate the wheat from the chaff, I am free to just spew out content regardless of its quality. This allows me to let go of perfectionism and just…

Good point! What do you use to do in-app A/B testing?

I implemented my own that I'll be making available for others soon at improve.ai (the site isn't up yet)