20 lines of code that beat A/B testing (2012) • Hacker News Board

Re: 20 lines of code that beat A/B testing (2012)

#81

Post by visarga » Wed, Apr 06, 2016, 2:14 PM UTC

PaulHoule wrote: post #64

It is really funny how communities don't talk. For instance, A/B testing with a 50-50 split has been baked into "business rules" framework from about as along ago as the Multi-armed bandit has been around, but nobody in that community has ever heard of the multi-armed bandit, and in the meantime, machine learning people are celebrating about the performance of NLP systems they build that are far worse than rule-based…

> in the meantime, machine learning people are celebrating about the performance of NLP systems they build that are far worse than rule-based systems people were using in industry and government 15 years ago

That was not my experience. I have been researching chat bots and it would seem I can hardly find one implemented with machine learning but instead almost all are rules based. I was quite disappointed. ML for NLP is just gearing up.

Re: 20 lines of code that beat A/B testing (2012)

#82

Post by paulsutter » Wed, Apr 06, 2016, 2:23 PM UTC

Author is confusing the map and the territory.

A/B testing is comparing the performance of alternatives. Epsilon-greedy is a (good) way to do that. it's better than the most common approach, but you're still testing A against B.

Re: 20 lines of code that beat A/B testing (2012)

#83

Post by katafalkas » Wed, Apr 06, 2016, 2:26 PM UTC

orasis wrote: post #74

I've been using bandits in my mobile apps for the last 3 years and am now packaging them as a saas REST API. Message me if you want an early look.

Hey, I was working on the same thing. Wanna talk about it ?

Re: 20 lines of code that beat A/B testing (2012)

#84

Post by mrspeaker » Wed, Apr 06, 2016, 2:43 PM UTC

orasis wrote: post #76

Here's what everyone is missing. Don't use bandits to A/B test UI elements, use them to optimize your content / mobile game levels. My app, 7 Second Meditation, is solid 5 stars, 100+ reviews because I use bandits to optimize my content. By having the system automatically separate the wheat from the chaff, I am free to just spew out content regardless of its quality. This allows me to let go of perfectionism and just…

I don't understand what you mean by "use them to optimize your content" - how are you doing that with your app? Are you serving different messages to different groups of people? How are you grouping/testing/rating them?

Re: 20 lines of code that beat A/B testing (2012)

#85

Post by zardeh » Wed, Apr 06, 2016, 2:47 PM UTC

metafunctor wrote: post #26

3dfan wrote: post #12

10% of the time, we choose a lever at random. The other 90% of the time, we choose the lever that has the highest expectation of rewards. There is a problem with strategies that change the distribution over time: Other factors change over time too. For example let's say over time the percentage of your traffic that comes from search engines increases. And this traffic converts better then your other traffic. And let'…

This is why you should segment traffic and run separate tests for each segment, whether you're using an A/B testing or a multi-armed banding algorithm.

Indeed, otherwise you might be guilty of have Simpson's paradox either way.

Re: 20 lines of code that beat A/B testing (2012)

#86

Post by adevine » Wed, Apr 06, 2016, 2:51 PM UTC

mangeletti wrote: post #68

jefftk wrote: post #65

Earlier quoted context omitted.

Specifically, I get p=0.15 for those numbers. And that's assuming that they're based on running the test for a fixed amount of time instead of stopping once they look good. So: probably better, but should have run it longer.

The lady who owned the site ended up changing ecommerce platforms before the test could complete, due to issues with the software. Sadly, her "add to cart" buttons on her new site are again styled to her brand... I wanted Optimizely to say it was 100% significant for a full week of it running before I ended the test, but the chart was interesting to me, because the conversion rate difference between the two remained…

> I wanted Optimizely to say it was 100% significant

That's not how statistical significance works...

Re: 20 lines of code that beat A/B testing (2012)

#87

Post by zardeh » Wed, Apr 06, 2016, 2:51 PM UTC

What I love about this example is that its the same algorithm applied in two completely different areas. This algorithm (or variants of it) can be used in place of A/B testing. The same algorithm can be applied to game playing and you get Monte Carlo Tree search, the basis of AlphaGo.

Re: 20 lines of code that beat A/B testing (2012)

#88

Post by conductrics » Wed, Apr 06, 2016, 2:55 PM UTC

When thinking about what type of approach is best, first think about the nature of the problem. First is it a real optimization problem, IOW are you more concerned with learning an optimal controller for your marketing application? If so then ask: 1) Is the problem/information perishable - for example Perishable: picking headlines for News articles; Not Perishable: Site redesign. If Perishable then Bandit might give you real returns. 2) Complexity: Are you using covariates (contextual bandits, Reinforcement learning with function approximation) or not. If you are, then you might want your targeting model to serve up best the predicted options in subspaces (frequent user types) that it has more experiences in and for it to explore more in less frequently visited areas (less common user types). 3) Scale/Automation: You have tons of transactional decision problems, and it just doesn't scale to have people running many AB Tests.

Often it is a mix - you might use a bandit approach with your predictive targeting, but you also should A/B tests the impact of your targeting model approach vs a current default and/or a random draw. see slides 59-65: http://www.slideshare.net/mgershoff/predictive-analytics-bro...

For a quick bandit overview check out: http://www.slideshare.net/mgershoff/conductrics-bandit-basic...

Re: 20 lines of code that beat A/B testing (2012)

#89

Post by occamrazor » Wed, Apr 06, 2016, 2:58 PM UTC

spiderfarmer wrote: post #17

I did a lot of A/B testing, but I think the examples that are used in a lot of articles about A/B testing are weird. For small changes like change the color / appearance of a button, the difference in conversion rate is not measurable. Maybe if you can test with traffic in the range of >100K unique visitors (from the same sources), you can say with confidence which button performed better. But how many websites / app…

People read about Google testing different shades of blue on their home page and want to do the same on their website.

They tend to forget that Google has a billion unique users per day, and their website a few hundreds.

Re: 20 lines of code that beat A/B testing (2012)

#90

Post by andreasklinger » Wed, Apr 06, 2016, 3:10 PM UTC

mrspeaker wrote: post #84

orasis wrote: post #76

Here's what everyone is missing. Don't use bandits to A/B test UI elements, use them to optimize your content / mobile game levels. My app, 7 Second Meditation, is solid 5 stars, 100+ reviews because I use bandits to optimize my content. By having the system automatically separate the wheat from the chaff, I am free to just spew out content regardless of its quality. This allows me to let go of perfectionism and just…

I don't understand what you mean by "use them to optimize your content" - how are you doing that with your app? Are you serving different messages to different groups of people? How are you grouping/testing/rating them?

i dont know the app but i assume he has a core action (-> which he iterates/churns out) and evaluates if people act (-> used as rating)

eg (guesses here)

new notification types "think for 9 seconds about a loved one" "what was your favorite vacation, go there"

if people act -> reward and reuse if not -> reuse less (only by random chance)

alternative to notification could be "completed medidation screen"