Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Surprising Power of Online Experiments (2017)

hbr.org

Post reply on HN
Print view View original

11–20 of 27 posts

Re: The Surprising Power of Online Experiments (2017)

#11

Post by jameslk »

Notably absent from this article is any discussion about multi-armed bandits. A/B testing only leads to finding the more optimal of treatments, where as multi-armed bandit algorithms help find the optimal treatment and exploit it during testing. If profit is involved, it seems obvious that you should be considering both exploring treatments but most importantly exploiting the treatments that currently yields the most profit.

Google has a pretty good FAQ on this:

https://support.google.com/analytics/answer/2847021?hl=en&re...

Re: The Surprising Power of Online Experiments (2017)

#12

Post by DigitalVerse »

schwax wrote: post #5

Anybody have a recommendation for an A/B testing service? We've talked to Optimizely but their pricing was going to come in at the same ballpark as our AWS spend (into the six-figure range), which seems absurd. They charge based on monthly users, but a lot of our traffic consists of organic search bounces. For now we just want to run ~5 experiments per month, want to record events server-side so we can be sure not to…

Split (https://www.split.io/) works pretty well for us

Re: The Surprising Power of Online Experiments (2017)

#13

Post by paulddraper »

jameslk wrote: post #11

Notably absent from this article is any discussion about multi-armed bandits. A/B testing only leads to finding the more optimal of treatments, where as multi-armed bandit algorithms help find the optimal treatment and exploit it during testing. If profit is involved, it seems obvious that you should be considering both exploring treatments but most importantly exploiting the treatments that currently yields the most…

Yep, and GA does multi armed bandit testing.

Definitely the way to go for A/B.

Re: The Surprising Power of Online Experiments (2017)

#14

Post by jdavis703 »

schwax wrote: post #5

Anybody have a recommendation for an A/B testing service? We've talked to Optimizely but their pricing was going to come in at the same ballpark as our AWS spend (into the six-figure range), which seems absurd. They charge based on monthly users, but a lot of our traffic consists of organic search bounces. For now we just want to run ~5 experiments per month, want to record events server-side so we can be sure not to…

At one place I worked we got around Optimizely pricing by only sending down Optimizely code to some small subset of our users — enough to yield statistical significance but not so much to break the bank.

Re: The Surprising Power of Online Experiments (2017)

#15

Post by boto3 »

schwax wrote: post #5

Anybody have a recommendation for an A/B testing service? We've talked to Optimizely but their pricing was going to come in at the same ballpark as our AWS spend (into the six-figure range), which seems absurd. They charge based on monthly users, but a lot of our traffic consists of organic search bounces. For now we just want to run ~5 experiments per month, want to record events server-side so we can be sure not to…

It's not that hard to DIY. You can start with: https://facebook.github.io/planout/

Re: The Surprising Power of Online Experiments (2017)

#18

Post by CharlesW »

skywhopper wrote: post #8

Good in-depth overview but unfortunately doesn’t get into bigger issues of potential ethical concerns of this sort of narrow maximization. At a certain point you end up becoming Facebook or YouTube and vastly amplifying toxic and dangerous content because it generates more comments or more time spent on the site. And even if you believe in pure amoral capitalism, blindly following what your algorithms tell you to do…

Supporting your point, YouTube claims to be backing off its "clicks at any cost" model.

As has been mentioned previously, our business depends on the trust users place in our services to provide reliable, high-quality information. The primary goal of our recommendation systems today is to create a trusted and positive experience for our users. Ensuring these recommendation systems less frequently provide fringe or low-quality disinformation content is a top priority for the company. The YouTube company-wide goal is framed not just as “Growth”, but as “Responsible Growth”.

[1] https://blog.google/documents/33/HowGoogleFightsDisinformati...

Re: The Surprising Power of Online Experiments (2017)

#20

Post by joshuamorton »

jameslk wrote: post #11

Notably absent from this article is any discussion about multi-armed bandits. A/B testing only leads to finding the more optimal of treatments, where as multi-armed bandit algorithms help find the optimal treatment and exploit it during testing. If profit is involved, it seems obvious that you should be considering both exploring treatments but most importantly exploiting the treatments that currently yields the most…

Gonna toot my own horn here for a second, but here[0] is a presentation I did on multi-armed bandits (and specifically Thompson sampling, the tragically underutilized optimal bandit method) to an undergrad ML group. Amusingly, the diagram I use on slide 3 is from a research paper from Microsoft. We had a guest speaker from MS's exp-platform team [1] the prior week, and she had discussed A/B testing, but not touched on bandits, and I felt a need to make this exact point.

[0]: https://gtagency.github.io/2016/experimentation-with-no-ragr...

[1]: https://exp-platform.com/

Post reply on HN