Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lines of code that beat A/B testing (2012)

stevehanov.ca

Post reply on HN
Print view View original

141–150 of 180 posts

Re: Lines of code that beat A/B testing (2012)

#141

Post by DanielHB »

dkarl wrote: post #83
throwup238 wrote: post #67

Earlier quoted context omitted.

> In short, not many people want to funnel users through N code paths with slightly different behaviors, because not many people have a ton of users, a ton of engineering capacity, and a ton of potential upside from marginal improvements. I’ve been in companies that have tried dozens if not hundreds of A/B tests with zero statistically significant results. I figure by the law of probabilities they would have gotten a…

> I’ve been in companies that have tried dozens if not hundreds of A/B tests with zero statistically significant results. What I've seen in practice is that some places trust their designers' decisions and only deploy A/B tests when competent people disagree, or there's no clear, sound reason to choose one design over another. Surprise surprise, those alternatives almost always test very close to each other! Other pl…

In paid SaaS B2B A/B testing is usually a very good idea for use acquisition flow and onboarding, but not in the actual product per se.

Once the user has committed to paying they probably will put up with whatever annoyance you put in their way, also if they are paying if something is _really_ annoying they often contact the SaaS people.

Most SaaS don't really care that much about "engagement" metrics (ie keeping users IN the product). These are the kinda of metrics are are the easiest to see move.

In fact most people want a product they can get in and out ASAP and move on with their lives.

Re: Lines of code that beat A/B testing (2012)

#143

Post by legendofbrando »

mikepurvis wrote: post #86
xp84 wrote: post #84

Earlier quoted context omitted.

> politically derisks the change, by tying it's deployment to rigorous testing that proves it at least does no harm to the existing process before applying it to all users. I just want to drop here the anecdata that I've worked for a total of about 10 years in startups that proudly call themselves "data-driven" and which worshipped "A/B testing." One of them hired a data science team which actually did some decently…

At a small enough scale, gut feelings can be totally reasonable; taste is important and I'd rather follow an opinionated leader with good taste than someone who sits on their hands waiting for "the data". Anyway, your investors want you to move quickly because they're A/B testing you for surviveability against everything else in their portfolio. The worst is surely when management make the investments in rigor but th…

Huge plus one to this. We undervalue when to bet on data and when to be comfortable with gut.

Re: Lines of code that beat A/B testing (2012)

#145

Post by dkarl »

DanielHB wrote: post #141
dkarl wrote: post #83

Earlier quoted context omitted.

> I’ve been in companies that have tried dozens if not hundreds of A/B tests with zero statistically significant results. What I've seen in practice is that some places trust their designers' decisions and only deploy A/B tests when competent people disagree, or there's no clear, sound reason to choose one design over another. Surprise surprise, those alternatives almost always test very close to each other! Other pl…

In paid SaaS B2B A/B testing is usually a very good idea for use acquisition flow and onboarding, but not in the actual product per se. Once the user has committed to paying they probably will put up with whatever annoyance you put in their way, also if they are paying if something is _really_ annoying they often contact the SaaS people. Most SaaS don't really care that much about "engagement" metrics (ie keeping use…

Many SaaS companies care about engagement metrics, especially if they have to sell the product, like their revenue depends on salespeople convincing customers to renew or upgrade their licenses at a certain level for so many seats at $x/year.

For example, I worked on a new feature for a product, and the engagement metrics showed a big increase in engagement by several customers' users, and showed that their users were not only using our software more but also doing their work much faster than before. We used that to justify raising our prices -- customers were satisfied with the product before, at the previous rates, and we could prove that we had just made it significantly more useful.

I know of at least one case where we shared engagement data with a power user at a customer who didn't have purchase authority but was able to join it with their internal data to show that use of our software correlated with increased customer satisfaction scores. They took that data to their boss, who immediately bought more seats and scheduled user training for all of their workers who weren't using our software.

We also used engagement data to convince customers not to cancel. A lot of times people don't know what's going on in their own company. They want to cancel because they think nobody is using the software, and it's important to be able to tell them how many daily and hourly users they have on average. You can also give them a list of the most active users and encourage them to reach out and ask what the software does for them and what the impact would be of cancelling.

Re: Lines of code that beat A/B testing (2012)

#146

Post by blagie »

bartread wrote: post #76
blagie wrote: post #64

Earlier quoted context omitted.

"Easy to implement" is a good reason to use bubble sort too. In a normal universe, you just import a different library, so both are the same amount of work to implement. Multiarmed bandit seems theoretically pretty, but it's rarely worth it. The complexity isn't the numerical algorithm but state management. * Most AB tests can be as simple as a client-side random() and a log file. * Multiarmed bandit means you need a…

Sorry but bubble sort is a terrible example here. You implement a more difficult sorting algorithm, like quicksort, because the benefits of doing so, versus using bubble sort, are in many cases huge. I.e., the juice is worth the squeeze. Whereas the comment you’re responding to is rightly pointing out that for most orgs, the marginal gains of using an approach more complex than Epsilon greedy probably aren’t worth it…

You can use FFT, if you prefer. There's no reason to not use optimized numerical code, since it's just a different import.

The difference in performance is smaller and the difference in complexity is much greater. Optimized FFTs are... hairy. But now that someone wrote them, free.

Re: Lines of code that beat A/B testing (2012)

#147

Post by abetusk »

sweezyjeezy wrote: post #80

One of the assumptions of vanilla multi-armed bandits is that the underlying reward rates are fixed. It's not valid to assume that in a lot of cases, including e-commerce. The author is dismissive and hard wavy about this and having worked in in e-commerce SaaS I'd be a bit more cautious. Imagine that you are running MAB on an website with a control/treatment variant. After a bit you end up sampling the treatment a l…

One of the other comment threads has a link to a James LeDoux post about MAB with EG, UCB1, BUCB and EXP3, with EXP3, from what I've seen, marketed as an "adversarial" MAB method [0] [1].

I found a post [2] of doing some very rudimentary testing on EXP3 against UCB to see if it performs better in what could be considered an adversarial environment. From what I can tell, it didn't perform all that well.

Do you, or anyone else, have an actual use case for when EXP3 performs better than any of the standard alternatives (UCB, TS, EG)? Do you have experience with running MAB in adversarial environments? Have you found EXP3 performs well?

[0] https://news.ycombinator.com/item?id=42650954#42686404

[1] https://jamesrledoux.com/algorithms/bandit-algorithms-epsilo...

[2] https://www.jeremykun.com/2013/11/08/adversarial-bandits-and...

Re: Lines of code that beat A/B testing (2012)

#148

Post by Dylan16807 »

zeroCalories wrote: post #109
hamandcheese wrote: post #99

Earlier quoted context omitted.

Is it a waste? You proved the change wasn't harmful.

Statistically insignificant means you didn't prove anything by usual standards. I do agree that it's not a waste, as knowing that you have a 70% chance that you're going in the right direction is better than nothing. The 2 sigma crowd can be both too pessimistic and not pessimistic enough.

Statistical significance depends on what you're trying to prove. Looking for substantial harm is a lot easier than figuring out which option is simply better, depending on what level of substantial you're looking for.

If your error bar for some change goes from negative 4 percent to positive 6 percent, it may or may not be better, but it's safe to switch to.

Re: Lines of code that beat A/B testing (2012)

#149

Post by closewith »

khafra wrote: post #138
closewith wrote: post #133

Earlier quoted context omitted.

In all cases, data-driven means we establish context for our gut decisions. In the end, it's always a judgement call.

Robin Hanson recently related the story of a firm which actually made data-driven decisions, back in the early 80's: https://www.overcomingbias.com/p/hail-jeffrey-wernick It went really well, and then nobody ever tried it again.

That's a counter-example. The prediction markets were used to inform gut feelings, not controlling the company, and in the end, when he wanted to stop, he stopped following the markets and shut the company.

Re: Lines of code that beat A/B testing (2012)

#150

Post by jacob019 »

hruk wrote: post #102
taion wrote: post #50

The problem with this approach is that it requires the system doing randomization to be aware of the rewards. That doesn't make a lot of sense architecturally – the rewards you care about often relate to how the user engages with your product, and you would generally expect those to be collected via some offline analytics system that is disjoint from your online serving system. Additionally, doing randomization on a…

I don't know, all of these are pretty surmountable. We've done dynamic pricing with contextual multi-armed bandits, in which each context gets a single decision per time block and gross profit is summed up at the end of each block and used to reward the agent. That being said, I agree that MABs are poor for experimentation (they produce biased estimates that depend on somewhat hard-to-quantify properties of your poli…

Hey, I'd love to hear more about dynamic pricing with contextual multi-armed bandits. If you're willing to share your experience, you can find my email on my profile.
Post reply on HN