Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing gets misused to juice metrics in the short term

zumsteg.net

Post reply on HN
Print view View original

281–290 of 346 posts

Re: A/B testing gets misused to juice metrics in the short term

#281

Post by im3w1l »

goodside wrote: post #5
causi wrote: post #2

The problem with AB testing is that it's a short-term strategy. For example, if a news site runs AB testing with headlines, they'll find that bullshit clickbait headlines get more pageviews than concise, accurate headlines, but the constant use of clickbait headlines will over time destroy overall traffic to your site. More frustratingly, sites run by smart people tend to fall into a balance where the worst articles…

There’s nothing about AB testing that requires you to use short-term metrics. I used to manage AB tests for online dating sites (OkCupid, Grindr) where subscription revenue is what matters, and the gains of any strategy will take months to materialize. We were well aware that, say, raising prices would yield more short-term revenue at the expense of long-term revenue. That didn’t stop us from testing, it just made th…

Did you A/B test the matching algorithms?

Re: A/B testing gets misused to juice metrics in the short term

#283

Post by lifeplusplus »

A nice way to summarize this article to think about local maxima and global maxima.

A/B testing right now is done on cohort basis and tests are ran for weeks to couple of months. This means where lifetime span of a customer is beyond few weeks and months, it's really not possible to tell if global maximum was missed.

I.e. you increase the number of promotional emails the customers get per week. You do it for 3 weeks and see that customers who got those emails had higher conversion. But you didn't get to see that customers who kept getting those higher number of emails completely unsubscribed after 3 months of pain. But by this time all customers are on higher frequency group so it's hard to tell what would be driving the unsubscriptions.

I'm no expert but here are some solutions:

1. You should have really delayed long running control groups. Preferably going well beyond average duration your customer sticks around. These groups should get onto new things a year after. But even then it'd be not possible to take out WHAT feature is affecting them, because in 1 year main group would have accumulated lot of features. But still something...

2. You should really have lots of secondary KPIs that measure things that affect long term KPIs. Sure conversion is better, but is time spent reading newsletters increasing? Are buyers feeling good about their experience with the brand... some of these KPI are more qualitative and can't be just automated.

what else?

Re: A/B testing gets misused to juice metrics in the short term

#284

Post by DrewADesign »

nemo44x wrote: post #118
jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

A lot of this happens because different managers have different metrics/KPIs they are optimizing around and they all find "good places" to do things to help meet their goals. The secondary effects aren't considered. One managers quest for outperforming their goals comes at the expense of another managers goals.

Having experience design expertise at the executive level can mitigate this. If nobody is advocating for good user experiences, nobody is advocating for the usefulness of your online product as a whole, and it shows.

Re: A/B testing gets misused to juice metrics in the short term

#285

Post by saagarjha »

d23 wrote: post #123
nemo44x wrote: post #118

Earlier quoted context omitted.

A lot of this happens because different managers have different metrics/KPIs they are optimizing around and they all find "good places" to do things to help meet their goals. The secondary effects aren't considered. One managers quest for outperforming their goals comes at the expense of another managers goals.

There was a point a few years ago where you could not see a single piece of user-generated content above the fold on the reddit home page. A bunch of teams had jockeyed for having their little carousels and banners put on top, and of course, metrics were always cited. I left a screenshot in slack and it ended up causing a couple of teams to have to roll back their widgets, but it always baffled me that we were able t…

It seems like every company goes through some version of this. At Twitter there was a channel called #ios-six-bars or something like that that started when an engineer posted a screenshot of the home timeline with six bars of things on it, all from jockeying teams trying to grab a spot on that page: Home, Spaces, all the new features just had to be thrown in the face of somebody who probably just wanted to read some tweets. Discussions were had, product cohesion was brought up, then things went quiet for a bit. Until someone posted a new screenshot a couple months later with seven bars on the screen.

Re: A/B testing gets misused to juice metrics in the short term

#287

Post by saagarjha »

costcofries wrote: post #57
hbn wrote: post #33

I've talked about this on HN but I'll say it again. A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a bi…

"To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. " Your perspective is extremely short-sighted. A/B testing can result in this type of behaviour but that's just poor A/B testing. Good A/B testing focuses on removing distractions from the experience and helping users derive…

Well, then, the reason why people hate A/B testing is because everyone does it wrong. People can write fast software in Python too but the reason it’s known as being a slow language is because it invariably gets used differently.

Re: A/B testing gets misused to juice metrics in the short term

#288

Post by saagarjha »

Lutger wrote: post #170
hbn wrote: post #33

I've talked about this on HN but I'll say it again. A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a bi…

> To me, capitalism (A/B testing) means you don't respect your citizens (users). You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. You just described doing business in todays world. Being a bit more generous towards A/B testing, I would make a counter point: _not_ doing any kind of user testing, of which large scale automated A/…

> Because it means you just assume you know what their experience is like, or worse: you don't even care about it and bother to learn something.

A/B tests are not the only, or even the best, way of collecting user feedback.

Re: A/B testing gets misused to juice metrics in the short term

#289

Post by treis »

josefx wrote: post #111
treis wrote: post #52

Earlier quoted context omitted.

>Instead try to improve the customer experience, make better products, improve customer service. Without a metric to say what is "better" and a method to measure it this is empty advice.

But does AB testing provide the only or even best metric for that? It probably is the cheapest way requiring the least engagement with the lab mice.

AB testing is how you isolate a change and measure the impact. It's the only real way to be able to associate cause and effect. Best you can do otherwise is measure something over time while making changes. You can try and correlate changes with outcomes but it's hard to be sure the change is what drove the outcome.

Re: A/B testing gets misused to juice metrics in the short term

#290

Post by saagarjha »

amluto wrote: post #143

Reading this makes me think of the handful of sites, often targeted at professionals, that highly optimize for the experience of actually buying things. McMaster-Carr comes to mind. Their users shop there over and over, and McMaster wants to keep them. So you can find things for $2 or $2000, shipping prices are inoffensive, customer service is friendly but rarely needed, and there are minimal distractions on the way…

> This is an interesting contrast to Amazon that also makes checkout easy

Hey, would you like Prime with that? Do you know we provide free two-day shipping with Prime? If you sign up for Prime today you can get a $100 discount!

Post reply on HN