Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing gets misused to juice metrics in the short term

zumsteg.net

Post reply on HN
Print view View original

11–20 of 346 posts

Re: A/B testing gets misused to juice metrics in the short term

#11

Post by mschuster91 »

smeyer wrote: post #4
Kaotique wrote: post #3

AB testing shows zero respect to your customers. It is the equivalent of testing your theories on lab rats. Instead try to improve the customer experience, make better products, improve customer service.

>Instead try to improve the customer experience AB testing can be (although isn't always) used to improve the customer experience. Assuming you know exactly what will make the customer experience best without actually testing it can also lead to a worse experience.

> Assuming you know exactly what will make the customer experience best without actually testing it can also lead to a worse experience.

For that you usually hire a market research company or do what they will do: take an interviewer, two cameras (one front-face, one top-hands) and hire an as-diverse-as-possible pool of test candidates that you then put through whatever workflow optimization you want to do. Then afterwards, you interview them - side benefit, you can get really interesting general side knowledge that you'd never gain from a dumbass A/B scheme: is your font style/color scheme legible, can the site be used by colorblind people, are there stock photo choices that give off stereotypical vibes...

It's real fun and a worthwhile experience for everyone involved.

Re: A/B testing gets misused to juice metrics in the short term

#12

Post by tsimionescu »

goodside wrote: post #5
causi wrote: post #2

The problem with AB testing is that it's a short-term strategy. For example, if a news site runs AB testing with headlines, they'll find that bullshit clickbait headlines get more pageviews than concise, accurate headlines, but the constant use of clickbait headlines will over time destroy overall traffic to your site. More frustratingly, sites run by smart people tend to fall into a balance where the worst articles…

There’s nothing about AB testing that requires you to use short-term metrics. I used to manage AB tests for online dating sites (OkCupid, Grindr) where subscription revenue is what matters, and the gains of any strategy will take months to materialize. We were well aware that, say, raising prices would yield more short-term revenue at the expense of long-term revenue. That didn’t stop us from testing, it just made th…

Sure, but in many cases, such as the example given by GP, long-term AB testing is hard or almost impossible. For the testing to have validity, you need the A and B cohorts to be stable, and have little or no overlap, and that is hard for long time spans for anything that is not account based (and somewhat dangerous even for account-based things, as people will almost certainly start to notice that they are getting a different experience than their peers, which may upset them).

Re: A/B testing gets misused to juice metrics in the short term

#13

Post by dr_dshiv »

causi wrote: post #2

The problem with AB testing is that it's a short-term strategy. For example, if a news site runs AB testing with headlines, they'll find that bullshit clickbait headlines get more pageviews than concise, accurate headlines, but the constant use of clickbait headlines will over time destroy overall traffic to your site. More frustratingly, sites run by smart people tend to fall into a balance where the worst articles…

I’m a huge fan of metrics. Huge! But they are worthless when not combined with qualitative experience. AB testing needs to be combined with human-centered “actually talking to people about their experiences.” Otherwise, you drift and the metrics no longer match the objective.

Re: A/B testing gets misused to juice metrics in the short term

#14

Post by jollybean »

Eddy_Viscosity2 wrote: post #6

I'd say this is a sub-category of the saying: "There is nothing in this world that an MBA can't and won't make worse". It happens when people change perspectives from building and sustaining businesses to exploiting and squeezing every employee, supplier, and customer for the last drop.

This is just casual bigotry.

Re: A/B testing gets misused to juice metrics in the short term

#15

Post by emsixteen »

mschuster91 wrote: post #7

My biggest issue with A/B testing isn't even mentioned here... gaslighting your customers is absolutely not OK . Particularly with older people, the constant "where the f..k did Outlook now put feature XYZ" (in the case that comes to my mind, the CC bar which used to be tab-reachable, now you have to tab+space or manually click on the tiny gray "cc" letters) onslaught is just absurd. When you change how applications…

> it's rare that companies offer actual, live-human support these days anyway.

This is one of my most intense frustrations in the modern age. Complete and utter disrespect for your customers' time and knowledge.

Re: A/B testing gets misused to juice metrics in the short term

#16

Post by ssharp »

throwaway290 wrote: post #8
smeyer wrote: post #4

Earlier quoted context omitted.

>Instead try to improve the customer experience AB testing can be (although isn't always) used to improve the customer experience. Assuming you know exactly what will make the customer experience best without actually testing it can also lead to a worse experience.

A/B testing helps you maximize a metric, not make customer experience better. Those are different things.

Better customer experiences often times lead to increased metrics. They are not totally different things.

Re: A/B testing gets misused to juice metrics in the short term

#17

Post by Akronymus »

mschuster91 wrote: post #7

My biggest issue with A/B testing isn't even mentioned here... gaslighting your customers is absolutely not OK . Particularly with older people, the constant "where the f..k did Outlook now put feature XYZ" (in the case that comes to my mind, the CC bar which used to be tab-reachable, now you have to tab+space or manually click on the tiny gray "cc" letters) onslaught is just absurd. When you change how applications…

Not to mention the lovely "This option that used to be here no longer is here." getting the response of "its still here (for me)". Youtube specifically loves to do this to me.

Re: A/B testing gets misused to juice metrics in the short term

#18

Post by regularfry »

kjhgkjghkj wrote: post #9

Intentional or not, one outcome on sites that are relentlessly A/B tested is that the resulting UI design lets users know that content they want is there, they just need to click and scroll a bit more to find it. Having left FB years ago, I now watch people "navigate" their site/apps with disbelief.

Isn't that exactly the problem? The resulting UI isn't designed, it's aggregated across a disjointed set of granular tweaks.

Re: A/B testing gets misused to juice metrics in the short term

#19

Post by mjburgess »

mschuster91 wrote: post #7

My biggest issue with A/B testing isn't even mentioned here... gaslighting your customers is absolutely not OK . Particularly with older people, the constant "where the f..k did Outlook now put feature XYZ" (in the case that comes to my mind, the CC bar which used to be tab-reachable, now you have to tab+space or manually click on the tiny gray "cc" letters) onslaught is just absurd. When you change how applications…

You're downvoted, but this issue is more common than it seems; and I agree, more serious than it seems.

You're describing adversarial UI changes to small populations of then unsupported customers. This can have outsized impacts on vulnerable populations, eg., esp. the elderly.

Re: A/B testing gets misused to juice metrics in the short term

#20

Post by ssharp »

mschuster91 wrote: post #11
smeyer wrote: post #4

Earlier quoted context omitted.

>Instead try to improve the customer experience AB testing can be (although isn't always) used to improve the customer experience. Assuming you know exactly what will make the customer experience best without actually testing it can also lead to a worse experience.

> Assuming you know exactly what will make the customer experience best without actually testing it can also lead to a worse experience. For that you usually hire a market research company or do what they will do: take an interviewer, two cameras (one front-face, one top-hands) and hire an as-diverse-as-possible pool of test candidates that you then put through whatever workflow optimization you want to do. Then afte…

It's not really an either/or option. You can use testing to validate the changes stemming from market research.

Having seen lots of site redesigns go horribly wrong due to 100% earnest people trying their best and utilizing the research that was afforded to the process, I always recommending incrementally testing into changes on high-value / high-risk applications, even when the "improvements" were backed by solid research. You never know until you release.

Post reply on HN