Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing gets misused to juice metrics in the short term

zumsteg.net

Post reply on HN
Print view View original

41–50 of 346 posts

Re: A/B testing gets misused to juice metrics in the short term

#41

Post by noirbot »

wongarsu wrote: post #29
ssharp wrote: post #16

Earlier quoted context omitted.

Better customer experiences often times lead to increased metrics. They are not totally different things.

Maybe if you are very aware of the fact that your goal and metric aren't totally aligned, but that often gets lost. As a result A/B testing for longer website visits can make websites that make it more obvious that the information you want is there, but also make the path to actually get it longer. A/B testing for engagement might promote divisive behavior and fights. A/B testing for read rate or clicks might lead to…

But that's just optimizing for bad metrics. At this point, anyone who thinks "engagement" and "time spent on page" are customer-positive metrics is in a different mental space than you and I. There's a lot of ineffable things that make up good customer experience that would be hard-to-impossible to A/B test, but it doesn't mean that A/B testing is "unsafe" just because it could be used to optimize for bad things any more than any other telemetry or metrics gathering could be bad because you could optimize for evil things. And at the same time, bad management and product leadership can optimize and develop towards bad goals with plenty of tools that aren't A/B testing.

It seems to miss the point to blame/stigmatize a specific tool because it's been used poorly by a few bad actors in a public way.

Re: A/B testing gets misused to juice metrics in the short term

#43

Post by grapeskin »

jollybean wrote: post #32
verve_rat wrote: post #24

Earlier quoted context omitted.

Calling that bigotry is offensive to anyone that has been subjected to actual bigotry. You should be ashamed.

Quoted post unavailable.

MBAs aren’t something people are born into or end up through no choice of their own. Bigotry against MBAs is impossible.

It’s like being a loan shark or pimp. People actively choose that life and society talks poorly about them because of the social harm they bring. There’s nothing stopping someone from washing their hands of any of those things and being productive.

Re: A/B testing gets misused to juice metrics in the short term

#44

Post by shijie »

jollybean wrote: post #32
verve_rat wrote: post #24

Earlier quoted context omitted.

Calling that bigotry is offensive to anyone that has been subjected to actual bigotry. You should be ashamed.

Quoted post unavailable.

Not every critical or mean-spirited comment is bigotry. You got your fee-fees hurt because there’s probably some truth to the comment, even if it’s mean. Buck up, quit complaining, prove the parent comment wrong in your day-to-day work. Right now you’re just deepening a few MBA stereotypes at the moment.

Re: A/B testing gets misused to juice metrics in the short term

#46

Post by aceazzameen »

hbn wrote: post #33

I've talked about this on HN but I'll say it again. A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a bi…

You nailed it. Google is constantly forcing users to relearn most of their products year after year. Give me Google products from a decade ago and I'd still be happy. Now I'm moving on from Google also. It's an untrustworthy brand.

Re: A/B testing gets misused to juice metrics in the short term

#47

Post by axg11 »

A/B testing is local optimization. It should only be done on a mature(-ish) product when you have given up on finding a global minimum.

Running experiments and A/B tests are popular because it is _guaranteed_ to give you signal. If you have a large engineering team and you're not sure how to filter the quality of results, gating everything through A/B tests is a well understood methodical way to ensure only positive work makes it way through.

Early stage startups should never A/B test. When you're searching for product market fit, you're doing global optimization within the search space. Your product will change drastically as you make new learnings. Premature optimization (A/B tests) will only be detrimental.

Re: A/B testing gets misused to juice metrics in the short term

#48

Post by 10x-dev »

benja123 wrote: post #27

I say this a lot and I will keep saying it. Conversion != customer obsession. There is a place for A/B testing. It is necessary and can be extremely beneficial in helping your customers enjoy and use your product more successfully. The main issue is that people mix conversion with customer obsession! Whenever you work on a product or feature you should be asking yourself "Is this really good for my customer" - if the…

My 'favorite' silly thing PMs do is UX research studies (typically on 5-10 people) and essentially ask completely untrained people if we should go with X/Y or Z. It's a super-effective way of avoiding responsibility for product decisions ("the data suggest we should go with Y"). If only building good products were as easy as asking what customers think they want.

Re: A/B testing gets misused to juice metrics in the short term

#49

Post by enlyth »

jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

Speaking of 'sign up to our newsletter', one of the latest dark patterns I've found that astounded me was adding a checkbox to the login form [0], where you'd normally expect the 'remember me' checkbox to be. You almost click it out of muscle memory if you don't read what it says.

[0] https://i.postimg.cc/HW89hs7r/Screenshot-2022-07-12-145957.p...

Re: A/B testing gets misused to juice metrics in the short term

#50

Post by goodside »

avisser wrote: post #36
goodside wrote: post #25

Earlier quoted context omitted.

In online dating, at least, this is a non-issue. Using an online dating app is, ironically, a solitary enough activity that people don’t go around comparing whether their UI is different from their friends’ UI. You of course can’t let the same user see two versions, but that just means doing permanent group assignment on signup. We used to A/B test subscription prices over enormous ranges (e.g., randomly giving some…

I wonder if you two are talking past each other a little. I'm thinking that A/B testing for content is a different beast than A/B testing for experience.

I’m not disagreeing — My point is really, “not all AB testing is bad, even if the kind you’re most familiar with leads to shitty content.” My second comment was just more of side note.
Post reply on HN