Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing gets misused to juice metrics in the short term

zumsteg.net

Post reply on HN
Print view View original

321–330 of 346 posts

Re: A/B testing gets misused to juice metrics in the short term

#321

Post by iudqnolq »

dotecoffee wrote: post #301
loriverkutya wrote: post #67

Earlier quoted context omitted.

I never seen an A/B test where the goal was not to maximise profit for the company.

Have you ever seen "you've already bought this" on Amazon? Highly tested, reduced sales. Overruled by judgement. Stayed on the site. AB test results don't mean you leave your judgement behind.

Maybe. I find that very useful when I'm trying to buy the same thing again. It acts as a trust marker.

I do wish they kept better historic purchase data. I assume it's deleted absurdly quickly for compliance?

Re: A/B testing gets misused to juice metrics in the short term

#322

Post by goodside »

im3w1l wrote: post #281
goodside wrote: post #5

Earlier quoted context omitted.

There’s nothing about AB testing that requires you to use short-term metrics. I used to manage AB tests for online dating sites (OkCupid, Grindr) where subscription revenue is what matters, and the gains of any strategy will take months to materialize. We were well aware that, say, raising prices would yield more short-term revenue at the expense of long-term revenue. That didn’t stop us from testing, it just made th…

Did you A/B test the matching algorithms?

Don’t recall specifically. There maybe may have been randomized A/B tests on the “special blend” at some point, I think — it was never spelled out on the site, but I think that was the experimental mix du jour and we tended to use that instead of forced randomization.

Re: A/B testing gets misused to juice metrics in the short term

#323

Post by epups »

fairity wrote: post #302

Yes, it's possible to use A/B testing in a short-sighted fashion. Yes, I'm sure there are plenty of examples thereof. No, that doesn't mean A/B testing is inherently short-sighted. It's entirely possible to measure long-term secondary effects of an A/B test. Just save a record of treatment groups, and remember to come back and compare long-term metrics like LTV down the road. We do this all the time at my startup, an…

Yes, the author was not talking about A/B testing at all, but about popups and annoying UX patterns that get rewarded by poorly used A/B tests or short-term thinking. But there is nothing stopping anybody from using long-term A/B tests (I assume most companies do, like you). Unfortunately dark patterns might simply be profitable.

Re: A/B testing gets misused to juice metrics in the short term

#324

Post by ratww »

tbranyen wrote: post #278
ratww wrote: post #175

Earlier quoted context omitted.

Sure, but that's not really A/B testing, those are more often called staged rollouts or progressive rollouts.

I'm talking about running week or month long tests with control and multi test cells containing new functionality, configuration, or code to determine the viability of a single or combination of changes by analyzing statistical output driven by p-value and pre-determined target metrics. These types of experiments are extremely valuable in uncovering hard-to-find bugs, assuming you have sufficient logging and confiden…

Yes, I know this and I agree. I understand they use the same techniques and terminologies, but this is just not the kinda thing the article is criticizing.

Re: A/B testing gets misused to juice metrics in the short term

#325

Post by [deleted] »

daniel_reetz wrote: post #261
teakettle42 wrote: post #120

Earlier quoted context omitted.

Apple unequivocally does not do active A/B testing on their users by changing applications or the operating system out from under them.

I can't think of an Apple engineer who would make such a definitive statement on behalf of the company, in public. I also can't think of anyone working there who would have definitive knowledge across all apps and software, because disclosure would prevent such knowledge. Apple introduced A/B testing to Testflight in 2017 and more A/B testing to the App store this year. Source: I worked at Apple, but not in software.

[deleted]

Re: A/B testing gets misused to juice metrics in the short term

#326

Post by account42 »

jamiek88 wrote: post #267
quickthrower2 wrote: post #161

Earlier quoted context omitted.

Then say goodbye to the internet and hello to a mesh of country-specific networks. Finally, true decentralisation!

We already have that. It’s just we are in denial as a society about it. Globalization is over. The post Berlin wall fall consensus undermined and ruined. The last to realize, loses.

The quickest way to achieve defeat is to convince yourself that you have already lost.

Re: A/B testing gets misused to juice metrics in the short term

#327

Post by account42 »

3dGrabber wrote: post #256
dspillett wrote: post #203

Earlier quoted context omitted.

This is why every company gets a different address for me. If junk starts coming in, that address is blocked, and I stop doing business with that company (if I haven't already). This sometimes falls foul of spammers adding some random addresses of the form blahblah@mycatchall.domain.tld or @mycatchall.domain.tld into their lists, but that hasn't happened often enough to be a problem. That it isn't much of a problem s…

> If junk starts coming in, that address is blocked, and I stop doing business with that company (if I haven't already). AND CALL them, if possible: “I’ve received marketing emails from your company recently, how is this possible, I’ve never signed up, yaddayadda… “ Generate some cost on their side.

Support calls will probably cost you more than the company unless you value your time very little - most companis don't post numbers answered by anyone paid highly enough for this to have an impact. Instead, call them out publicly or mail/call a someone in a leadership position if you can find contact information.

Re: A/B testing gets misused to juice metrics in the short term

#328

Post by account42 »

iudqnolq wrote: post #321
dotecoffee wrote: post #301

Earlier quoted context omitted.

Have you ever seen "you've already bought this" on Amazon? Highly tested, reduced sales. Overruled by judgement. Stayed on the site. AB test results don't mean you leave your judgement behind.

Maybe. I find that very useful when I'm trying to buy the same thing again. It acts as a trust marker. I do wish they kept better historic purchase data. I assume it's deleted absurdly quickly for compliance?

Huh, Amazon keeps years of order history for me and you can't even remove it, just "hide" orders which will then still be available just with an extra step. And I'm in the EU so no idea where else they would have more compliance problems with that.

Re: A/B testing gets misused to juice metrics in the short term

#329

Post by HelloNurse »

Kaotique wrote: post #117
Saturdays wrote: post #10

Earlier quoted context omitted.

That's a very weak blanket statement, there are totally reasonable A/B tests you can run that don't deteriorate a user's experience, and the results can guide you to a better customer experience overall.

It did not mean it too seriously, of course there are also good AB tests, but there are a lot of bad ones out there. Those are what the article was about. (edited for clarification)

Obviously, it depends on what the A/B test is about. Molesting or not the customer for the sake of some shortsighted metric is a bad choice; deciding what content should go above the fold or not (e.g. Amazon places images, short description, details/specs and similar products in that order) is a good choice.

Re: A/B testing gets misused to juice metrics in the short term

#330

Post by account42 »

zanecodes wrote: post #269
selfhoster11 wrote: post #187

Earlier quoted context omitted.

At one point I was trying to set up my grandma with a popular video calling app on a dedicated device so we could stay in touch. Before I could give her the freshly grandparent-proofed device, said video calling app upgraded on my parents' PC first and changed literally every single element of the UI beyond recognition. To someone the age of my grandma, that would be literally like bricking the device remotely, becau…

In the same way that we try to use semantic versioning to prevent unintended breakages in dependents of our public-facing APIs, I think we developers should start considering the UI/UX of our apps to be their public-facing API, and use the philosophy of semantic versioning accordingly. When we rename or remove a function from an API, that's a breaking change, any dependent software will no longer work unless it's mod…

We should also keep arround long-term support releases of older versions of user-facing applications so that people can update at their own convenience and not at a point determined by some release manager. With server software and libraries no one expects everyone to upgrade immediately to new major versions but somehow this is an acceptable expectation from end users.
Post reply on HN