Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing gets misused to juice metrics in the short term

zumsteg.net

Post reply on HN
Print view View original

81–90 of 346 posts

Re: A/B testing gets misused to juice metrics in the short term

#81

Post by ravivyas »

axg11 wrote: post #47

A/B testing is local optimization. It should only be done on a mature(-ish) product when you have given up on finding a global minimum. Running experiments and A/B tests are popular because it is _guaranteed_ to give you signal. If you have a large engineering team and you're not sure how to filter the quality of results, gating everything through A/B tests is a well understood methodical way to ensure only positive…

"Running experiments and A/B tests are popular" ... because you can give up on your own judgment and opinions and say "the data says"

Re: A/B testing gets misused to juice metrics in the short term

#82

Post by mattgreenrocks »

I recall when Booking.com rolled out the false urgency features. I was amazed at how utterly trashy and desperate they were.

The problem is it's not subtle at all; there's a handful of those features that, when combined, end up being overbearing and noisy: "3 people looked at this listing within the past 3 days! 12 rooms left at this rate!" I don't care. I'm looking to book business or vacation travel. If a spot fills up I'll just go somewhere else. It'll be fine either way.

I don't use them anymore for that reason. Old soul (me) is old. (I'm probably in a minority, judging by their advertisement budget.)

Re: A/B testing gets misused to juice metrics in the short term

#83

Post by mattgreenrocks »

CSMastermind wrote: post #72
jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

Recently I got an account on a developer tool (Checkly) because the company I joined uses it. I then got 5 different emails from them in a 48 hour period. Like I'm sure many users sign up then drop out of your funnel but I'm part of an organization that's a paying customer. I'm already going to use your stuff. What possible business benefit could there be to you spamming me? If anything you're risking the inverse - i…

The 'cost' of email is borne by the recipient, mostly.

Re: A/B testing gets misused to juice metrics in the short term

#84

Post by jaggederest »

axg11 wrote: post #47

A/B testing is local optimization. It should only be done on a mature(-ish) product when you have given up on finding a global minimum. Running experiments and A/B tests are popular because it is _guaranteed_ to give you signal. If you have a large engineering team and you're not sure how to filter the quality of results, gating everything through A/B tests is a well understood methodical way to ensure only positive…

> Running experiments and A/B tests are popular because it is _guaranteed_ to give you signal. If you have a large engineering team and you're not sure how to filter the quality of results, gating everything through A/B tests is a well understood methodical way to ensure only positive work makes it way through.

It's almost guaranteed to ensure only false positive work makes its way through. If you're picking 0.05 as your P value, and you're running dozens to hundreds of tests, your false positives are almost certain to exceed your actual positives.

When I'm working for clients that do a lot of A/B testing, I suggest that they should always run A/A tests to ensure that they're not incorrectly rejecting the null hypothesis. If your A/A tests are showing significant differences, you have a problem in your testing pipeline that by definition can't be cured by more testing. You need holdout groups and selectivity about what to test, instead of just throwing everything at the proverbial wall.

Re: A/B testing gets misused to juice metrics in the short term

#85

Post by Eddy_Viscosity2 »

jollybean wrote: post #14
Eddy_Viscosity2 wrote: post #6

I'd say this is a sub-category of the saying: "There is nothing in this world that an MBA can't and won't make worse". It happens when people change perspectives from building and sustaining businesses to exploiting and squeezing every employee, supplier, and customer for the last drop.

This is just casual bigotry.

I'll say that there are managers out there who do fantastic and important work, but they are the ones that have a build & sustain perspective. If you feel offended by this comment, its because you are in the squeeze & exploit group and you know it.

Re: A/B testing gets misused to juice metrics in the short term

#86

Post by verve_rat »

jollybean wrote: post #32
verve_rat wrote: post #24

Earlier quoted context omitted.

Calling that bigotry is offensive to anyone that has been subjected to actual bigotry. You should be ashamed.

Quoted post unavailable.

I hate to break it to you, but if you have a problem with 'technical types', this might not be the best place to hang out...

Re: A/B testing gets misused to juice metrics in the short term

#87

Post by chairmanwow1 »

CSMastermind wrote: post #72
jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

Recently I got an account on a developer tool (Checkly) because the company I joined uses it. I then got 5 different emails from them in a 48 hour period. Like I'm sure many users sign up then drop out of your funnel but I'm part of an organization that's a paying customer. I'm already going to use your stuff. What possible business benefit could there be to you spamming me? If anything you're risking the inverse - i…

Sounds like they are optimizing for a KPI on time to full integration. Someone else paid for it, now they want to make sure that you are actually using it.

Still absurd, but I know this is a problem friends of mine have had.

Re: A/B testing gets misused to juice metrics in the short term

#88

Post by naravara »

jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

Honestly instead of a cookie law I wish GDPR has imposed a rule that required all those stupid interstitial pop-ups to conform to a standard that could be easily blocked by the browser. I mean they are asking for emails, which is a massive and totally unnecessary proliferation of personally identifiable information.

I hate them so much. It makes it feel like so much more of a chore to try to do research or look for things online. I'd honestly prefer 56k page-load speeds if the pages were free of this garbage.

Re: A/B testing gets misused to juice metrics in the short term

#89

Post by costcofries »

loriverkutya wrote: post #67
costcofries wrote: post #57

Earlier quoted context omitted.

"To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. " Your perspective is extremely short-sighted. A/B testing can result in this type of behaviour but that's just poor A/B testing. Good A/B testing focuses on removing distractions from the experience and helping users derive…

I never seen an A/B test where the goal was not to maximise profit for the company.

That's probably intentional and a sign that it was a well designed A/B test :)

Re: A/B testing gets misused to juice metrics in the short term

#90

Post by purplerabbit »

jaggederest wrote: post #84
axg11 wrote: post #47

A/B testing is local optimization. It should only be done on a mature(-ish) product when you have given up on finding a global minimum. Running experiments and A/B tests are popular because it is _guaranteed_ to give you signal. If you have a large engineering team and you're not sure how to filter the quality of results, gating everything through A/B tests is a well understood methodical way to ensure only positive…

> Running experiments and A/B tests are popular because it is _guaranteed_ to give you signal. If you have a large engineering team and you're not sure how to filter the quality of results, gating everything through A/B tests is a well understood methodical way to ensure only positive work makes it way through. It's almost guaranteed to ensure only false positive work makes its way through. If you're picking 0.05 as…

Great insight. Without this approach, A/B testing could be used to generate an infinite stream of meaningless work
Post reply on HN