Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing gets misused to juice metrics in the short term

zumsteg.net

Post reply on HN
Print view View original

231–240 of 346 posts

Re: A/B testing gets misused to juice metrics in the short term

#231

Post by the_gipsy »

nemo44x wrote: post #150
d23 wrote: post #123

Earlier quoted context omitted.

There was a point a few years ago where you could not see a single piece of user-generated content above the fold on the reddit home page. A bunch of teams had jockeyed for having their little carousels and banners put on top, and of course, metrics were always cited. I left a screenshot in slack and it ended up causing a couple of teams to have to roll back their widgets, but it always baffled me that we were able t…

> but it always baffled me that we were able to focus so much on the individual trees of metric optimization that we would miss the forest to that extent Always look to the decision maker's incentives and you'll almost always discover why things are the way they are. And often, to your point, there's an aspect of tunnel vision associated with it because considering the bigger picture is difficult as a company grows a…

Gervais principle says that senior leaders work for themselves, not the company, up to the point of working against the company.

Re: A/B testing gets misused to juice metrics in the short term

#232

Post by nuc1e0n »

This kind of optimisation for short term gains at the expense of long term sustainability is what is causing climate change and the collapse of the global economy. But the politicians and heads of industry who preside over this situation will all be retired/dead before it becomes a problem. Or so they thought.

Re: A/B testing gets misused to juice metrics in the short term

#233

Post by thatguy0900 »

corrral wrote: post #210
selfhoster11 wrote: post #187

Earlier quoted context omitted.

At one point I was trying to set up my grandma with a popular video calling app on a dedicated device so we could stay in touch. Before I could give her the freshly grandparent-proofed device, said video calling app upgraded on my parents' PC first and changed literally every single element of the UI beyond recognition. To someone the age of my grandma, that would be literally like bricking the device remotely, becau…

> STOP CHANGING THINGS! Even if the new UI is better (debatable), some people just like or rely on a particular layout to operate the device or app. Don't rearrange without giving a ~permanent setting to use the old layout. The fact that we're 15 years into smartphones being popular and that phones & computerized address books were basically fully-solved interfaces long before those took off, and they're still fuckin…

Well, all of these companies employ teams of ui and ux designers. They will never show up to work one day and tell their boss the ui is done. No matter how perfect the design currently is

Re: A/B testing gets misused to juice metrics in the short term

#234

Post by nebusoft »

enlyth wrote: post #196
_puk wrote: post #163

Earlier quoted context omitted.

Predicting the flight will be cancelled in 3.. 2.. 1.. But then the newsletter will haunt you for far far longer.. Not a burnt WizzAir customer at all! /s

Happened to my dad on Sunday, and the only replacement flight they would offer is for this Friday, what a complete joke air travel is in 2022.

I mean you're complaining about an ultra-low cost airline. why would you expect it to have a good customer experience?

Re: A/B testing gets misused to juice metrics in the short term

#236

Post by [deleted] »

corrral wrote: post #210
selfhoster11 wrote: post #187

Earlier quoted context omitted.

At one point I was trying to set up my grandma with a popular video calling app on a dedicated device so we could stay in touch. Before I could give her the freshly grandparent-proofed device, said video calling app upgraded on my parents' PC first and changed literally every single element of the UI beyond recognition. To someone the age of my grandma, that would be literally like bricking the device remotely, becau…

> STOP CHANGING THINGS! Even if the new UI is better (debatable), some people just like or rely on a particular layout to operate the device or app. Don't rearrange without giving a ~permanent setting to use the old layout. The fact that we're 15 years into smartphones being popular and that phones & computerized address books were basically fully-solved interfaces long before those took off, and they're still fuckin…

[deleted]

Re: A/B testing gets misused to juice metrics in the short term

#237

Post by CodesInChaos »

enlyth wrote: post #49
jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

Speaking of 'sign up to our newsletter', one of the latest dark patterns I've found that astounded me was adding a checkbox to the login form [0], where you'd normally expect the 'remember me' checkbox to be. You almost click it out of muscle memory if you don't read what it says. [0] https://i.postimg.cc/HW89hs7r/Screenshot-2022-07-12-145957.p...

atlassian does this as well: https://i.postimg.cc/zfwbG5Ft/atlassian-login.png

Re: A/B testing gets misused to juice metrics in the short term

#238

Post by bsuvc »

joe_the_user wrote: post #220
jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

Let me propose a different possibility. Suppose the site isn't concerned about the sale very much at all? Suppose the thing that the site uses to reel people in, is a good deal that isn't very profitable to the site but what the site then tries to sell is a very profitable near-scam/ripoff. Scaring off half the ordinary customers becomes worth it to get even 10% of the customers buying the scam. What seems like "poor…

Exactly this.

Many here are focusing on a single interaction. While the outcome of that single interaction is negative to the company, the aggregate outcome must be positive somehow, perhaps in the way you said, but it doesn't even have to be a scam or ripoff. Some products just have a higher margin and/or customer LTV.

As an individual, it is annoying, but the company is focusing only on the macro effect when it does something like this.

Re: A/B testing gets misused to juice metrics in the short term

#239

Post by walrus01 »

If you really want to see a massive amount of additional offers and small/partially hidden "no thanks" links, check out the work flow to reserve and rent a small light duty trailer with U-Haul.

You have to click through at least 10 pages of additional offers (and many extra price things that are added by default!) before you get to the actual checkout page.

Re: A/B testing gets misused to juice metrics in the short term

#240

Post by jakubmazanec »

jaggederest wrote: post #84
axg11 wrote: post #47

A/B testing is local optimization. It should only be done on a mature(-ish) product when you have given up on finding a global minimum. Running experiments and A/B tests are popular because it is _guaranteed_ to give you signal. If you have a large engineering team and you're not sure how to filter the quality of results, gating everything through A/B tests is a well understood methodical way to ensure only positive…

> Running experiments and A/B tests are popular because it is _guaranteed_ to give you signal. If you have a large engineering team and you're not sure how to filter the quality of results, gating everything through A/B tests is a well understood methodical way to ensure only positive work makes it way through. It's almost guaranteed to ensure only false positive work makes its way through. If you're picking 0.05 as…

That's why we calculate stuff like effect size and power of a test (or even better, use Bayesian statistics); just p < 0.05 is practically almost meaningless.
Post reply on HN