Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing gets misused to juice metrics in the short term

zumsteg.net

Post reply on HN
Print view View original

191–200 of 346 posts

Re: A/B testing gets misused to juice metrics in the short term

#191

Post by tshaddox »

jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

The thing is, you haven’t really shown that these sites aren’t successfully optimizing for conversions. Couldn’t it very well be the case that UI which annoys some high-intent users by interrupting them or adding steps to the checkout process also increases overall conversions?

Re: A/B testing gets misused to juice metrics in the short term

#192

Post by multivariate »

causi wrote: post #2

The problem with AB testing is that it's a short-term strategy. For example, if a news site runs AB testing with headlines, they'll find that bullshit clickbait headlines get more pageviews than concise, accurate headlines, but the constant use of clickbait headlines will over time destroy overall traffic to your site. More frustratingly, sites run by smart people tend to fall into a balance where the worst articles…

I write A/B tests for headlines for a news site, this is too broad a generalization. Clickbait titles aren't great for building subscribers or establishing trust, which is what we really care about (LTV). To the author's credit, our deepest testing insights come from analyzing a lot of historical data (not just last week's).

Re: A/B testing gets misused to juice metrics in the short term

#193

Post by adrr »

hbn wrote: post #33

I've talked about this on HN but I'll say it again. A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a bi…

We AB tested a performance enhancement to our frontend web app to show that speed had significant benefits to the business. we use the results justify the investment cost. We spent the next six months working on making the site faster because of it. It is a tool. How would you measure things without ab testing?

Re: A/B testing gets misused to juice metrics in the short term

#194

Post by dang »

Eddy_Viscosity2 wrote: post #6

I'd say this is a sub-category of the saying: "There is nothing in this world that an MBA can't and won't make worse". It happens when people change perspectives from building and sustaining businesses to exploiting and squeezing every employee, supplier, and customer for the last drop.

Please don't take HN threads into flamewar. The one you started here was particularly shallow and gratuitous.

https://news.ycombinator.com/newsguidelines.html

Re: A/B testing gets misused to juice metrics in the short term

#195

Post by dang »

jollybean wrote: post #14
Eddy_Viscosity2 wrote: post #6

I'd say this is a sub-category of the saying: "There is nothing in this world that an MBA can't and won't make worse". It happens when people change perspectives from building and sustaining businesses to exploiting and squeezing every employee, supplier, and customer for the last drop.

This is just casual bigotry.

Please don't feed flamewars on HN, regardless of how bad another comment is or you feel it is. It just makes everything worse.

https://news.ycombinator.com/newsguidelines.html

Edit: you broke the site guidelines particularly badly later in the thread. We ban accounts that do that, so please don't do it again. More here: https://news.ycombinator.com/item?id=32072856.

Re: A/B testing gets misused to juice metrics in the short term

#196

Post by enlyth »

_puk wrote: post #163
aroccoli wrote: post #79

Earlier quoted context omitted.

Yeah, I booked a flight on WizzAir two days ago, and this felt like a low blow even from WizzAir.

Predicting the flight will be cancelled in 3.. 2.. 1.. But then the newsletter will haunt you for far far longer.. Not a burnt WizzAir customer at all! /s

Happened to my dad on Sunday, and the only replacement flight they would offer is for this Friday, what a complete joke air travel is in 2022.

Re: A/B testing gets misused to juice metrics in the short term

#197

Post by dang »

verve_rat wrote: post #24
jollybean wrote: post #14

Earlier quoted context omitted.

This is just casual bigotry.

Calling that bigotry is offensive to anyone that has been subjected to actual bigotry. You should be ashamed.

Taking HN threads further into hellish flamewar is against the site guidelines, so please don't.

Attacking someone personally is not ok. We're trying to avoid the online callout/shaming culture here: https://hn.algolia.com/?sort=byDate&type=comment&dateRange=a...

Having a flamewar about the definition of a word is particularly pointless because different people have very different associations with the same word—especially when you remember that this place literally has commenters coming from all over the world.

Edit: you've unfortunately been breaking the site guidelines in other threads as well, e.g. here: https://news.ycombinator.com/item?id=32050947. We eventually ban accounts that do that, so please don't do that. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

Re: A/B testing gets misused to juice metrics in the short term

#198

Post by jl6 »

tshaddox wrote: post #191
jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

The thing is, you haven’t really shown that these sites aren’t successfully optimizing for conversions. Couldn’t it very well be the case that UI which annoys some high-intent users by interrupting them or adding steps to the checkout process also increases overall conversions?

True, A could be “annoy users” and B could be “don’t annoy users”, and A could perform better overall, but in this framework you might be missing C which is “annoy users except those already deep in the funnel”.

Re: A/B testing gets misused to juice metrics in the short term

#199

Post by dang »

jollybean wrote: post #78
Bancakes wrote: post #45

Earlier quoted context omitted.

Quoted post unavailable.

I'm both a Software Engineer and an MBA, and I can assure you that you're wrong, and with a comedic, glib kind of 'Musk-ian' lack of self awareness that's weirdly common among technical types. I mean it's getting funny now, I was hoping that your comment was satire (!), alas, no.

If you keep posting like this we're going to end up having to ban you again, so please stop.

Flamewar comments are not ok, and personal attacks are very not ok. No more of this please, regardless of how wrong someone else is or you feel they are.

https://news.ycombinator.com/newsguidelines.html

Re: A/B testing gets misused to juice metrics in the short term

#200

Post by dang »

Bancakes wrote: post #99
jollybean wrote: post #78

Earlier quoted context omitted.

I'm both a Software Engineer and an MBA, and I can assure you that you're wrong, and with a comedic, glib kind of 'Musk-ian' lack of self awareness that's weirdly common among technical types. I mean it's getting funny now, I was hoping that your comment was satire (!), alas, no.

>You're wrong because..... ugh! you just are, okay?!

Posting like this will get you banned here. If you'd please review https://news.ycombinator.com/newsguidelines.html and stick to the rules when posting here, we'd appreciate it.
Post reply on HN