A/B testing gets misused to juice metrics in the short term

Post by int_19h » Tue, Jul 12, 2022, 6:02 PM UTC

amluto wrote: post #143

Reading this makes me think of the handful of sites, often targeted at professionals, that highly optimize for the experience of actually buying things. McMaster-Carr comes to mind. Their users shop there over and over, and McMaster wants to keep them. So you can find things for $2 or $2000, shipping prices are inoffensive, customer service is friendly but rarely needed, and there are minimal distractions on the way…

Post by jklinger410 » Tue, Jul 12, 2022, 6:06 PM UTC

bentcorner wrote: post #177

hbn wrote: post #33

I've talked about this on HN but I'll say it again. A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a bi…

Google recently shut off Hangouts on me with a flight (since several of my contacts reported that Hangouts worked fine for them). It's kind of mind boggling they'd decide to do that - the replacement they direct me to (Google Chat) doesn't even have feature parity so I just dropped them and moved my social circle using Hangouts to a different app (since at this point they all faced the same problem and we decided on…

Post by thatguy0900 » Tue, Jul 12, 2022, 6:08 PM UTC

costcofries wrote: post #77

drewcoo wrote: post #70

Earlier quoted context omitted.

"You're doing it, but in some subtle but very important way that's not at all obvious to you, you're doing it wrong." How many tech startup patterns fit that? That's a sign that either the pattern does not generalize well or it's snake oil.

I think of it as validating and improving imperfect user assumptions.

Post by Philadelphia » Tue, Jul 12, 2022, 6:13 PM UTC

goodside wrote: post #5

causi wrote: post #2

The problem with AB testing is that it's a short-term strategy. For example, if a news site runs AB testing with headlines, they'll find that bullshit clickbait headlines get more pageviews than concise, accurate headlines, but the constant use of clickbait headlines will over time destroy overall traffic to your site. More frustratingly, sites run by smart people tend to fall into a balance where the worst articles…

There’s nothing about AB testing that requires you to use short-term metrics. I used to manage AB tests for online dating sites (OkCupid, Grindr) where subscription revenue is what matters, and the gains of any strategy will take months to materialize. We were well aware that, say, raising prices would yield more short-term revenue at the expense of long-term revenue. That didn’t stop us from testing, it just made th…

Post by tnolet » Tue, Jul 12, 2022, 6:20 PM UTC

LeonM wrote: post #139

mattgreenrocks wrote: post #82

I recall when Booking.com rolled out the false urgency features. I was amazed at how utterly trashy and desperate they were. The problem is it's not subtle at all; there's a handful of those features that, when combined, end up being overbearing and noisy: "3 people looked at this listing within the past 3 days! 12 rooms left at this rate!" I don't care. I'm looking to book business or vacation travel. If a spot fill…

Booking.com (or any other hotel/flight booking sites) are the masters of dark patterns. This is what happens when software is 'finished', companies start to optimize for profit, regardless of customer experience. But unfortunately, it works. I've seen friends that I consider intelligent panic buy tickets/hotels, "because prices are going up since the last time I checked!" Next time you want to book anything, browse a…