Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing gets misused to juice metrics in the short term

zumsteg.net

Post reply on HN
Print view View original

111–120 of 346 posts

Re: A/B testing gets misused to juice metrics in the short term

#111

Post by josefx »

treis wrote: post #52
Kaotique wrote: post #3

AB testing shows zero respect to your customers. It is the equivalent of testing your theories on lab rats. Instead try to improve the customer experience, make better products, improve customer service.

>Instead try to improve the customer experience, make better products, improve customer service. Without a metric to say what is "better" and a method to measure it this is empty advice.

But does AB testing provide the only or even best metric for that? It probably is the cheapest way requiring the least engagement with the lab mice.

Re: A/B testing gets misused to juice metrics in the short term

#112

Post by noirbot »

rout39574 wrote: post #65
noirbot wrote: post #41

Earlier quoted context omitted.

But that's just optimizing for bad metrics. At this point, anyone who thinks "engagement" and "time spent on page" are customer-positive metrics is in a different mental space than you and I. There's a lot of ineffable things that make up good customer experience that would be hard-to-impossible to A/B test, but it doesn't mean that A/B testing is "unsafe" just because it could be used to optimize for bad things any…

I think the point is that most metrics are "bad metrics" for this purpose, as suggested by Goodheart's law. https://en.wikipedia.org/wiki/Goodhart%27s_law Further, I imagine that the obvious "known bad" metrics are not selected only by "A few bad apples". I think it's likely they are selected by the mass of business actors looking for current quarter results.

For sure. I don't think there's general "overall" metrics that you want to be testing against every single time on every change outside of basic performance metrics for loading or rendering in real-world environments.

I wasn't at all trying to say that only a few places are optimizing for bad things, but as you see all over this thread, there's a number of companies that immediately come to people's minds as bad actors when it comes to A/B testing - Google, Meta, Microsoft. There's plenty of other companies that are more ethical about it, or use it as part of rolling out general changes and collecting feedback. I know half of the time I log into the AWS Console it has some sort of "Hey, we're testing out a new upcoming UX for this page. Click here if you want to go back to the old one", which seems like a decent way for them to get feedback on the new designs while not drastically disrupting things.

Re: A/B testing gets misused to juice metrics in the short term

#113

Post by coldcode »

mattgreenrocks wrote: post #82

I recall when Booking.com rolled out the false urgency features. I was amazed at how utterly trashy and desperate they were. The problem is it's not subtle at all; there's a handful of those features that, when combined, end up being overbearing and noisy: "3 people looked at this listing within the past 3 days! 12 rooms left at this rate!" I don't care. I'm looking to book business or vacation travel. If a spot fill…

Having worked in the OTA space, every time I see their (sometimes funny) ads, I want to call them Booking.Nope

OTA make comparisons a bit easier, but everything is negotiated and contractually controlled to keep people from just going to the hotel directly. Secret hotel prices (like HotWire if that still exists (Expedia) or Travelocity's Top Secret hotels if that still exists (also Expedia)) are an even more crazy negotiation. Hotel Tonight at least used to contact the hotel chains every day for that day's options, though since they were bought by AirBnB who knows what they do.

These days I just find a nice hotel and book with them/their system directly. Airlines too, since airlines fail to give all their options to the OTAs.

In some ways its sad that aggregators don't work all that well in the main travel industry (Flight/Hotel/Car) but travel is extremely complicated, highly competitive and still very fractured except for airlines. Pricing comparisons are not very useful since they are so mangled and obfuscated that you may as well just go to several sites and do it yourself by hand. For example Spirit Airlines used to give us prices for their tickets at $X and were always cheaper than everyone else; yet once you booked at that price they hit you for everything extra (bags, res, for all I know oxygen) then our customers complained we were fooling them and the real cost was higher.

Re: A/B testing gets misused to juice metrics in the short term

#114

Post by badRNG »

jollybean wrote: post #56
grapeskin wrote: post #43

Earlier quoted context omitted.

MBAs aren’t something people are born into or end up through no choice of their own. Bigotry against MBAs is impossible. It’s like being a loan shark or pimp. People actively choose that life and society talks poorly about them because of the social harm they bring. There’s nothing stopping someone from washing their hands of any of those things and being productive.

Quoted post unavailable.

> "obstinate or unreasonable attachment to a belief, opinion, or faction; in particular, prejudice against a person or people on the basis of their membership of a particular group"

This is just the fallacy of definition. Technically being anti-racist or anti-terrorism is "bigotry" on a fallaciously "strict" interpretation of some short definition of the word if one refuses to acknowledge its social and historical context (and how the word is used in the real world.)

Re: A/B testing gets misused to juice metrics in the short term

#115

Post by chunkyks »

"If a study came out that said deafening high-pitched noises increased conversion rates, we would all be bleeding from our ears by end of business tomorrow, right?"

Netflix auto play? Is that you? You were a hateful idea, no one liked you, yet you stubbornly hung on for far too long

Re: A/B testing gets misused to juice metrics in the short term

#116

Post by ThalesX »

koheripbal wrote: post #106
loriverkutya wrote: post #67

Earlier quoted context omitted.

I never seen an A/B test where the goal was not to maximise profit for the company.

This comment is way too broad and cynical. It reminds me of the sentiment I sometimes hear from my teenage children before I explain to them that the world isn't a Bernie Sander/Reddit gotcha soundbite and that reality is complicated.

What would the other advantages of A|B testing be for a corporation instead of widening the funnel, increasting lifetime revenue or other such bottom-line focused goals?

Re: A/B testing gets misused to juice metrics in the short term

#117

Post by Kaotique »

Saturdays wrote: post #10
Kaotique wrote: post #3

AB testing shows zero respect to your customers. It is the equivalent of testing your theories on lab rats. Instead try to improve the customer experience, make better products, improve customer service.

That's a very weak blanket statement, there are totally reasonable A/B tests you can run that don't deteriorate a user's experience, and the results can guide you to a better customer experience overall.

It did not mean it too seriously, of course there are also good AB tests, but there are a lot of bad ones out there. Those are what the article was about.

(edited for clarification)

Re: A/B testing gets misused to juice metrics in the short term

#118

Post by nemo44x »

jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

A lot of this happens because different managers have different metrics/KPIs they are optimizing around and they all find "good places" to do things to help meet their goals. The secondary effects aren't considered. One managers quest for outperforming their goals comes at the expense of another managers goals.

Re: A/B testing gets misused to juice metrics in the short term

#119

Post by dpe82 »

jerf wrote: post #54
Jenk wrote: post #51

Earlier quoted context omitted.

> I'm signalling interest in product X as hard as I can, and you've AB tested that this is a great time to jangle your keys over there instead? If I may... I have seen data from a big retailer that shows any user that doesn't immediately purchase an item, is actually not that interested in the product on the screen. If a customer is going to buy something, they will do it promptly. Anyone else is just browsing. YMMV,…

In this case, what I'm referring to is: 1. Clicked on page. 2. Took maybe 10 seconds to take in what is "above the fold". 3. Scrolled down to see what else there is. 4. BAM! Popup triggered by scrolling down. While I understand what you're getting at, they do not yet have the info to know that I'm browsing or whatever. They were so excited about their stupid popup that they didn't even get that far. I will say, gener…

In fact, you want the dismiss button to be easily discovered and used. Dismissals are an important signal about the quality of the content; just as important as clicks. When you make the button impossible to use you rob yourself of that signal while simultaneously making click data far less reliable.

Re: A/B testing gets misused to juice metrics in the short term

#120

Post by teakettle42 »

koheripbal wrote: post #107
costcofries wrote: post #57

Earlier quoted context omitted.

"To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. " Your perspective is extremely short-sighted. A/B testing can result in this type of behaviour but that's just poor A/B testing. Good A/B testing focuses on removing distractions from the experience and helping users derive…

Do people honestly think that Apple doesn't do A/B testing?

Apple unequivocally does not do active A/B testing on their users by changing applications or the operating system out from under them.
Post reply on HN