Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing gets misused to juice metrics in the short term

zumsteg.net

Post reply on HN
Print view View original

31–40 of 346 posts

Re: A/B testing gets misused to juice metrics in the short term

#31

Post by johnnymorgan »

mschuster91 wrote: post #7

My biggest issue with A/B testing isn't even mentioned here... gaslighting your customers is absolutely not OK . Particularly with older people, the constant "where the f..k did Outlook now put feature XYZ" (in the case that comes to my mind, the CC bar which used to be tab-reachable, now you have to tab+space or manually click on the tiny gray "cc" letters) onslaught is just absurd. When you change how applications…

Love the post and I tend to agree.

As Product manager/owner I've only found A/B testing useful when trying to narrow in on a specific demographic and you are trying to find some optimization.

The marketing/sales funnel use of it is kind of gross and has ruined , imo, something that has utility in a very narrow scope.

Cheers, also very much agree customers should be informed and allowed to opt out.

'hey we have a new UX to try..would you like to switch?' the data from people that opt-in is way better

Re: A/B testing gets misused to juice metrics in the short term

#33

Post by hbn »

I've talked about this on HN but I'll say it again.

A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a big part of that was the constant A/B/C/D/E/F testing. I never felt like I was using a complete product, everything felt like a constant beta that could be changed or rearranged at any point, and I was just doing free testing work for them while they harvest all my data.

Every app update was a risk of the app rearranging itself, or features appearing/disappearing. Eventually it didn't even come from app updates in the Play Store, and new interfaces would just appear one day when a server somewhere marked your account as being in the group that gets the new UI. This app that you were familiar with could at any point be rearranged when you open it on any given day. Then maybe a week later you open it and it's back to how it was before. A button you thought was here suddenly isn't, and you question whether something actually changed or if you're losing your mind. It's a subtle gaslighting that eventually I couldn't stand any more.

To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. That's not to say a company like Apple is creating products out of the goodness of their heart, but at least it feels like it was developed by humans who made an opinionated call as to what they thought was the right design decision, and what they would want to use. And in my 2 years of owning an iPhone, I've never opened my reminders app to find out that it's completely unrecognizable, or my messages app has been renamed or rethemed for the umpteenth time.

Re: A/B testing gets misused to juice metrics in the short term

#34

Post by noirbot »

throwaway290 wrote: post #8
smeyer wrote: post #4

Earlier quoted context omitted.

>Instead try to improve the customer experience AB testing can be (although isn't always) used to improve the customer experience. Assuming you know exactly what will make the customer experience best without actually testing it can also lead to a worse experience.

A/B testing helps you maximize a metric, not make customer experience better. Those are different things.

They're only different if you've selected bad metrics. If you've got two different search algorithms, running an A/B test and measuring how often the user selects the first item returned is a good measure for how well your search algorithm is returning the information the customer wanted, which is good customer experience.

Re: A/B testing gets misused to juice metrics in the short term

#36

Post by avisser »

goodside wrote: post #25
tsimionescu wrote: post #12

Earlier quoted context omitted.

Sure, but in many cases, such as the example given by GP, long-term AB testing is hard or almost impossible. For the testing to have validity, you need the A and B cohorts to be stable, and have little or no overlap, and that is hard for long time spans for anything that is not account based (and somewhat dangerous even for account-based things, as people will almost certainly start to notice that they are getting a…

In online dating, at least, this is a non-issue. Using an online dating app is, ironically, a solitary enough activity that people don’t go around comparing whether their UI is different from their friends’ UI. You of course can’t let the same user see two versions, but that just means doing permanent group assignment on signup. We used to A/B test subscription prices over enormous ranges (e.g., randomly giving some…

I wonder if you two are talking past each other a little. I'm thinking that A/B testing for content is a different beast than A/B testing for experience.

Re: A/B testing gets misused to juice metrics in the short term

#37

Post by Jasper_ »

jollybean wrote: post #32
verve_rat wrote: post #24

Earlier quoted context omitted.

Calling that bigotry is offensive to anyone that has been subjected to actual bigotry. You should be ashamed.

Quoted post unavailable.

Before you commented, I thought "there is nothing in this world an MBA can't make worse" was a bit too harsh. Now, I see it's actually accurate.

Re: A/B testing gets misused to juice metrics in the short term

#38

Post by nkrisc »

rgavuliak wrote: post #30
Kaotique wrote: post #3

AB testing shows zero respect to your customers. It is the equivalent of testing your theories on lab rats. Instead try to improve the customer experience, make better products, improve customer service.

I have experience where the company paid a UX agency to create a flow that was by all standards better customer experience and a better product, nicer too. They ran an AB test, turns out people were more likely to pay with the old version. AB testing is good that it challenges what UX people think is better experience or product with hard metrics.

Which is why A/B testing is an important part of the UX toolkit. It's a tool among others, and is one way to validate assumptions. A good UX designer will try to base their designs on data and reasonable hypotheses drawn from the data, but a new design or flow is necessarily based on some amount of assumptions, so it requires validation.

That said, an A/B test does not tell you why something didn't work. You can make further assumptions based on the results and develop new hypotheses, but it never tells you why. Typically you would do some kind of qualitative UX research on a prototype or even static concepts beforehand to identify these kinds of issues before you even expend the effort to do a live A/B test. Far cheaper to do a study with 6-12 people and a prototype than to build out a full, functioning A/B test experience.

It's possible the flow they created was generally better but perhaps it had one fatal flaw. Perhaps that flaw could easily be remedied once identified.

A/B testing is just one small part of a good UX process.

Re: A/B testing gets misused to juice metrics in the short term

#39

Post by AndrewThrowaway »

Imagine this beautiful business software which during the years and numerous A/B tests, "best UX practices", design languages and whatnot became this all "applesque", minimalist UI with 80% of it being a white space. By the way winning numerous design awards.

However entering e.g. client's information take a lot of steps, you are constantly clicking "Next" throughout these beautiful wizards and pages. After some time everybody starts to feel that there must be a better way.

What is the solution?

Spreadsheet import! Where you can just do everything in this "complicated" UI of Microsoft Excel, with formulas, and hundred buttons at once on the screen. Fill in hundreds of rows of information and just import it to the "beautiful business system".

Post reply on HN