Live data from Hacker News

A/B testing gets misused to juice metrics in the short term

zumsteg.net

241–250 of 346 posts

Re: A/B testing gets misused to juice metrics in the short term

#241
post #33

I've talked about this on HN but I'll say it again. A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a bi…

"To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. " Your perspective is extremely short-sighted. A/B testing can result in this type of behaviour but that's just poor A/B testing. Good A/B testing focuses on removing distractions from the experience and helping users derive…

UX feedback is usually retrieved via user interviews.

I personally have never heard from a product person „Let’s A/B test whether this is delightful“. And I think that’s because delightfulness or satisfaction is impossible to quantify in A/B tests. You only get to measure things like engagement, signup rates, retention etc. - cold hard taps on the screen, and no more.

And I must say that I‘m glad that, right now, apps can’t just scan my face (or cortisol levels, or pheromones or…) for emotional clues while I read their pesky push notifications that want to coax me back into their daily active user base.

Re: A/B testing gets misused to juice metrics in the short term

#242
Anecdotal: we released plenty of improved features, like a better gallery to see the items in our shop, users used it a lot +250%, but conversion rate went down 4%.

They spent more time seeing the items and..didn't like the pics and conversion went down. In the end we reverted to the crap gallery we had before, they don't click it anymore and conversion went back up again..

Re: A/B testing gets misused to juice metrics in the short term

#243

I had a PM who pushed us to A/B test _everything_. We hired a new graphic designer who suggested that we change our product links from ALL CAPS to Title Case (a very popular idea on the team, and his first real suggestion after a few weeks with us), and she insisted that we A/B test it first. It felt like an insult to him, and a dumb test since title case looked way better. The three key outcomes I observed from the…

I've had a similar experience, although my learning was "people, even experts, are really terrible at understanding which treatment will perform better".

We always run >=3 variants, surveyed the dozen team members on which one they thought would run. Over the years, there was no clear pattern over who could make that prediction.

IE, it's not possible to predict which is the most effective treatment, even when you include a really bad idea in the treatments!

Re: A/B testing gets misused to juice metrics in the short term

#244

Anecdotal: we released plenty of improved features, like a better gallery to see the items in our shop, users used it a lot +250%, but conversion rate went down 4%. They spent more time seeing the items and..didn't like the pics and conversion went down. In the end we reverted to the crap gallery we had before, they don't click it anymore and conversion went back up again..

If it possible that there'd be a long term effect like:

* Users know you have a nice gallery

* They are more likely to shop at your store

* In the end, you get more sales despite the lower conversation rate

Re: A/B testing gets misused to juice metrics in the short term

#245

Earlier quoted context omitted.

> STOP CHANGING THINGS! Even if the new UI is better (debatable), some people just like or rely on a particular layout to operate the device or app. Don't rearrange without giving a ~permanent setting to use the old layout. The fact that we're 15 years into smartphones being popular and that phones & computerized address books were basically fully-solved interfaces long before those took off, and they're still fuckin…

Well, all of these companies employ teams of ui and ux designers. They will never show up to work one day and tell their boss the ui is done. No matter how perfect the design currently is

of course not. that's the responsibility of the company management to say "okay UI is good lets divert resources elsewhere"

Re: A/B testing gets misused to juice metrics in the short term

#246

Earlier quoted context omitted.

Google recently shut off Hangouts on me with a flight (since several of my contacts reported that Hangouts worked fine for them). It's kind of mind boggling they'd decide to do that - the replacement they direct me to (Google Chat) doesn't even have feature parity so I just dropped them and moved my social circle using Hangouts to a different app (since at this point they all faced the same problem and we decided on…

> Google Chat) doesn't even have feature parity Which features are you missing?

check the AB test results

Re: A/B testing gets misused to juice metrics in the short term

#247
post #214
post #193

Earlier quoted context omitted.

We AB tested a performance enhancement to our frontend web app to show that speed had significant benefits to the business. we use the results justify the investment cost. We spent the next six months working on making the site faster because of it. It is a tool. How would you measure things without ab testing?

The fact that you need an A/B test to demonstrate that frontend performance has an impact (on user experience first and foremost) in 2022 speaks volumes.

this is the stance I take on AB tests. The are objectively good at things you shouldnt need objective evidence for

Re: A/B testing gets misused to juice metrics in the short term

#248
post #214
post #193

Earlier quoted context omitted.

We AB tested a performance enhancement to our frontend web app to show that speed had significant benefits to the business. we use the results justify the investment cost. We spent the next six months working on making the site faster because of it. It is a tool. How would you measure things without ab testing?

The fact that you need an A/B test to demonstrate that frontend performance has an impact (on user experience first and foremost) in 2022 speaks volumes.

But how big is the impact and does that return compare to that of other investments you aren't making because you are improving performance instead?

Re: A/B testing gets misused to juice metrics in the short term

#249
post #176
post #48

Earlier quoted context omitted.

My 'favorite' silly thing PMs do is UX research studies (typically on 5-10 people) and essentially ask completely untrained people if we should go with X/Y or Z. It's a super-effective way of avoiding responsibility for product decisions ("the data suggest we should go with Y"). If only building good products were as easy as asking what customers think they want.

Either they're doing the UX research wrong or (more likely) you're misunderstanding the process. You don't ask them if you should do X/Y/Z. You ask them to do X in the program, and see that none of them can find widget Y which controls it because they keep clicking on widget Z. It's about observing the users fumble through your UX when you know their motivation.

I wish the problem were my misunderstanding the process, because then I could fix it easily by learning more about the process. I do get where you're coming from though.

Re: A/B testing gets misused to juice metrics in the short term

#250

Earlier quoted context omitted.

Google recently shut off Hangouts on me with a flight (since several of my contacts reported that Hangouts worked fine for them). It's kind of mind boggling they'd decide to do that - the replacement they direct me to (Google Chat) doesn't even have feature parity so I just dropped them and moved my social circle using Hangouts to a different app (since at this point they all faced the same problem and we decided on…

> Google Chat) doesn't even have feature parity Which features are you missing?

Video chat. It redirects me to Google Meet (yet another app!) and the model that Google Meet uses is completely backwards (create a meeting, then invite people) compared to how I use Hangouts (call someone).

It's a complete pita and the family-acceptable-factor is low.

Post reply on HN