A/B testing gets misused to juice metrics in the short term • Hacker News Board

Re: A/B testing gets misused to juice metrics in the short term

#171

Post by bottled_poe » Tue, Jul 12, 2022, 3:59 PM UTC

eloisius wrote: post #103

enlyth wrote: post #49

Earlier quoted context omitted.

Speaking of 'sign up to our newsletter', one of the latest dark patterns I've found that astounded me was adding a checkbox to the login form [0], where you'd normally expect the 'remember me' checkbox to be. You almost click it out of muscle memory if you don't read what it says. [0] https://i.postimg.cc/HW89hs7r/Screenshot-2022-07-12-145957.p...

Email marketing in general blows my mind. Marketers typically have absolutely no respect for consent, and the costs are completely borne by the recipient. The whole industry depends on dark patterns, shady list sharing, and scraping your email to add it to their lists despite you having no relationship with them. I know it's not simple, and it's just my frustration speaking, but I don't understand how my mail host ca…

Mostly, it doesn’t even seem to matter whether you agree or not. Inevitably you end up receiving affiliate emails regardless.

Re: A/B testing gets misused to juice metrics in the short term

#172

Post by 3pt14159 » Tue, Jul 12, 2022, 4:02 PM UTC

jaggederest wrote: post #84

axg11 wrote: post #47

A/B testing is local optimization. It should only be done on a mature(-ish) product when you have given up on finding a global minimum. Running experiments and A/B tests are popular because it is _guaranteed_ to give you signal. If you have a large engineering team and you're not sure how to filter the quality of results, gating everything through A/B tests is a well understood methodical way to ensure only positive…

> Running experiments and A/B tests are popular because it is _guaranteed_ to give you signal. If you have a large engineering team and you're not sure how to filter the quality of results, gating everything through A/B tests is a well understood methodical way to ensure only positive work makes it way through. It's almost guaranteed to ensure only false positive work makes its way through. If you're picking 0.05 as…

Even checking A/A tests won't surface all the issues. A proper A/B test is one that samples over a long enough time to adjust to the true audience of the service.

For example, imagine a costume shop that ran a couple dozen A/B tests over the summer. Those results may look statistically significant. They may even stand up against the A/A test. But people that buy costumes in the summer are very, very different than people that buy them in October, and if 90% of the store's business is in the run up to halloween, then all these micro optimizations could actually make your total business performance worse.

I'm a A/B testing skeptic too, though I admit they have a time and a place. My favourite are ones that can be reasoned about as actual hypotheses. This usually involves some degree of data analysis or segmentation. For example, increasing font sizes may boost conversion, and a later analysis shows that this was almost solely a lift in conversion rates amongst the 45+ cohort. The data in this case isn't just blindly driving design decisions, it's helping inform the staff on how to better design in the future for the audience we have.

Re: A/B testing gets misused to juice metrics in the short term

#173

Post by pjc50 » Tue, Jul 12, 2022, 4:08 PM UTC

jollybean wrote: post #32

verve_rat wrote: post #24

Earlier quoted context omitted.

Calling that bigotry is offensive to anyone that has been subjected to actual bigotry. You should be ashamed.

Quoted post unavailable.

The problem here is the use of identity "I AM an MBA" rather than mere posession "I have an MBA" or even "I have been awarded an MBA". You've made a certificate into part of your identity and are getting into fights on that basis.

"Engineer" can at least be a verb. I don't think MBA can?

Re: A/B testing gets misused to juice metrics in the short term

#174

Post by Lutger » Tue, Jul 12, 2022, 4:09 PM UTC

drewcoo wrote: post #70

costcofries wrote: post #57

Earlier quoted context omitted.

"To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. " Your perspective is extremely short-sighted. A/B testing can result in this type of behaviour but that's just poor A/B testing. Good A/B testing focuses on removing distractions from the experience and helping users derive…

"You're doing it, but in some subtle but very important way that's not at all obvious to you, you're doing it wrong." How many tech startup patterns fit that? That's a sign that either the pattern does not generalize well or it's snake oil.

I don't think so. A/B testing is just not a cure-all and alternative for good design, that is all. I mean, if the pattern is 'don't bother thinking too much about ux, just A/B test everything in PROD to death', then yeah, you are right.

However, there are many products which allow users to enter into a beta or test group, where they are the willing subjects to their experiments (in exchange for the latest shiny new stuff). This has the aspect of consent, leaving the 'stable ring' free of such variability. The fact that google and many startups are not using such consent and offering stability, doesn't mean it can't be done or isn't done.

Re: A/B testing gets misused to juice metrics in the short term

#175

Post by ratww » Tue, Jul 12, 2022, 4:11 PM UTC

tbranyen wrote: post #108

EricMausler wrote: post #92

/rant AB testing is and always has been fish oil for management. The only things it can actually prove, are more easily identifiable by common sense. So wherever it actually works, it was probably a waste of time / overkill for evidence. - sincerely, a business analyst

Have to disagree. I've found plenty of issues that affect real production users through the use of AB testing. Problems that were small enough to escape review, testing, and reporting, but large enough to be stat-sig. They always lead to a bug, or issue with test vs control. I will always use AB testing for uncertain code in the future. I was skeptical when I first started writing AB tests, but they have proven their…

Sure, but that's not really A/B testing, those are more often called staged rollouts or progressive rollouts.

Re: A/B testing gets misused to juice metrics in the short term

#176

Post by HWR_14 » Tue, Jul 12, 2022, 4:17 PM UTC

10x-dev wrote: post #48

benja123 wrote: post #27

I say this a lot and I will keep saying it. Conversion != customer obsession. There is a place for A/B testing. It is necessary and can be extremely beneficial in helping your customers enjoy and use your product more successfully. The main issue is that people mix conversion with customer obsession! Whenever you work on a product or feature you should be asking yourself "Is this really good for my customer" - if the…

My 'favorite' silly thing PMs do is UX research studies (typically on 5-10 people) and essentially ask completely untrained people if we should go with X/Y or Z. It's a super-effective way of avoiding responsibility for product decisions ("the data suggest we should go with Y"). If only building good products were as easy as asking what customers think they want.

Either they're doing the UX research wrong or (more likely) you're misunderstanding the process. You don't ask them if you should do X/Y/Z. You ask them to do X in the program, and see that none of them can find widget Y which controls it because they keep clicking on widget Z.

It's about observing the users fumble through your UX when you know their motivation.

Re: A/B testing gets misused to juice metrics in the short term

#177

Post by bentcorner » Tue, Jul 12, 2022, 4:22 PM UTC

hbn wrote: post #33

I've talked about this on HN but I'll say it again. A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a bi…

Google recently shut off Hangouts on me with a flight (since several of my contacts reported that Hangouts worked fine for them).

It's kind of mind boggling they'd decide to do that - the replacement they direct me to (Google Chat) doesn't even have feature parity so I just dropped them and moved my social circle using Hangouts to a different app (since at this point they all faced the same problem and we decided on a different platform).

I'm really curious how the A/B testing for this went down - Google is willingly throwing customers away because somebody wants to pump numbers for a new app that is objectively worse than the old one.

At this point Google Maps is the only product that is keeping me with them, but even that one is beginning to wear thin.

Re: A/B testing gets misused to juice metrics in the short term

#178

Post by stalfosknight » Tue, Jul 12, 2022, 4:27 PM UTC

tnolet wrote: post #166

tnolet wrote: post #141

Earlier quoted context omitted.

You are correct. I will figure out two things. 1. Why our fancy expensive mailing/marketing tool (Intercom) does not spread these messages in a relaxed fashion (it should) 2. If of those five messages, maybe two were the obligatory "confirm your email" and standard "Welcome to Checkly, this is what we do" TL;DR we should not spam.

We checked our Intercom. We spammed. One user got six emails in a 5 day span. There were some separate initiatives going on. We didn't check the settings and current outgoing mail. We will change it. BTW: all of this was done without any bad intent. It's 100% us being stupid and not coordinating and being diligent enough.

You only get one chance to make a first impression.

Re: A/B testing gets misused to juice metrics in the short term

#179

Post by [deleted] » Tue, Jul 12, 2022, 4:34 PM UTC

mgoetzke wrote: post #59

hbn wrote: post #33

I've talked about this on HN but I'll say it again. A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a bi…

Recently they remove YouTube PiP on iOS. Then it came back to my device but not others in the family. We pay for YT Premium. This is beyond infurating

[deleted]

Re: A/B testing gets misused to juice metrics in the short term

#180

Post by bentcorner » Tue, Jul 12, 2022, 4:34 PM UTC

loriverkutya wrote: post #67

costcofries wrote: post #57

Earlier quoted context omitted.

"To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. " Your perspective is extremely short-sighted. A/B testing can result in this type of behaviour but that's just poor A/B testing. Good A/B testing focuses on removing distractions from the experience and helping users derive…

I never seen an A/B test where the goal was not to maximise profit for the company.

What would an A/B test where the goal was not to maximize profit even look like? The very act of creating an A/B test is because the worker wants to improve something in search of higher profits.