Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing gets misused to juice metrics in the short term

zumsteg.net

Post reply on HN
Print view View original

301–310 of 346 posts

Re: A/B testing gets misused to juice metrics in the short term

#301

Post by dotecoffee »

loriverkutya wrote: post #67
costcofries wrote: post #57

Earlier quoted context omitted.

"To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. " Your perspective is extremely short-sighted. A/B testing can result in this type of behaviour but that's just poor A/B testing. Good A/B testing focuses on removing distractions from the experience and helping users derive…

I never seen an A/B test where the goal was not to maximise profit for the company.

Have you ever seen "you've already bought this" on Amazon? Highly tested, reduced sales. Overruled by judgement. Stayed on the site. AB test results don't mean you leave your judgement behind.

Re: A/B testing gets misused to juice metrics in the short term

#302

Post by fairity »

Yes, it's possible to use A/B testing in a short-sighted fashion. Yes, I'm sure there are plenty of examples thereof.

No, that doesn't mean A/B testing is inherently short-sighted. It's entirely possible to measure long-term secondary effects of an A/B test. Just save a record of treatment groups, and remember to come back and compare long-term metrics like LTV down the road. We do this all the time at my startup, and of the dark patterns that we've tested, we rarely see a long-term negative impact on LTV that outweighs the positive conversion rate impact.

If you want to make a valid argument against dark patterns (which is basically what 90% of this thread is trying to do), it's unlikely to be grounded in efficacy. This is coming from a business owner who spends seven figures monthly on advertising, constantly split tests, and is heavily invested in only making decisions that are in the long-term interest of the business.

Re: A/B testing gets misused to juice metrics in the short term

#304

Post by civilized »

ThalesX wrote: post #272
HWR_14 wrote: post #176

Earlier quoted context omitted.

Either they're doing the UX research wrong or (more likely) you're misunderstanding the process. You don't ask them if you should do X/Y/Z. You ask them to do X in the program, and see that none of them can find widget Y which controls it because they keep clicking on widget Z. It's about observing the users fumble through your UX when you know their motivation.

> It's about observing the users fumble through your UX when you know their motivation. Some time ago we did such a test. We called 10 customers to our offices and had them do some flows in the application. They didn't fumble. They pretty much did what they had to do and left positive reviews. That whole thing got scrapped because consultants convinced our CEO that qualitative data is not good for global scoped start…

Data analysis is the lowest common denominator of business thinking: the simplest, easiest thing that feels meaningful and objective. Anybody can sum up two lists of numbers in Excel and see which one is bigger.

Re: A/B testing gets misused to juice metrics in the short term

#305

Post by morelandjs »

I work in this field. Author makes great points. Here are some additional thoughts: * Don't use bad optimization to discredit all optimization * Incremental A/B testing assumes that all accumulated features are independent with no interactions, which is almost assuredly a poor assumption as a website grows and space becomes limited * A/B testing is the only reliable tool to test hypotheses; causal inference is great but not a substitute * Incremental A/B testing (small changes) should be periodically coupled with mutational A/B testing (large changes) to tunnel from local optima to global optima

Re: A/B testing gets misused to juice metrics in the short term

#306

Post by hahajk »

dspillett wrote: post #203
bottled_poe wrote: post #171

Earlier quoted context omitted.

Mostly, it doesn’t even seem to matter whether you agree or not. Inevitably you end up receiving affiliate emails regardless.

This is why every company gets a different address for me. If junk starts coming in, that address is blocked, and I stop doing business with that company (if I haven't already). This sometimes falls foul of spammers adding some random addresses of the form blahblah@mycatchall.domain.tld or @mycatchall.domain.tld into their lists, but that hasn't happened often enough to be a problem. That it isn't much of a problem s…

You don't have to use the company name in the email address, just use a unique email. If you start getting spam on a certain email address, search your email archives to see what company was associated with that address to link it back to them.

Re: A/B testing gets misused to juice metrics in the short term

#307

Post by earthmancash »

This post is right up my alley, as I'm a.) the CEO of an AB experimentation platform www.geteppo.com and b.) I'm an Airbnb alum where we rolled out an analogous feature that labeled a listing as "this is a rare find!"

And the funny thing is, I agree with this article. Both the content and the heading of this hackernews article:

1. Notification/scare spam can have long term retention ramifications. The previous generation of experiment platforms made long term metrics literally impossible to read. But now companies can use holdout groups and long term metrics like retention to give more clarity.

2. Even if you can read long term metrics that include retention, the scare/notification spam could lead to less word of mouth growth. For travel, I am guessing that you will be swayed to drive WoM growth more by differentiated inventory, reliable service, and cost, so maybe it's just merely annoying but not a risk to the business.

3. Notice that Airbnb's UI is very, very different from Booking, Expedia. We made a conscious choice to always make sure Airbnb came across as a "sincerely helpful friend" as a booking platform. An AB experiment showing that metrics improve doesn't mean that you have to launch it. You can look at those results and say, "this metric lift isn't worth how ugly it's making my site", and that's a completely valid choice. (a choice we made often at Airbnb)

Re: A/B testing gets misused to juice metrics in the short term

#308

Post by geoduck14 »

musicale wrote: post #298
tnolet wrote: post #126

Earlier quoted context omitted.

Hey CSMastermind, I'm founder at Checkly and I got a ping we were mentioned. We do send out some "getting started" messages on autopilot. We also did a product launch Thursday and then our regular changelog on Monday. That probably was overwhelming. If you could email me on tim -at- checklyhq -dot- com I will track down if we hit the the spam cannon too hard.

> if we hit the the spam cannon too hard Maybe consider, oh I don't know, not deploying a spam cannon in the first place? The good news is that this sort of thing is enough to trigger the spam filter in my email program so I'll never see it. Companies have this astonishing arrogance assuming they should be able to interrupt actual humans with robot messages from noreply@crapcorp.com - typically marketing messages tha…

>Maybe consider, oh I don't know, not deploying a spam cannon in the first place?

I'm not at all related to the people you are talking to, but I am a dev who has accidentally built spam cannons before. It is surprisingly easy to spam people - you need to deliberately build in checks - and put them in the right places to prevent spam cannons.

Re: A/B testing gets misused to juice metrics in the short term

#310

Post by geoduck14 »

stalfosknight wrote: post #178
tnolet wrote: post #166

Earlier quoted context omitted.

We checked our Intercom. We spammed. One user got six emails in a 5 day span. There were some separate initiatives going on. We didn't check the settings and current outgoing mail. We will change it. BTW: all of this was done without any bad intent. It's 100% us being stupid and not coordinating and being diligent enough.

You only get one chance to make a first impression.

>You only get one chance to make a first impression.

Well, this person had 6 first impressions.

Post reply on HN