Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing gets misused to juice metrics in the short term

zumsteg.net

Post reply on HN
Print view View original

331–340 of 346 posts

Re: A/B testing gets misused to juice metrics in the short term

#331

Post by izacus »

daniel_reetz wrote: post #261
teakettle42 wrote: post #120

Earlier quoted context omitted.

Apple unequivocally does not do active A/B testing on their users by changing applications or the operating system out from under them.

I can't think of an Apple engineer who would make such a definitive statement on behalf of the company, in public. I also can't think of anyone working there who would have definitive knowledge across all apps and software, because disclosure would prevent such knowledge. Apple introduced A/B testing to Testflight in 2017 and more A/B testing to the App store this year. Source: I worked at Apple, but not in software.

Great, but what exactly does that mean for this conversation and how users percieve Apple products?

Re: A/B testing gets misused to juice metrics in the short term

#332

Post by Jenk »

jerf wrote: post #54
Jenk wrote: post #51

Earlier quoted context omitted.

> I'm signalling interest in product X as hard as I can, and you've AB tested that this is a great time to jangle your keys over there instead? If I may... I have seen data from a big retailer that shows any user that doesn't immediately purchase an item, is actually not that interested in the product on the screen. If a customer is going to buy something, they will do it promptly. Anyone else is just browsing. YMMV,…

In this case, what I'm referring to is: 1. Clicked on page. 2. Took maybe 10 seconds to take in what is "above the fold". 3. Scrolled down to see what else there is. 4. BAM! Popup triggered by scrolling down. While I understand what you're getting at, they do not yet have the info to know that I'm browsing or whatever. They were so excited about their stupid popup that they didn't even get that far. I will say, gener…

They know you clicked on a page that is for a given (set of?) product(s). They know you stayed on that page for longer than a moment, then scrolled down.

They may also have data that >90% of customers who make a purchase choose/click in They could interpret all of that as you inspected the product(s) on the page, didn't choose/click one, and _now_ is the best time to prompt you with suggestions to maintain engagement in >75% of instances.

You could also be one of the lucky few customers in a sacrificial control group unknowingly testing their new shitty popup to measure its impact on sales.

Whilst I can't say for certain that any of this is your case, I absolutely guarantee there are many retail sites that do this - I've worked at several now - and this is bread-and-butter material to data teams working with customer engagement.

Re: A/B testing gets misused to juice metrics in the short term

#333

Post by dan_mctree »

goodside wrote: post #260
Philadelphia wrote: post #228

Earlier quoted context omitted.

OkCupid has completely destroyed its interface and utility, so whatever they’re doing doesn’t seem to be working anymore.

I left in 2015, as soon as it became apparent the party was over. OkCupid went downhill for a lot of reasons, but overly aggressive A/B testing wasn't one of them.

What do you think the reasons were?

Re: A/B testing gets misused to juice metrics in the short term

#334

Post by oba2311 »

treis wrote: post #289
josefx wrote: post #111

Earlier quoted context omitted.

But does AB testing provide the only or even best metric for that? It probably is the cheapest way requiring the least engagement with the lab mice.

AB testing is how you isolate a change and measure the impact. It's the only real way to be able to associate cause and effect. Best you can do otherwise is measure something over time while making changes. You can try and correlate changes with outcomes but it's hard to be sure the change is what drove the outcome.

That sounds pretty accurate. Anyone who's gone through some econometric classes would know that split testing is (almost) the only way to get to (close to) causality - all the rest is black magic called correlation which can also lead you to the conclusion that Nicolas Cage's movies lead to people drowning:

https://www.thenationalnews.com/uae/nicolas-cage-movies-link....

https://www.wnycstudios.org/podcasts/otm/articles/spurious-c...

Re: A/B testing gets misused to juice metrics in the short term

#335

Post by InCityDreams »

Jenk wrote: post #51
jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

> I'm signalling interest in product X as hard as I can, and you've AB tested that this is a great time to jangle your keys over there instead? If I may... I have seen data from a big retailer that shows any user that doesn't immediately purchase an item, is actually not that interested in the product on the screen. If a customer is going to buy something, they will do it promptly. Anyone else is just browsing. YMMV,…

>I have seen data from a big retailer that shows any user that doesn't immediately purchase an item, is actually not that interested in the product on the screen.

Fuck that. Unless it's an emergency (in which case I'll go to a shop), anything i purchase online is carefully considered, sometimes over several weeks. My revolving user-agent, vpn etc may give the illusion I'm not interested....but i am indeed, just browsing...

Re: A/B testing gets misused to juice metrics in the short term

#336

Post by kthejoker2 »

tnolet wrote: post #258
oxfordmale wrote: post #254

Earlier quoted context omitted.

tnolet, if I get five emails from a company with 48 hours, I will set up a spam rule for them. If I really need to get that much help to get started, your UX design is likely not very good.

You are correct. I’m the same. Our emails are even pretty good. Our UX is pretty damn good. Somehow we dropped the ball here.

When you fix it, write up a post mortem so others can lesrn from your mistake!

Since it's distressingly common.

Re: A/B testing gets misused to juice metrics in the short term

#337

Post by GoblinSlayer »

b112 wrote: post #125
eloisius wrote: post #103

Earlier quoted context omitted.

Email marketing in general blows my mind. Marketers typically have absolutely no respect for consent, and the costs are completely borne by the recipient. The whole industry depends on dark patterns, shady list sharing, and scraping your email to add it to their lists despite you having no relationship with them. I know it's not simple, and it's just my frustration speaking, but I don't understand how my mail host ca…

I'm not a big fan of ipv6, but fan or not, I bet if all spammy mailchimp type provider IP ranges were confiscated and freed, we'd be in ipv4 land for another 20 years. And as a second thought, the way China amd Russia are going, maybe we should just reclaim all their ipv4 addresses, and just give each country 1 IP, they can proxy through it on their end.

One address should be enough for everyone? Bill Gates just reached zen.

Re: A/B testing gets misused to juice metrics in the short term

#338

Post by ryanmcbride »

zx8080 wrote: post #312
ryanmcbride wrote: post #268

Earlier quoted context omitted.

That's why the only email subscription service on my site is completely transparent and details exactly what we're storing, and it's impossible to click by accident. Everything I do for myself I try to build like a service I'd like to use myself. But the second managers or marketers are involved it's all out the window. I remember early in my career my boss had me add every email included in a TED booklet to their ma…

> That's why the only email subscription service on my site is completely transparent Sorry, but honestly speaking even checking your profile does not reveal what your web site is. Probably you are marketing person from your words, but that's not clear. Hope this feedback helps!

That's intentional, I'm not trying to plug my website I was just talking about my approach in non-specific terms. I'm a dev.

Re: A/B testing gets misused to juice metrics in the short term

#339

Post by slotrans »

The way A/B testing is practiced by actual people in the actual world, it is fundamentally broken.

No one EVER tests for mean-reversion over time.

17 years I've seen companies do A/B tests. I doubt I've seen a single convincing, durable result the whole time.

Re: A/B testing gets misused to juice metrics in the short term

#340

Post by slotrans »

jaggederest wrote: post #84
axg11 wrote: post #47

A/B testing is local optimization. It should only be done on a mature(-ish) product when you have given up on finding a global minimum. Running experiments and A/B tests are popular because it is _guaranteed_ to give you signal. If you have a large engineering team and you're not sure how to filter the quality of results, gating everything through A/B tests is a well understood methodical way to ensure only positive…

> Running experiments and A/B tests are popular because it is _guaranteed_ to give you signal. If you have a large engineering team and you're not sure how to filter the quality of results, gating everything through A/B tests is a well understood methodical way to ensure only positive work makes it way through. It's almost guaranteed to ensure only false positive work makes its way through. If you're picking 0.05 as…

The first company I worked for, and also the first company I saw A/B tests at, once ran an A/A test because someone was a bit skeptical about some of the test results that had been claimed.

Predictably, whatever metric we were watching on it (probably conversion) swung wildly to either side over the first few days. The look on some of the product managers' faces was pretty great. After about 2 weeks, it settled into a steady state where each "version" performed equally (measured cumulatively, so just large numbers in action).

The conclusion from this exercise was...

"It takes 2 weeks."

¯\_(ツ)_/¯

Post reply on HN