Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing gets misused to juice metrics in the short term

zumsteg.net

Post reply on HN
Print view View original

151–160 of 346 posts

Re: A/B testing gets misused to juice metrics in the short term

#151

Post by marcosdumay »

jaggederest wrote: post #84
axg11 wrote: post #47

A/B testing is local optimization. It should only be done on a mature(-ish) product when you have given up on finding a global minimum. Running experiments and A/B tests are popular because it is _guaranteed_ to give you signal. If you have a large engineering team and you're not sure how to filter the quality of results, gating everything through A/B tests is a well understood methodical way to ensure only positive…

> Running experiments and A/B tests are popular because it is _guaranteed_ to give you signal. If you have a large engineering team and you're not sure how to filter the quality of results, gating everything through A/B tests is a well understood methodical way to ensure only positive work makes it way through. It's almost guaranteed to ensure only false positive work makes its way through. If you're picking 0.05 as…

Well, if you are running hundreds of tests with 0.05 p-value, you will get plenty of false ok A/A tests, and there isn't much of a reason to expect them to be correlated to actual signal on your A/B tests.

A/A tests do test your methodology as you said. But they do not fix a p-value one order of magnitude higher than it should be. (And yeah, I'm aware you know that, but your comment places them on the same context, so it got misleading.)

Re: A/B testing gets misused to juice metrics in the short term

#152

Post by propogandist »

eloisius wrote: post #103
enlyth wrote: post #49

Earlier quoted context omitted.

Speaking of 'sign up to our newsletter', one of the latest dark patterns I've found that astounded me was adding a checkbox to the login form [0], where you'd normally expect the 'remember me' checkbox to be. You almost click it out of muscle memory if you don't read what it says. [0] https://i.postimg.cc/HW89hs7r/Screenshot-2022-07-12-145957.p...

Email marketing in general blows my mind. Marketers typically have absolutely no respect for consent, and the costs are completely borne by the recipient. The whole industry depends on dark patterns, shady list sharing, and scraping your email to add it to their lists despite you having no relationship with them. I know it's not simple, and it's just my frustration speaking, but I don't understand how my mail host ca…

beware that Meta/FB and Tiktok scripts are among those that siphon off email data even before a web form is submitted

https://arstechnica.com/information-technology/2022/05/some-...

Re: A/B testing gets misused to juice metrics in the short term

#153

Post by kaoD »

tnolet wrote: post #126
CSMastermind wrote: post #72

Earlier quoted context omitted.

Recently I got an account on a developer tool (Checkly) because the company I joined uses it. I then got 5 different emails from them in a 48 hour period. Like I'm sure many users sign up then drop out of your funnel but I'm part of an organization that's a paying customer. I'm already going to use your stuff. What possible business benefit could there be to you spamming me? If anything you're risking the inverse - i…

Hey CSMastermind, I'm founder at Checkly and I got a ping we were mentioned. We do send out some "getting started" messages on autopilot. We also did a product launch Thursday and then our regular changelog on Monday. That probably was overwhelming. If you could email me on tim -at- checklyhq -dot- com I will track down if we hit the the spam cannon too hard.

I know this is well-intentioned and not an automated message, but I find it ironic that you managed to get an additional message to him over here.

Re: A/B testing gets misused to juice metrics in the short term

#154

Post by axus »

Jenk wrote: post #51
jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

> I'm signalling interest in product X as hard as I can, and you've AB tested that this is a great time to jangle your keys over there instead? If I may... I have seen data from a big retailer that shows any user that doesn't immediately purchase an item, is actually not that interested in the product on the screen. If a customer is going to buy something, they will do it promptly. Anyone else is just browsing. YMMV,…

Could the popup be a punishment for reading the fine print?

Re: A/B testing gets misused to juice metrics in the short term

#155

Post by jen20 »

koheripbal wrote: post #106
loriverkutya wrote: post #67

Earlier quoted context omitted.

I never seen an A/B test where the goal was not to maximise profit for the company.

This comment is way too broad and cynical. It reminds me of the sentiment I sometimes hear from my teenage children before I explain to them that the world isn't a Bernie Sander/Reddit gotcha soundbite and that reality is complicated.

> This comment is way too broad and cynical

Or, as anyone who has been on the internet since the 90s will tell you, bang on the damn money.

Re: A/B testing gets misused to juice metrics in the short term

#156

Post by notriddle »

ThalesX wrote: post #116
koheripbal wrote: post #106

Earlier quoted context omitted.

This comment is way too broad and cynical. It reminds me of the sentiment I sometimes hear from my teenage children before I explain to them that the world isn't a Bernie Sander/Reddit gotcha soundbite and that reality is complicated.

What would the other advantages of A|B testing be for a corporation instead of widening the funnel, increasting lifetime revenue or other such bottom-line focused goals?

Assuming you aren’t running a charity, increasing the bottom line is always your goal. The difference between a company that burns user goodwill in the process, and one that doesn’t, is the difference between short-term and long-term thinking.

The downside with hired testers is that they are unlikely to be a statistically representative sample of your target market. I don’t think this is actually a problem for most startups or new products being launched by major players, since a lot of UX issues early in product development will be obvious to anyone who isn’t the original developers, and the number of actual users you have won’t be big enough to serve as a statistically representative sample of your target market anyway.

Re: A/B testing gets misused to juice metrics in the short term

#157

Post by retcon »

nemo44x wrote: post #118
jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

A lot of this happens because different managers have different metrics/KPIs they are optimizing around and they all find "good places" to do things to help meet their goals. The secondary effects aren't considered. One managers quest for outperforming their goals comes at the expense of another managers goals.

The primary not secondary effect of random sampling is noise.

Re: A/B testing gets misused to juice metrics in the short term

#158

Post by jen20 »

jollybean wrote: post #56
grapeskin wrote: post #43

Earlier quoted context omitted.

MBAs aren’t something people are born into or end up through no choice of their own. Bigotry against MBAs is impossible. It’s like being a loan shark or pimp. People actively choose that life and society talks poorly about them because of the social harm they bring. There’s nothing stopping someone from washing their hands of any of those things and being productive.

Quoted post unavailable.

> MBA's are generally better communicators

"Convenience Fee" "Synergy" "Circle back to this" "Double-click on this issue"

Re: A/B testing gets misused to juice metrics in the short term

#159

Post by rightbyte »

amluto wrote: post #143

Reading this makes me think of the handful of sites, often targeted at professionals, that highly optimize for the experience of actually buying things. McMaster-Carr comes to mind. Their users shop there over and over, and McMaster wants to keep them. So you can find things for $2 or $2000, shipping prices are inoffensive, customer service is friendly but rarely needed, and there are minimal distractions on the way…

McMaster-Carr might be the single thing I miss the most from my time in the US. It is like ... stupid good. Their listings catalogization is like godlike compared to alternatives.

The Amazon or Google way of throwing all things into the bin and spew it out to the users is BS. We are saying we live in an information age but I firmly believe stuff were way better catalogizised back when it was done manually by paid gatekeepers.

Re: A/B testing gets misused to juice metrics in the short term

#160

Post by spookthesunset »

nemo44x wrote: post #118
jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

A lot of this happens because different managers have different metrics/KPIs they are optimizing around and they all find "good places" to do things to help meet their goals. The secondary effects aren't considered. One managers quest for outperforming their goals comes at the expense of another managers goals.

This is why any good org will make sure to observe all important KPI’s while doing an A/B test. If your “email signup” KPI went to the moon but tanked your “bought shit” metric… you should probably roll back.
Post reply on HN