Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing gets misused to juice metrics in the short term

zumsteg.net

Post reply on HN
Print view View original

311–320 of 346 posts

Re: A/B testing gets misused to juice metrics in the short term

#311

Post by civilized »

morelandjs wrote: post #305

I work in this field. Author makes great points. Here are some additional thoughts: * Don't use bad optimization to discredit all optimization * Incremental A/B testing assumes that all accumulated features are independent with no interactions, which is almost assuredly a poor assumption as a website grows and space becomes limited * A/B testing is the only reliable tool to test hypotheses; causal inference is great…

Whenever someone says "X tends to be bad", people will show up to say "not all X is bad, only bad X is bad" which is really beside the point.

Re: A/B testing gets misused to juice metrics in the short term

#312

Post by zx8080 »

ryanmcbride wrote: post #268
eloisius wrote: post #103

Earlier quoted context omitted.

Email marketing in general blows my mind. Marketers typically have absolutely no respect for consent, and the costs are completely borne by the recipient. The whole industry depends on dark patterns, shady list sharing, and scraping your email to add it to their lists despite you having no relationship with them. I know it's not simple, and it's just my frustration speaking, but I don't understand how my mail host ca…

That's why the only email subscription service on my site is completely transparent and details exactly what we're storing, and it's impossible to click by accident. Everything I do for myself I try to build like a service I'd like to use myself. But the second managers or marketers are involved it's all out the window. I remember early in my career my boss had me add every email included in a TED booklet to their ma…

> That's why the only email subscription service on my site is completely transparent

Sorry, but honestly speaking even checking your profile does not reveal what your web site is. Probably you are marketing person from your words, but that's not clear.

Hope this feedback helps!

Re: A/B testing gets misused to juice metrics in the short term

#313

Post by m463 »

jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

Amazon does this in a horrible way.

1) I open a product in a tab. I click "add to cart" and a "related products" sidebar slides in. I close the tab in annoyance.

However, some items exhibit a similar pattern, EXCEPT...

2) I open a product in a tab. I click "add to cart" and a stupid extended warranty sidebar slides in. I close the tab in annoyance.

The difference?

Item #1 gets added to my cart

Item #2 doesn't make it to the shopping cart.

Amazon just silently deleted my purchase.

I actually don't know when it dawned on me that this happened, but amazon lost money on me because I didn't buy certain things.

Re: A/B testing gets misused to juice metrics in the short term

#314

Post by mgraczyk »

manmal wrote: post #241
costcofries wrote: post #57

Earlier quoted context omitted.

"To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. " Your perspective is extremely short-sighted. A/B testing can result in this type of behaviour but that's just poor A/B testing. Good A/B testing focuses on removing distractions from the experience and helping users derive…

UX feedback is usually retrieved via user interviews. I personally have never heard from a product person „Let’s A/B test whether this is delightful“. And I think that’s because delightfulness or satisfaction is impossible to quantify in A/B tests. You only get to measure things like engagement, signup rates, retention etc. - cold hard taps on the screen, and no more. And I must say that I‘m glad that, right now, app…

This is not really true. You can pop up surveys in the app or on the site. They can be super non-intrusive. For small effects and large user bases, you simply cannot get useful information from UX studies.

Re: A/B testing gets misused to juice metrics in the short term

#315

Post by brianwawok »

fatnoah wrote: post #162
eloisius wrote: post #103

Earlier quoted context omitted.

Email marketing in general blows my mind. Marketers typically have absolutely no respect for consent, and the costs are completely borne by the recipient. The whole industry depends on dark patterns, shady list sharing, and scraping your email to add it to their lists despite you having no relationship with them. I know it's not simple, and it's just my frustration speaking, but I don't understand how my mail host ca…

> Email marketing in general blows my mind My favorite thing is when the companies outsource the email marketing, so that it has absolutely zero relevance. I've been using the same online tax preparer for 10 years, and I've had exactly zero refunds, yet their emails during tax season always let me know that "my refund is waiting".

The point of email marketing is to hit the right targets. If they overspray and hit 100000 wrong targets.., oh well, they lost 3 cents. So this overspray isn’t a sign that they are dumb, but they are effective.

Re: A/B testing gets misused to juice metrics in the short term

#316

Post by 8note »

costcofries wrote: post #57
hbn wrote: post #33

I've talked about this on HN but I'll say it again. A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a bi…

"To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. " Your perspective is extremely short-sighted. A/B testing can result in this type of behaviour but that's just poor A/B testing. Good A/B testing focuses on removing distractions from the experience and helping users derive…

A/B testing is always going to be distracting to your users though.

Sometimes they'll be in A, and sometimes in B.

The button moves or disappears or appears.

Your user does not get an experience they can rely on

Re: A/B testing gets misused to juice metrics in the short term

#317

Post by hazz99 »

MegaButts wrote: post #271
thatguy0900 wrote: post #226

Earlier quoted context omitted.

This reads exactly like parody corporatespeak jargon.

Clearly it's the users who are wrong with their stupid assumptions and we will correct them with our mandatory weekly update.

Assumptions about the users, not assumptions made by the users.

Re: A/B testing gets misused to juice metrics in the short term

#318

Post by ninkendo »

As a rule, I open sites I’ve never been to in a new tab, with my hand hovering over the Cmd and W keys. The moment I get an popup of any kind that obscures content offering me to sign up for a mailing list or offer me some bullshit coupon, my fingers come down and I close the tab. I really hope the A/B tests actually show a bounce here, but I doubt it because I’m actually closing the tab, not using the back button (too many sites hijack it) and I’m not sure they can detect that I’ve left. I also do it so quickly now that they probably don’t even get the signal that I saw the popup either.

Site owners: please stop doing this. You’re turning the web into a cesspit. You’re part of the problem.

Re: A/B testing gets misused to juice metrics in the short term

#319

Post by chunkyks »

_tom_ wrote: post #251
chunkyks wrote: post #115

"If a study came out that said deafening high-pitched noises increased conversion rates, we would all be bleeding from our ears by end of business tomorrow, right?" Netflix auto play? Is that you? You were a hateful idea, no one liked you, yet you stubbornly hung on for far too long

I'm convinced that Netflix uses "number of hours watched" as a success metric. Autoplay raises that. I'd pay twice as much to watch half as much quality programming, but that would tank what they think is a positive metric.

I'm pretty confident that that horrorshow was a case study in A/B testing: People can't decide what to watch and agitate over flicking through titles, so some bright spark had the idea of just starting stuff in the background to see if that helped.

It did! People started watching stuff sooner!

Mostly because it was so incredibly irritating that I'd start a show just to make the random autoplay cease. Of course, what it really meant was that I'd go to amazon and agonize over what to watch, but at least I was doing it at my own speed and not being harassed by noise autoplay.

Re: A/B testing gets misused to juice metrics in the short term

#320

Post by nemo44x »

the_gipsy wrote: post #231
nemo44x wrote: post #150

Earlier quoted context omitted.

> but it always baffled me that we were able to focus so much on the individual trees of metric optimization that we would miss the forest to that extent Always look to the decision maker's incentives and you'll almost always discover why things are the way they are. And often, to your point, there's an aspect of tunnel vision associated with it because considering the bigger picture is difficult as a company grows a…

Gervais principle says that senior leaders work for themselves, not the company, up to the point of working against the company.

That was an interesting rabbit hole I hadn’t been down before. Interesting!
Post reply on HN