Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing gets misused to juice metrics in the short term

zumsteg.net

Post reply on HN
Print view View original

211–220 of 346 posts

Re: A/B testing gets misused to juice metrics in the short term

#211

Post by corrral »

jl6 wrote: post #202
corrral wrote: post #73

Earlier quoted context omitted.

> Every app update was a risk of the app rearranging itself, or features appearing/disappearing. This shit drives my parents insane. Me too, when I have to help them. I've had to spend tens of seconds looking at a major screen in the phone app , of all things, to figure out WTF I'm looking at so I could help them figure out what was up. Re-arranged every update (or new phone) for absolutely no reason, terrible afford…

If you put a dollar value on your own time spent providing training and tech support to your parents, the iPhone options start to look much cheaper.

I've considered just getting them one but that's another new interface for them to learn. Though one that's much more stable across years.

Re: A/B testing gets misused to juice metrics in the short term

#212

Post by mcrad »

loriverkutya wrote: post #67
costcofries wrote: post #57

Earlier quoted context omitted.

"To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. " Your perspective is extremely short-sighted. A/B testing can result in this type of behaviour but that's just poor A/B testing. Good A/B testing focuses on removing distractions from the experience and helping users derive…

I never seen an A/B test where the goal was not to maximise profit for the company.

Oh yeah? I've never seen a marketing department with a decent understanding of microeconomics. Most in fact are trying to maximize budget ie. expense.

Re: A/B testing gets misused to juice metrics in the short term

#213

Post by tshaddox »

jl6 wrote: post #198
tshaddox wrote: post #191

Earlier quoted context omitted.

The thing is, you haven’t really shown that these sites aren’t successfully optimizing for conversions. Couldn’t it very well be the case that UI which annoys some high-intent users by interrupting them or adding steps to the checkout process also increases overall conversions?

True, A could be “annoy users” and B could be “don’t annoy users”, and A could perform better overall, but in this framework you might be missing C which is “annoy users except those already deep in the funnel”.

My point is more that there can be two groups of users with mutually exclusive desires, and it can be practical to choose to satisfy one group over the other based on your particular goals. There's not always some monotonically increasing function where you can over time satisfy a higher and higher portion of your users.

Re: A/B testing gets misused to juice metrics in the short term

#214

Post by fisf »

adrr wrote: post #193
hbn wrote: post #33

I've talked about this on HN but I'll say it again. A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a bi…

We AB tested a performance enhancement to our frontend web app to show that speed had significant benefits to the business. we use the results justify the investment cost. We spent the next six months working on making the site faster because of it. It is a tool. How would you measure things without ab testing?

The fact that you need an A/B test to demonstrate that frontend performance has an impact (on user experience first and foremost) in 2022 speaks volumes.

Re: A/B testing gets misused to juice metrics in the short term

#215

Post by jollybean »

Eddy_Viscosity2 wrote: post #85
jollybean wrote: post #14

Earlier quoted context omitted.

This is just casual bigotry.

I'll say that there are managers out there who do fantastic and important work, but they are the ones that have a build & sustain perspective. If you feel offended by this comment, its because you are in the squeeze & exploit group and you know it.

Wow, this is some really strange gaslighting.

I'm an Engineer and an MBA, very proud of both designations, and proud of almost all of my colleagues in both of those fields.

I'm a better person for knowing the people I worked and studied with.

I'm not hugely offended by the OP comment (or even yours), it's not a big deal, rather I'm pointing out that it is straight up a kind of misplaced bigotry - maybe not the best word, but it's correct. It's more ridiculous than anything.

Just re-read what you wrote: "because you are in the squeeze & exploit group and you know it"

Seriously? What the F is your problem? Why would you even conceive to write that to a random commenter on HN?

Do you folks not see this weirdly dark and perverse cynicism coming out here? What's wrong with you people?

I think maybe there is an odd, intellectually lazy thing happening whereby some people, possibly lacking the understanding of a lot of the mechanics of the 'business world', and knowing that 'bad business people exist' ... just want to throw it onto 'MBAs' for some strange reason, not understanding how odd and misinformed that rationalization is. It's really weird. Guys, stop this, it's just misinformed.

Re: A/B testing gets misused to juice metrics in the short term

#216

Post by aixi »

dang wrote: post #206
aixi wrote: post #53

Earlier quoted context omitted.

You're being responded by a bunch of resentful engineers, but you're correct; it's the definition of bigotry.

Flamewar comments will get you banned here, regardless of how wrong someone else is or you feel they are. If you'd please review https://news.ycombinator.com/newsguidelines.html and stick to the rules when posting here, we'd appreciate it.

yeah, my bad

Re: A/B testing gets misused to juice metrics in the short term

#217

Post by jcelerier »

costcofries wrote: post #57
hbn wrote: post #33

I've talked about this on HN but I'll say it again. A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a bi…

"To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. " Your perspective is extremely short-sighted. A/B testing can result in this type of behaviour but that's just poor A/B testing. Good A/B testing focuses on removing distractions from the experience and helping users derive…

> Your perspective is extremely short-sighted.

it's the perspective of the normal users.

every time i'm using a website and it does not behave exactly the same than for other people or I notice some AB testing, in my head it goes "who the fuck these people think they are?". The computing experience must be consistent and repeatable. If I wanted something that can change depending on the current position of the stars I'd ask another human, not a computer.

A/B testing can result in this type of behaviour but that's just poor A/B testing.

Re: A/B testing gets misused to juice metrics in the short term

#218

Post by Spooky23 »

dwighttk wrote: post #189
bottled_poe wrote: post #171

Earlier quoted context omitted.

Mostly, it doesn’t even seem to matter whether you agree or not. Inevitably you end up receiving affiliate emails regardless.

Yeah my mortgage company sent me a letter saying to opt out of affiliate marketing Emails or snail mail I have to send them a letter requesting it. This was 4 months after they bought my mortgage so the most the letter woulda done is stop them after selling my info for 4 months.

I had a coworker who sent those letters as a side hustle. He had a few different ones and would send the letters certified mail. Companies are very poor at compliance, and certain violations allow you to sue the company.

Re: A/B testing gets misused to juice metrics in the short term

#219

Post by aaaaaaaaaaab »

dspillett wrote: post #203
bottled_poe wrote: post #171

Earlier quoted context omitted.

Mostly, it doesn’t even seem to matter whether you agree or not. Inevitably you end up receiving affiliate emails regardless.

This is why every company gets a different address for me. If junk starts coming in, that address is blocked, and I stop doing business with that company (if I haven't already). This sometimes falls foul of spammers adding some random addresses of the form blahblah@mycatchall.domain.tld or @mycatchall.domain.tld into their lists, but that hasn't happened often enough to be a problem. That it isn't much of a problem s…

You can just use [id]-[sha1 hmac]@domain.tld

The id could be anything, and the SHA1 HMAC takes 32 characters in base32 (which is an email-address-safe encoding). Then just configure your spamfilter to reject any address where the HMAC doesn’t check out.

Of course, the drawback is that you’ll need a computer to generate a new address… At which point you may as well store an explicit whitelist of valid addresses.

Re: A/B testing gets misused to juice metrics in the short term

#220

Post by joe_the_user »

jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

Let me propose a different possibility.

Suppose the site isn't concerned about the sale very much at all?

Suppose the thing that the site uses to reel people in, is a good deal that isn't very profitable to the site but what the site then tries to sell is a very profitable near-scam/ripoff. Scaring off half the ordinary customers becomes worth it to get even 10% of the customers buying the scam.

What seems like "poor optimization" can easily optimization for something and could be seen as "the scammification of the web".

Post reply on HN