Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing gets misused to juice metrics in the short term

zumsteg.net

Post reply on HN
Print view View original

181–190 of 346 posts

Re: A/B testing gets misused to juice metrics in the short term

#181

Post by bluGill »

nemo44x wrote: post #150
d23 wrote: post #123

Earlier quoted context omitted.

There was a point a few years ago where you could not see a single piece of user-generated content above the fold on the reddit home page. A bunch of teams had jockeyed for having their little carousels and banners put on top, and of course, metrics were always cited. I left a screenshot in slack and it ended up causing a couple of teams to have to roll back their widgets, but it always baffled me that we were able t…

> but it always baffled me that we were able to focus so much on the individual trees of metric optimization that we would miss the forest to that extent Always look to the decision maker's incentives and you'll almost always discover why things are the way they are. And often, to your point, there's an aspect of tunnel vision associated with it because considering the bigger picture is difficult as a company grows a…

The best metric is the end of year bonus, tided to yearly total company financial results, but that only gets measured once per year. I can measure many things on every transaction, but how they in total work out to my end of year bonus and paycheck are much harder to see.

Of course if my bonus is some small KPI I can optimize that at the expense of overall performance.

Re: A/B testing gets misused to juice metrics in the short term

#182

Post by b112 »

quickthrower2 wrote: post #161
b112 wrote: post #125

Earlier quoted context omitted.

I'm not a big fan of ipv6, but fan or not, I bet if all spammy mailchimp type provider IP ranges were confiscated and freed, we'd be in ipv4 land for another 20 years. And as a second thought, the way China amd Russia are going, maybe we should just reclaim all their ipv4 addresses, and just give each country 1 IP, they can proxy through it on their end.

Then say goodbye to the internet and hello to a mesh of country-specific networks. Finally, true decentralisation!

? That's my point, as this is China already, and Russia is not far behind.

Re: A/B testing gets misused to juice metrics in the short term

#183

Post by selfhoster11 »

drewcoo wrote: post #70
costcofries wrote: post #57

Earlier quoted context omitted.

"To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. " Your perspective is extremely short-sighted. A/B testing can result in this type of behaviour but that's just poor A/B testing. Good A/B testing focuses on removing distractions from the experience and helping users derive…

"You're doing it, but in some subtle but very important way that's not at all obvious to you, you're doing it wrong." How many tech startup patterns fit that? That's a sign that either the pattern does not generalize well or it's snake oil.

> "You're doing it, but in some subtle but very important way that's not at all obvious to you, you're doing it wrong."

Doesn't that exact statement describe how Agile (or many other concepts, really) is used in a lot of companies? There's nothing wrong with it in principle, but practice is all over the place.

Re: A/B testing gets misused to juice metrics in the short term

#184

Post by selfhoster11 »

teakettle42 wrote: post #120
koheripbal wrote: post #107

Earlier quoted context omitted.

Do people honestly think that Apple doesn't do A/B testing?

Apple unequivocally does not do active A/B testing on their users by changing applications or the operating system out from under them.

It's almost like they use a dedicated focus/beta testing group or something, instead of making all their users join in.

Re: A/B testing gets misused to juice metrics in the short term

#186

Post by dwighttk »

enlyth wrote: post #49
jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

Speaking of 'sign up to our newsletter', one of the latest dark patterns I've found that astounded me was adding a checkbox to the login form [0], where you'd normally expect the 'remember me' checkbox to be. You almost click it out of muscle memory if you don't read what it says. [0] https://i.postimg.cc/HW89hs7r/Screenshot-2022-07-12-145957.p...

Heh. I’ve stopped clicking remember me boxes because they never work.

Re: A/B testing gets misused to juice metrics in the short term

#187

Post by selfhoster11 »

corrral wrote: post #73
hbn wrote: post #33

I've talked about this on HN but I'll say it again. A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a bi…

> Every app update was a risk of the app rearranging itself, or features appearing/disappearing. This shit drives my parents insane. Me too, when I have to help them. I've had to spend tens of seconds looking at a major screen in the phone app , of all things, to figure out WTF I'm looking at so I could help them figure out what was up. Re-arranged every update (or new phone) for absolutely no reason, terrible afford…

At one point I was trying to set up my grandma with a popular video calling app on a dedicated device so we could stay in touch.

Before I could give her the freshly grandparent-proofed device, said video calling app upgraded on my parents' PC first and changed literally every single element of the UI beyond recognition. To someone the age of my grandma, that would be literally like bricking the device remotely, because none of the buttons would look the same, and she would not be able to work out how to use the new interface.

STOP CHANGING THINGS! Even if the new UI is better (debatable), some people just like or rely on a particular layout to operate the device or app. Don't rearrange without giving a ~permanent setting to use the old layout.

Re: A/B testing gets misused to juice metrics in the short term

#188

Post by adrr »

loriverkutya wrote: post #67
costcofries wrote: post #57

Earlier quoted context omitted.

"To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. " Your perspective is extremely short-sighted. A/B testing can result in this type of behaviour but that's just poor A/B testing. Good A/B testing focuses on removing distractions from the experience and helping users derive…

I never seen an A/B test where the goal was not to maximise profit for the company.

Isn't that any goal of an organization? Everything ladders up to the success of the company. Especially if you're a corporation where you have fiduciary responsibility to the shareholders.

AB testing is beyond UI. It could be AB testing algorithms for recommendations or search results. When i worked for a CPG company, we would AB test the sample sizes in the boxes. To answer the question what is right amount of product for a sample. We would test shipping to speed, was it worth the upgrade to expedited shipping in terms of LTV to justify the extra cost.

Re: A/B testing gets misused to juice metrics in the short term

#189

Post by dwighttk »

bottled_poe wrote: post #171
eloisius wrote: post #103

Earlier quoted context omitted.

Email marketing in general blows my mind. Marketers typically have absolutely no respect for consent, and the costs are completely borne by the recipient. The whole industry depends on dark patterns, shady list sharing, and scraping your email to add it to their lists despite you having no relationship with them. I know it's not simple, and it's just my frustration speaking, but I don't understand how my mail host ca…

Mostly, it doesn’t even seem to matter whether you agree or not. Inevitably you end up receiving affiliate emails regardless.

Yeah my mortgage company sent me a letter saying to opt out of affiliate marketing Emails or snail mail I have to send them a letter requesting it. This was 4 months after they bought my mortgage so the most the letter woulda done is stop them after selling my info for 4 months.

Re: A/B testing gets misused to juice metrics in the short term

#190

Post by commandlinefan »

> Next to some hotels, a message that supply was limited.

It's also worth noting that there's no way in hell they actually know that with any sort of precision. No GDS has proper up-to-date knowledge of bookings from all the various sources that hotel reservations actually go through (they overbook airline flights). What they're really saying is that the small inventory of rooms that are reserved for them to book exclusively are almost gone.

Post reply on HN