Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing gets misused to juice metrics in the short term

zumsteg.net

Post reply on HN
Print view View original

121–130 of 346 posts

Re: A/B testing gets misused to juice metrics in the short term

#121

Post by svnpenn »

costcofries wrote: post #57
hbn wrote: post #33

I've talked about this on HN but I'll say it again. A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a bi…

"To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. " Your perspective is extremely short-sighted. A/B testing can result in this type of behaviour but that's just poor A/B testing. Good A/B testing focuses on removing distractions from the experience and helping users derive…

> Your perspective is extremely short-sighted

no, yours is. if some company wants to do some testing, they SHOULD PAY users for that. A/B testing is just exploiting users to get free testing.

Re: A/B testing gets misused to juice metrics in the short term

#122

Post by mschuster91 »

ssharp wrote: post #20
mschuster91 wrote: post #11

Earlier quoted context omitted.

> Assuming you know exactly what will make the customer experience best without actually testing it can also lead to a worse experience. For that you usually hire a market research company or do what they will do: take an interviewer, two cameras (one front-face, one top-hands) and hire an as-diverse-as-possible pool of test candidates that you then put through whatever workflow optimization you want to do. Then afte…

It's not really an either/or option. You can use testing to validate the changes stemming from market research. Having seen lots of site redesigns go horribly wrong due to 100% earnest people trying their best and utilizing the research that was afforded to the process, I always recommending incrementally testing into changes on high-value / high-risk applications, even when the "improvements" were backed by solid re…

The "or" was meant to be the distinction on who does the user testing - I've seen both in-house testing operations and outsourced ones. For small scale operations, it may actually be cheaper to run them in-house and only hire external testers... cameras are dirt cheap these days.

Hiring a market research company is usually worth it if you have a contract with them anyway (which gives you better rates on the testing) or lack someone on staff who knows how to deal with cameras.

Re: A/B testing gets misused to juice metrics in the short term

#123

Post by d23 »

nemo44x wrote: post #118
jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

A lot of this happens because different managers have different metrics/KPIs they are optimizing around and they all find "good places" to do things to help meet their goals. The secondary effects aren't considered. One managers quest for outperforming their goals comes at the expense of another managers goals.

There was a point a few years ago where you could not see a single piece of user-generated content above the fold on the reddit home page. A bunch of teams had jockeyed for having their little carousels and banners put on top, and of course, metrics were always cited.

I left a screenshot in slack and it ended up causing a couple of teams to have to roll back their widgets, but it always baffled me that we were able to focus so much on the individual trees of metric optimization that we would miss the forest to that extent.

Re: A/B testing gets misused to juice metrics in the short term

#124

Post by [deleted] »

jollybean wrote: post #56
grapeskin wrote: post #43

Earlier quoted context omitted.

MBAs aren’t something people are born into or end up through no choice of their own. Bigotry against MBAs is impossible. It’s like being a loan shark or pimp. People actively choose that life and society talks poorly about them because of the social harm they bring. There’s nothing stopping someone from washing their hands of any of those things and being productive.

Quoted post unavailable.

[deleted]

Re: A/B testing gets misused to juice metrics in the short term

#125

Post by b112 »

eloisius wrote: post #103
enlyth wrote: post #49

Earlier quoted context omitted.

Speaking of 'sign up to our newsletter', one of the latest dark patterns I've found that astounded me was adding a checkbox to the login form [0], where you'd normally expect the 'remember me' checkbox to be. You almost click it out of muscle memory if you don't read what it says. [0] https://i.postimg.cc/HW89hs7r/Screenshot-2022-07-12-145957.p...

Email marketing in general blows my mind. Marketers typically have absolutely no respect for consent, and the costs are completely borne by the recipient. The whole industry depends on dark patterns, shady list sharing, and scraping your email to add it to their lists despite you having no relationship with them. I know it's not simple, and it's just my frustration speaking, but I don't understand how my mail host ca…

I'm not a big fan of ipv6, but fan or not, I bet if all spammy mailchimp type provider IP ranges were confiscated and freed, we'd be in ipv4 land for another 20 years.

And as a second thought, the way China amd Russia are going, maybe we should just reclaim all their ipv4 addresses, and just give each country 1 IP, they can proxy through it on their end.

Re: A/B testing gets misused to juice metrics in the short term

#126

Post by tnolet »

CSMastermind wrote: post #72
jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

Recently I got an account on a developer tool (Checkly) because the company I joined uses it. I then got 5 different emails from them in a 48 hour period. Like I'm sure many users sign up then drop out of your funnel but I'm part of an organization that's a paying customer. I'm already going to use your stuff. What possible business benefit could there be to you spamming me? If anything you're risking the inverse - i…

Hey CSMastermind, I'm founder at Checkly and I got a ping we were mentioned. We do send out some "getting started" messages on autopilot. We also did a product launch Thursday and then our regular changelog on Monday. That probably was overwhelming. If you could email me on tim -at- checklyhq -dot- com I will track down if we hit the the spam cannon too hard.

Re: A/B testing gets misused to juice metrics in the short term

#127

Post by ben_w »

jollybean wrote: post #78
Bancakes wrote: post #45

Earlier quoted context omitted.

Quoted post unavailable.

I'm both a Software Engineer and an MBA, and I can assure you that you're wrong, and with a comedic, glib kind of 'Musk-ian' lack of self awareness that's weirdly common among technical types. I mean it's getting funny now, I was hoping that your comment was satire (!), alas, no.

You can only choose your own responses to what you experience, not other people's.

Engineering without professional leadership leads to wasted effort, solutions in search of problems, and also occasional successes such as Linux.

Leadership without engineering (or equivalent in other fields) is the B-Ark.

If you let yourself respond with anger, no matter how justified you feel, you're going to end up in the second group — leadership is a position of strength, and if there's one thing people like even less than sore losers, it is sore winners.

Re: A/B testing gets misused to juice metrics in the short term

#128

Post by Kaotique »

treis wrote: post #52
Kaotique wrote: post #3

AB testing shows zero respect to your customers. It is the equivalent of testing your theories on lab rats. Instead try to improve the customer experience, make better products, improve customer service.

>Instead try to improve the customer experience, make better products, improve customer service. Without a metric to say what is "better" and a method to measure it this is empty advice.

There are many other metrics that do not involve AB testing. You can just survey customer experience before, during and after a purchase for example. I never said to throw out all metrics.

With AB testing your are optimising for a specific outcome. Usually higher conversion. As pointed out in the article eventually you'll end up with a bunch of colourful buttons and scary texts that persuade the user to click. A lot of the "only 2 seats/rooms available" are lies to scare the user into a conversion.

Re: A/B testing gets misused to juice metrics in the short term

#129

Post by Miraste »

jollybean wrote: post #56
grapeskin wrote: post #43

Earlier quoted context omitted.

MBAs aren’t something people are born into or end up through no choice of their own. Bigotry against MBAs is impossible. It’s like being a loan shark or pimp. People actively choose that life and society talks poorly about them because of the social harm they bring. There’s nothing stopping someone from washing their hands of any of those things and being productive.

Quoted post unavailable.

> MBA's are generally better communicators and much easier to get along with.

This is exactly the problem. The MBA skill set is climbing corporate hierarchies. Not product design, customer service, empathy, or long-term thinking; only promotions. That's the whole reason anyone gets an MBA.

MBAs often lack domain knowledge, but even when they have it, the MBA problem remains because their priorities are unrelated to product improvement. They spend their time ingratiating themselves and taking over the top layers of management, hiring more MBAs all the while, until a formerly functional company ends up like Boeing or Sculley's Apple.

Too many MBAs will inevitably degrade product quality.

Re: A/B testing gets misused to juice metrics in the short term

#130

Post by throwaway98797 »

ThalesX wrote: post #116
koheripbal wrote: post #106

Earlier quoted context omitted.

This comment is way too broad and cynical. It reminds me of the sentiment I sometimes hear from my teenage children before I explain to them that the world isn't a Bernie Sander/Reddit gotcha soundbite and that reality is complicated.

What would the other advantages of A|B testing be for a corporation instead of widening the funnel, increasting lifetime revenue or other such bottom-line focused goals?

you can genuinely find out what message makes more sense.

explaining what your product does is hard when you’re the product expert. A/b testing ad copy helps folks parse what your product does.

If your product is actually good then having more people understand what it is, is also good.

Post reply on HN