Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing gets misused to juice metrics in the short term

zumsteg.net

Post reply on HN
Print view View original

91–100 of 346 posts

Re: A/B testing gets misused to juice metrics in the short term

#91

Post by bombcar »

enlyth wrote: post #49
jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

Speaking of 'sign up to our newsletter', one of the latest dark patterns I've found that astounded me was adding a checkbox to the login form [0], where you'd normally expect the 'remember me' checkbox to be. You almost click it out of muscle memory if you don't read what it says. [0] https://i.postimg.cc/HW89hs7r/Screenshot-2022-07-12-145957.p...

"Remember me in your newsletter list" is the next one. Send me money!

Re: A/B testing gets misused to juice metrics in the short term

#92

Post by EricMausler »

/rant

AB testing is and always has been fish oil for management. The only things it can actually prove, are more easily identifiable by common sense. So wherever it actually works, it was probably a waste of time / overkill for evidence.

- sincerely, a business analyst

Re: A/B testing gets misused to juice metrics in the short term

#93

Post by iamben »

mattgreenrocks wrote: post #82

I recall when Booking.com rolled out the false urgency features. I was amazed at how utterly trashy and desperate they were. The problem is it's not subtle at all; there's a handful of those features that, when combined, end up being overbearing and noisy: "3 people looked at this listing within the past 3 days! 12 rooms left at this rate!" I don't care. I'm looking to book business or vacation travel. If a spot fill…

It's amazing how much this works though. I remember getting a call from my mum saying "the website is telling me London is already 78% booked for these dates!" It felt ridiculous having to say "Mum, it's March. You're staying there in November. I promise you it'll be fine..."

Re: A/B testing gets misused to juice metrics in the short term

#94

Post by verve_rat »

jollybean wrote: post #56
grapeskin wrote: post #43

Earlier quoted context omitted.

MBAs aren’t something people are born into or end up through no choice of their own. Bigotry against MBAs is impossible. It’s like being a loan shark or pimp. People actively choose that life and society talks poorly about them because of the social harm they bring. There’s nothing stopping someone from washing their hands of any of those things and being productive.

Quoted post unavailable.

Your behaviour here isn't a good argument for this thread's feelings about MBAs being "obstinate or unreasonable".

Re: A/B testing gets misused to juice metrics in the short term

#95

Post by at_a_remove »

jollybean wrote: post #56
grapeskin wrote: post #43

Earlier quoted context omitted.

MBAs aren’t something people are born into or end up through no choice of their own. Bigotry against MBAs is impossible. It’s like being a loan shark or pimp. People actively choose that life and society talks poorly about them because of the social harm they bring. There’s nothing stopping someone from washing their hands of any of those things and being productive.

Quoted post unavailable.

Most shysters are great communicators and quite a lot of fun to be around. It's how they operate.

Re: A/B testing gets misused to juice metrics in the short term

#96

Post by bombcar »

chairmanwow1 wrote: post #87
CSMastermind wrote: post #72

Earlier quoted context omitted.

Recently I got an account on a developer tool (Checkly) because the company I joined uses it. I then got 5 different emails from them in a 48 hour period. Like I'm sure many users sign up then drop out of your funnel but I'm part of an organization that's a paying customer. I'm already going to use your stuff. What possible business benefit could there be to you spamming me? If anything you're risking the inverse - i…

Sounds like they are optimizing for a KPI on time to full integration. Someone else paid for it, now they want to make sure that you are actually using it. Still absurd, but I know this is a problem friends of mine have had.

The "here's how to use the product you paid for" emails that trickle out over a few weeks or a month is vastly different than "throw you into every single marketing email bucket we have" but companies seem to lean toward the latter.

Re: A/B testing gets misused to juice metrics in the short term

#98

Post by naravara »

causi wrote: post #2

The problem with AB testing is that it's a short-term strategy. For example, if a news site runs AB testing with headlines, they'll find that bullshit clickbait headlines get more pageviews than concise, accurate headlines, but the constant use of clickbait headlines will over time destroy overall traffic to your site. More frustratingly, sites run by smart people tend to fall into a balance where the worst articles…

> but the constant use of clickbait headlines will over time destroy overall traffic to your site

I'd add a bit of nuance here. They are very good at driving traffic, but very bad at building an audience. You do this long enough and your news site is now optimized for attracting hot-take appreciators who engage with the news like a tabloid. This drives away everyone who doesn't want to be reading a tabloid and makes you more dependent on keeping up with traffic-gaming strategies to continuously drive traffic. You've basically shifted your business from being a place that produces journalism to being a place that figures out ways to game social media trends and SEO.

Re: A/B testing gets misused to juice metrics in the short term

#99

Post by Bancakes »

jollybean wrote: post #78
Bancakes wrote: post #45

Earlier quoted context omitted.

Quoted post unavailable.

I'm both a Software Engineer and an MBA, and I can assure you that you're wrong, and with a comedic, glib kind of 'Musk-ian' lack of self awareness that's weirdly common among technical types. I mean it's getting funny now, I was hoping that your comment was satire (!), alas, no.

>You're wrong because..... ugh! you just are, okay?!

Re: A/B testing gets misused to juice metrics in the short term

#100

Post by josefx »

jollybean wrote: post #78
Bancakes wrote: post #45

Earlier quoted context omitted.

Quoted post unavailable.

I'm both a Software Engineer and an MBA, and I can assure you that you're wrong, and with a comedic, glib kind of 'Musk-ian' lack of self awareness that's weirdly common among technical types. I mean it's getting funny now, I was hoping that your comment was satire (!), alas, no.

[flagged]
Post reply on HN