Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing gets misused to juice metrics in the short term

zumsteg.net

Post reply on HN
Print view View original

251–260 of 346 posts

Re: A/B testing gets misused to juice metrics in the short term

#251

Post by _tom_ »

chunkyks wrote: post #115

"If a study came out that said deafening high-pitched noises increased conversion rates, we would all be bleeding from our ears by end of business tomorrow, right?" Netflix auto play? Is that you? You were a hateful idea, no one liked you, yet you stubbornly hung on for far too long

I'm convinced that Netflix uses "number of hours watched" as a success metric. Autoplay raises that.

I'd pay twice as much to watch half as much quality programming, but that would tank what they think is a positive metric.

Re: A/B testing gets misused to juice metrics in the short term

#252

Post by brynjolf »

hbn wrote: post #33

I've talked about this on HN but I'll say it again. A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a bi…

Ironically the place I'm getting forced A/B testing now is the Playstore. They moved the update apps section so it is now 2-3 clicks more, making me have to make a seperate shortcut just to reach the app update quickly.

The point is of course to make it annoying to manually update apps and enable auto update. I have been burned too many times with a auto update so I refuse.

This wasn't enough, they really want to force me enable auto updates to the point of the update section of the app having 50% of the visible space on my big screen being covered with a message to enable auto update over WiFi. [0]

Whoever is doing this at Google... Stop. Just stop. It is cringe.

[0] https://i.imgur.com/MbXn9gy.jpg

Re: A/B testing gets misused to juice metrics in the short term

#253

Post by travisjungroth »

spookthesunset wrote: post #160
nemo44x wrote: post #118

Earlier quoted context omitted.

A lot of this happens because different managers have different metrics/KPIs they are optimizing around and they all find "good places" to do things to help meet their goals. The secondary effects aren't considered. One managers quest for outperforming their goals comes at the expense of another managers goals.

This is why any good org will make sure to observe all important KPI’s while doing an A/B test. If your “email signup” KPI went to the moon but tanked your “bought shit” metric… you should probably roll back.

It's really easy for this to be noise from false negatives. On an A/A test with five guardrail metrics and a threshold of p>0.05, you'll get a false negative 22.6% of the time.

Re: A/B testing gets misused to juice metrics in the short term

#254

Post by oxfordmale »

tnolet wrote: post #126
CSMastermind wrote: post #72

Earlier quoted context omitted.

Recently I got an account on a developer tool (Checkly) because the company I joined uses it. I then got 5 different emails from them in a 48 hour period. Like I'm sure many users sign up then drop out of your funnel but I'm part of an organization that's a paying customer. I'm already going to use your stuff. What possible business benefit could there be to you spamming me? If anything you're risking the inverse - i…

Hey CSMastermind, I'm founder at Checkly and I got a ping we were mentioned. We do send out some "getting started" messages on autopilot. We also did a product launch Thursday and then our regular changelog on Monday. That probably was overwhelming. If you could email me on tim -at- checklyhq -dot- com I will track down if we hit the the spam cannon too hard.

tnolet, if I get five emails from a company with 48 hours, I will set up a spam rule for them. If I really need to get that much help to get started, your UX design is likely not very good.

Re: A/B testing gets misused to juice metrics in the short term

#255

Post by enlyth »

nebusoft wrote: post #234
enlyth wrote: post #196

Earlier quoted context omitted.

Happened to my dad on Sunday, and the only replacement flight they would offer is for this Friday, what a complete joke air travel is in 2022.

I mean you're complaining about an ultra-low cost airline. why would you expect it to have a good customer experience?

Yes I am. They also overbooked seven people on that flight, delayed it for hours, and then completely cancelled. There were people in wheelchairs left stranded at the airport after waiting there all day. This is just plain incompetence.

Re: A/B testing gets misused to juice metrics in the short term

#256

Post by 3dGrabber »

dspillett wrote: post #203
bottled_poe wrote: post #171

Earlier quoted context omitted.

Mostly, it doesn’t even seem to matter whether you agree or not. Inevitably you end up receiving affiliate emails regardless.

This is why every company gets a different address for me. If junk starts coming in, that address is blocked, and I stop doing business with that company (if I haven't already). This sometimes falls foul of spammers adding some random addresses of the form blahblah@mycatchall.domain.tld or @mycatchall.domain.tld into their lists, but that hasn't happened often enough to be a problem. That it isn't much of a problem s…

> If junk starts coming in, that address is blocked, and I stop doing business with that company (if I haven't already).

AND CALL them, if possible: “I’ve received marketing emails from your company recently, how is this possible, I’ve never signed up, yaddayadda… “

Generate some cost on their side.

Re: A/B testing gets misused to juice metrics in the short term

#257

Post by fouric »

hbn wrote: post #33

I've talked about this on HN but I'll say it again. A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a bi…

A few months ago, I came to the belief that Google is the ADHD toddler of user-facing software development - absolutely unable to sit still and concentrate on anything, hence the constant UI/UX churn, half-baked products, and graveyard[1] of shiny things that they worked on for a few years before abandoning.

Google seems to be really good at making developer tools like Borg and Blaze - however, I think that as an organization they have some deficit that makes them not responsible enough to develop user-facing software (like, uh, an operating system).

Maybe Google would be better as a B2B company.

[1] https://killedbygoogle.com/

Re: A/B testing gets misused to juice metrics in the short term

#258

Post by tnolet »

oxfordmale wrote: post #254
tnolet wrote: post #126

Earlier quoted context omitted.

Hey CSMastermind, I'm founder at Checkly and I got a ping we were mentioned. We do send out some "getting started" messages on autopilot. We also did a product launch Thursday and then our regular changelog on Monday. That probably was overwhelming. If you could email me on tim -at- checklyhq -dot- com I will track down if we hit the the spam cannon too hard.

tnolet, if I get five emails from a company with 48 hours, I will set up a spam rule for them. If I really need to get that much help to get started, your UX design is likely not very good.

You are correct. I’m the same. Our emails are even pretty good. Our UX is pretty damn good. Somehow we dropped the ball here.

Re: A/B testing gets misused to juice metrics in the short term

#259

Post by boruto »

causi wrote: post #2

The problem with AB testing is that it's a short-term strategy. For example, if a news site runs AB testing with headlines, they'll find that bullshit clickbait headlines get more pageviews than concise, accurate headlines, but the constant use of clickbait headlines will over time destroy overall traffic to your site. More frustratingly, sites run by smart people tend to fall into a balance where the worst articles…

Indeed,

If do more ad placements increase revenue is the test and then there is 20% jump what are you as an engineer going to do? Tell to management that its bad?

Re: A/B testing gets misused to juice metrics in the short term

#260

Post by goodside »

Philadelphia wrote: post #228
goodside wrote: post #5

Earlier quoted context omitted.

There’s nothing about AB testing that requires you to use short-term metrics. I used to manage AB tests for online dating sites (OkCupid, Grindr) where subscription revenue is what matters, and the gains of any strategy will take months to materialize. We were well aware that, say, raising prices would yield more short-term revenue at the expense of long-term revenue. That didn’t stop us from testing, it just made th…

OkCupid has completely destroyed its interface and utility, so whatever they’re doing doesn’t seem to be working anymore.

I left in 2015, as soon as it became apparent the party was over. OkCupid went downhill for a lot of reasons, but overly aggressive A/B testing wasn't one of them.
Post reply on HN