A/B testing gets misused to juice metrics in the short term • Hacker News Board

Re: A/B testing gets misused to juice metrics in the short term

#141

Post by tnolet » Tue, Jul 12, 2022, 3:18 PM UTC

wccrawford wrote: post #135

tnolet wrote: post #126

Earlier quoted context omitted.

Hey CSMastermind, I'm founder at Checkly and I got a ping we were mentioned. We do send out some "getting started" messages on autopilot. We also did a product launch Thursday and then our regular changelog on Monday. That probably was overwhelming. If you could email me on tim -at- checklyhq -dot- com I will track down if we hit the the spam cannon too hard.

I don't use your tool and I'm unrelated to CSMastermind, but I can tell you that you absolutely hit the "spam cannon" too hard. 5 messages in 48 hours? That's absolutely ridiculous. They could all have been 1 message if it was important that they get that info, and it could have been 0 messages if it wasn't.

You are correct. I will figure out two things.

1. Why our fancy expensive mailing/marketing tool (Intercom) does not spread these messages in a relaxed fashion (it should)

2. If of those five messages, maybe two were the obligatory "confirm your email" and standard "Welcome to Checkly, this is what we do"

TL;DR we should not spam.

Re: A/B testing gets misused to juice metrics in the short term

#142

Post by skupig » Tue, Jul 12, 2022, 3:18 PM UTC

koheripbal wrote: post #106

loriverkutya wrote: post #67

Earlier quoted context omitted.

I never seen an A/B test where the goal was not to maximise profit for the company.

This comment is way too broad and cynical. It reminds me of the sentiment I sometimes hear from my teenage children before I explain to them that the world isn't a Bernie Sander/Reddit gotcha soundbite and that reality is complicated.

Let me be even broader and more cynical: everything a company does is ultimately to maximize profit.

Re: A/B testing gets misused to juice metrics in the short term

#143

Post by amluto » Tue, Jul 12, 2022, 3:22 PM UTC

Reading this makes me think of the handful of sites, often targeted at professionals, that highly optimize for the experience of actually buying things. McMaster-Carr comes to mind. Their users shop there over and over, and McMaster wants to keep them. So you can find things for $2 or $2000, shipping prices are inoffensive, customer service is friendly but rarely needed, and there are minimal distractions on the way to checking out or even after checking out. The only real issues are mostly related to the fact that they sell so many products that one can get lost in the 4000+ items that all match the search. Well done.

This is an interesting contrast to Amazon that also makes checkout easy but bombasts the user with thousands of listings, mostly mildly fraudulent and consisting of absolute crap, and still somehow gets repeat business.

Re: A/B testing gets misused to juice metrics in the short term

#144

Post by 411111111111111 » Tue, Jul 12, 2022, 3:25 PM UTC

ravivyas wrote: post #81

axg11 wrote: post #47

A/B testing is local optimization. It should only be done on a mature(-ish) product when you have given up on finding a global minimum. Running experiments and A/B tests are popular because it is _guaranteed_ to give you signal. If you have a large engineering team and you're not sure how to filter the quality of results, gating everything through A/B tests is a well understood methodical way to ensure only positive…

"Running experiments and A/B tests are popular" ... because you can give up on your own judgment and opinions and say "the data says"

> give up on your own judgment and opinions and say "the data says"

The beauty of AB testing is that you don't have to give up your opinion. You can just change irrelevant things until the result you desire gets proven by chance and now you've got data to base your opinion on!

Re: A/B testing gets misused to juice metrics in the short term

#145

Post by bombcar » Tue, Jul 12, 2022, 3:25 PM UTC

corrral wrote: post #73

hbn wrote: post #33

I've talked about this on HN but I'll say it again. A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a bi…

> Every app update was a risk of the app rearranging itself, or features appearing/disappearing. This shit drives my parents insane. Me too, when I have to help them. I've had to spend tens of seconds looking at a major screen in the phone app , of all things, to figure out WTF I'm looking at so I could help them figure out what was up. Re-arranged every update (or new phone) for absolutely no reason, terrible afford…

It's doubly sad because the larger phone or iPad would be perfect, but every year it's another $200 to tell Google they're doing the right thing.

At least on iPad/iPhone you can set the apps to access Google mail, etc, which doesn't change as often, but still too often.

Re: A/B testing gets misused to juice metrics in the short term

#146

Post by jdlshore » Tue, Jul 12, 2022, 3:25 PM UTC

naravara wrote: post #88

jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

Honestly instead of a cookie law I wish GDPR has imposed a rule that required all those stupid interstitial pop-ups to conform to a standard that could be easily blocked by the browser. I mean they are asking for emails, which is a massive and totally unnecessary proliferation of personally identifiable information. I hate them so much. It makes it feel like so much more of a chore to try to do research or look for t…

Pet peeve: GDPR is not the cookie law, and is fact a very sensible collection of restrictions on how companies can collect and process personal data. The annoying banners you see are against the spirit of the GDPR, and quite possibly against the letter of it, too.

Re: A/B testing gets misused to juice metrics in the short term

#147

Post by ryanmarsh » Tue, Jul 12, 2022, 3:26 PM UTC

For many businesses revenue is a function of aggressive deal making. Full stop. In an undifferentiated market of discretionary (impulse) purchases if you don't hustle the customer you make less. The author of this article is confusing companies that are bad at hustling with hustling being bad.

One time offers, limited time offers, mailing list signups, up-sells, and cross sells are time tested ways to increase sales dating as far back as radio era telephone and catalog sales.

Steve Madden is a perfect example of this. They sell undifferentiated popular shoe styles less expensive than high fashion but more expensive than knockoffs. They have to hustle you to get you on their mailing list (for 10% off your order) in the hopes that you'll make another impulse purchase later when you get a text or email from them. If they weren't as aggressive you might never make another impulse purchase with them again as there are tons of brands selling nearly identical products.

Some companies are just horrible at hustling so they actually get in the way of you completing your purchase. In a competitive market this is a self correcting problem.

Re: A/B testing gets misused to juice metrics in the short term

#148

Post by fairity » Tue, Jul 12, 2022, 3:31 PM UTC

It doesn’t seem like the author has any hard data that supports his claim that long term LTV and K-factor losses outweigh short term conversion rate wins. Maybe I missed it? Without said evidence, it’s probably safe to assume his generalized claim is wrong in most cases.

Re: A/B testing gets misused to juice metrics in the short term

#149

Post by time_to_smile » Tue, Jul 12, 2022, 3:32 PM UTC

benja123 wrote: post #27

I say this a lot and I will keep saying it. Conversion != customer obsession. There is a place for A/B testing. It is necessary and can be extremely beneficial in helping your customers enjoy and use your product more successfully. The main issue is that people mix conversion with customer obsession! Whenever you work on a product or feature you should be asking yourself "Is this really good for my customer" - if the…

The term "customer obsession" has become a red flag for me when interviewing because I've never worked at or chatted with a company that had "customer obsession" as value that wasn't aggressively working to squeeze every dime from their users with zero interest in whether or not this squeezing was harmful to the customer.

An actual, sincere customer obsession (and btw I think we both completely agree here) means that you are willing to lose out on some conversion and revenue in order to make sure your customers are top priority.

Real customer obsession isn't just an ethical principle either, it makes business sense. The problem is that the value of customer obsession is realized over the span of years or decades. Companies that have a sincere customer obsession are the kinds of places that survive economic ups and downs, where people's children grow up and are loyal to the product because they remember the time their parents were treated well by the company.

If your only company focus is Q4 KPIs then you really can't have "customer obsession".

Re: A/B testing gets misused to juice metrics in the short term

#150

Post by nemo44x » Tue, Jul 12, 2022, 3:32 PM UTC

d23 wrote: post #123

nemo44x wrote: post #118

Earlier quoted context omitted.

A lot of this happens because different managers have different metrics/KPIs they are optimizing around and they all find "good places" to do things to help meet their goals. The secondary effects aren't considered. One managers quest for outperforming their goals comes at the expense of another managers goals.

There was a point a few years ago where you could not see a single piece of user-generated content above the fold on the reddit home page. A bunch of teams had jockeyed for having their little carousels and banners put on top, and of course, metrics were always cited. I left a screenshot in slack and it ended up causing a couple of teams to have to roll back their widgets, but it always baffled me that we were able t…

> but it always baffled me that we were able to focus so much on the individual trees of metric optimization that we would miss the forest to that extent

Always look to the decision maker's incentives and you'll almost always discover why things are the way they are. And often, to your point, there's an aspect of tunnel vision associated with it because considering the bigger picture is difficult as a company grows and becomes more complex and creates friction in achieving goals.

Ultimately, this is the purpose of senior leadership. But the Peter Principal really begins to kick in at that level and the truth is, many senior leaders are in over their head and are unable to materialize the broader strategy and understand how their various units are affecting it. So we end up with crappy products.