A/B testing gets misused to juice metrics in the short term • Hacker News Board

Re: A/B testing gets misused to juice metrics in the short term

#161

Post by quickthrower2 » Tue, Jul 12, 2022, 3:45 PM UTC

b112 wrote: post #125

eloisius wrote: post #103

Earlier quoted context omitted.

Email marketing in general blows my mind. Marketers typically have absolutely no respect for consent, and the costs are completely borne by the recipient. The whole industry depends on dark patterns, shady list sharing, and scraping your email to add it to their lists despite you having no relationship with them. I know it's not simple, and it's just my frustration speaking, but I don't understand how my mail host ca…

I'm not a big fan of ipv6, but fan or not, I bet if all spammy mailchimp type provider IP ranges were confiscated and freed, we'd be in ipv4 land for another 20 years. And as a second thought, the way China amd Russia are going, maybe we should just reclaim all their ipv4 addresses, and just give each country 1 IP, they can proxy through it on their end.

Then say goodbye to the internet and hello to a mesh of country-specific networks.

Finally, true decentralisation!

Re: A/B testing gets misused to juice metrics in the short term

#162

Post by fatnoah » Tue, Jul 12, 2022, 3:48 PM UTC

eloisius wrote: post #103

enlyth wrote: post #49

Earlier quoted context omitted.

Speaking of 'sign up to our newsletter', one of the latest dark patterns I've found that astounded me was adding a checkbox to the login form [0], where you'd normally expect the 'remember me' checkbox to be. You almost click it out of muscle memory if you don't read what it says. [0] https://i.postimg.cc/HW89hs7r/Screenshot-2022-07-12-145957.p...

Email marketing in general blows my mind. Marketers typically have absolutely no respect for consent, and the costs are completely borne by the recipient. The whole industry depends on dark patterns, shady list sharing, and scraping your email to add it to their lists despite you having no relationship with them. I know it's not simple, and it's just my frustration speaking, but I don't understand how my mail host ca…

> Email marketing in general blows my mind

My favorite thing is when the companies outsource the email marketing, so that it has absolutely zero relevance. I've been using the same online tax preparer for 10 years, and I've had exactly zero refunds, yet their emails during tax season always let me know that "my refund is waiting".

Re: A/B testing gets misused to juice metrics in the short term

#163

Post by _puk » Tue, Jul 12, 2022, 3:49 PM UTC

aroccoli wrote: post #79

enlyth wrote: post #49

Earlier quoted context omitted.

Speaking of 'sign up to our newsletter', one of the latest dark patterns I've found that astounded me was adding a checkbox to the login form [0], where you'd normally expect the 'remember me' checkbox to be. You almost click it out of muscle memory if you don't read what it says. [0] https://i.postimg.cc/HW89hs7r/Screenshot-2022-07-12-145957.p...

Yeah, I booked a flight on WizzAir two days ago, and this felt like a low blow even from WizzAir.

Predicting the flight will be cancelled in 3.. 2.. 1.. But then the newsletter will haunt you for far far longer..

Not a burnt WizzAir customer at all! /s

Re: A/B testing gets misused to juice metrics in the short term

#164

Post by quickthrower2 » Tue, Jul 12, 2022, 3:49 PM UTC

hbn wrote: post #33

I've talked about this on HN but I'll say it again. A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a bi…

I moved back to iPhone in the 2015s onwards (I think) because of the crap quality of devices available, and the bloatware atop if android.

In many cases the hardware was so poor it was hard to make a call due to the touchscreen.

Since the primary thing I want the phone to do is make a call I switched to the “it just works” camp and haven’t regretted it.

Except getting photos off the phone. Until I realised the best tool for that is … Ubuntu!

Re: A/B testing gets misused to juice metrics in the short term

#165

Post by legalcorrection » Tue, Jul 12, 2022, 3:49 PM UTC

jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

Indeed, quantitative data without qualitative understanding is useless. You can't understand data without understanding mechanisms, because there's an infinite number of possible confounding factors that you can only dismiss through your qualitative understanding of the dynamic you are measuring.

Re: A/B testing gets misused to juice metrics in the short term

#166

Post by tnolet » Tue, Jul 12, 2022, 3:51 PM UTC

tnolet wrote: post #141

wccrawford wrote: post #135

Earlier quoted context omitted.

I don't use your tool and I'm unrelated to CSMastermind, but I can tell you that you absolutely hit the "spam cannon" too hard. 5 messages in 48 hours? That's absolutely ridiculous. They could all have been 1 message if it was important that they get that info, and it could have been 0 messages if it wasn't.

You are correct. I will figure out two things. 1. Why our fancy expensive mailing/marketing tool (Intercom) does not spread these messages in a relaxed fashion (it should) 2. If of those five messages, maybe two were the obligatory "confirm your email" and standard "Welcome to Checkly, this is what we do" TL;DR we should not spam.

We checked our Intercom. We spammed. One user got six emails in a 5 day span. There were some separate initiatives going on. We didn't check the settings and current outgoing mail. We will change it.

BTW: all of this was done without any bad intent. It's 100% us being stupid and not coordinating and being diligent enough.

Re: A/B testing gets misused to juice metrics in the short term

#167

Post by NyxWulf » Tue, Jul 12, 2022, 3:52 PM UTC

costcofries wrote: post #57

hbn wrote: post #33

I've talked about this on HN but I'll say it again. A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a bi…

"To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. " Your perspective is extremely short-sighted. A/B testing can result in this type of behaviour but that's just poor A/B testing. Good A/B testing focuses on removing distractions from the experience and helping users derive…

A/B testing is a tool, and as Deming said, the aim defines the system. In your definition of Good, you are defining the aim.

I've been in a similar situation, where I created a relatively sophisticated A/* testing and control system. My idea of good use of the system ended up being very different from how the team employing the system thought about it.

I believe that is part of the point of the post, that unintended, and even unimagined side effects plague even the best of ideas.

Re: A/B testing gets misused to juice metrics in the short term

#168

Post by AlexandrB » Tue, Jul 12, 2022, 3:55 PM UTC

jollybean wrote: post #56

grapeskin wrote: post #43

Earlier quoted context omitted.

MBAs aren’t something people are born into or end up through no choice of their own. Bigotry against MBAs is impossible. It’s like being a loan shark or pimp. People actively choose that life and society talks poorly about them because of the social harm they bring. There’s nothing stopping someone from washing their hands of any of those things and being productive.

Quoted post unavailable.

The best communicators I've worked with have generally been scientists, especially physicists. Many MBAs seem to communicate in platitudes and vague business jargon that makes simple concepts sound complicated. This is obvious when you start probing for specifics after terms like "value add" or "synergy" are thrown around.

Re: A/B testing gets misused to juice metrics in the short term

#169

Post by mattnewton » Tue, Jul 12, 2022, 3:56 PM UTC

loriverkutya wrote: post #67

costcofries wrote: post #57

Earlier quoted context omitted.

"To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. " Your perspective is extremely short-sighted. A/B testing can result in this type of behaviour but that's just poor A/B testing. Good A/B testing focuses on removing distractions from the experience and helping users derive…

I never seen an A/B test where the goal was not to maximise profit for the company.

Over what time horizon? You should optimize short term metrics (for feasibility) that are (as best you can approximate) the cause of long term profits. The profits part is what makes it the business, using metrics for decisions are what makes it an A/B test. I think where people run into trouble is when they are optimizing only for this quarter’s revenue directly because long term value is too hard to measure.

Re: A/B testing gets misused to juice metrics in the short term

#170

Post by Lutger » Tue, Jul 12, 2022, 3:57 PM UTC

hbn wrote: post #33

I've talked about this on HN but I'll say it again. A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a bi…

> To me, capitalism (A/B testing) means you don't respect your citizens (users). You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them.

You just described doing business in todays world.

Being a bit more generous towards A/B testing, I would make a counter point: _not_ doing any kind of user testing, of which large scale automated A/B tests are just a subset, means you don't respect your users. Because it means you just assume you know what their experience is like, or worse: you don't even care about it and bother to learn something.

Your complaint seems to be more about the scale and aspect of automation honestly, and continuity of the services, which is a valid complaint against Google but not about A/B testing in general.