A/B testing gets misused to juice metrics in the short term • Hacker News Board

Re: A/B testing gets misused to juice metrics in the short term

#201

Post by tpoacher » Tue, Jul 12, 2022, 5:07 PM UTC

kaoD wrote: post #153

tnolet wrote: post #126

Earlier quoted context omitted.

Hey CSMastermind, I'm founder at Checkly and I got a ping we were mentioned. We do send out some "getting started" messages on autopilot. We also did a product launch Thursday and then our regular changelog on Monday. That probably was overwhelming. If you could email me on tim -at- checklyhq -dot- com I will track down if we hit the the spam cannon too hard.

I know this is well-intentioned and not an automated message, but I find it ironic that you managed to get an additional message to him over here.

one asking them to establish contact for communication to continue, no less. xD

I can't help be reminded of those "if you'd like to unsubscribe call us so we can harass you with offers even more" you see in some company terms and conditions.

Re: A/B testing gets misused to juice metrics in the short term

#202

Post by jl6 » Tue, Jul 12, 2022, 5:08 PM UTC

corrral wrote: post #73

hbn wrote: post #33

I've talked about this on HN but I'll say it again. A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a bi…

> Every app update was a risk of the app rearranging itself, or features appearing/disappearing. This shit drives my parents insane. Me too, when I have to help them. I've had to spend tens of seconds looking at a major screen in the phone app , of all things, to figure out WTF I'm looking at so I could help them figure out what was up. Re-arranged every update (or new phone) for absolutely no reason, terrible afford…

If you put a dollar value on your own time spent providing training and tech support to your parents, the iPhone options start to look much cheaper.

Re: A/B testing gets misused to juice metrics in the short term

#203

Post by dspillett » Tue, Jul 12, 2022, 5:09 PM UTC

bottled_poe wrote: post #171

eloisius wrote: post #103

Earlier quoted context omitted.

Email marketing in general blows my mind. Marketers typically have absolutely no respect for consent, and the costs are completely borne by the recipient. The whole industry depends on dark patterns, shady list sharing, and scraping your email to add it to their lists despite you having no relationship with them. I know it's not simple, and it's just my frustration speaking, but I don't understand how my mail host ca…

Mostly, it doesn’t even seem to matter whether you agree or not. Inevitably you end up receiving affiliate emails regardless.

This is why every company gets a different address for me. If junk starts coming in, that address is blocked, and I stop doing business with that company (if I haven't already).

This sometimes falls foul of spammers adding some random addresses of the form blahblah@mycatchall.domain.tld or @mycatchall.domain.tld into their lists, but that hasn't happened often enough to be a problem. That it isn't much of a problem surprises me a little, given how much @domain.tld (no sub-domain) addresses are used this way. I have considered trying the pattern somename@.domain.tld as an alternative if that becomes a problem, but before implementing that I need to change my email setup (doing that anyway soon as running Zimbra's OSS version is going to get more difficult next year) and maybe my DNS server of choice (if wildcard MX records are an issue, I've not looked into that).

Sometimes I get funny looks for addresses like this, especially as I usually work the company/other name in there somewhere. I had one website refuse to accept an address based on their name, which was a rad flag and I backed away from going any further into dealing with that organisation.

Re: A/B testing gets misused to juice metrics in the short term

#204

Post by dang » Tue, Jul 12, 2022, 5:09 PM UTC

shijie wrote: post #44

jollybean wrote: post #32

Earlier quoted context omitted.

Quoted post unavailable.

Not every critical or mean-spirited comment is bigotry. You got your fee-fees hurt because there’s probably some truth to the comment, even if it’s mean. Buck up, quit complaining, prove the parent comment wrong in your day-to-day work. Right now you’re just deepening a few MBA stereotypes at the moment.

Flamewar comments, name-calling, and personal attacks will get you banned here, regardless of how wrong someone else is or you feel they are. Please don't post like this to HN.

If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

Re: A/B testing gets misused to juice metrics in the short term

#205

Post by dang » Tue, Jul 12, 2022, 5:09 PM UTC

Jasper_ wrote: post #37

jollybean wrote: post #32

Earlier quoted context omitted.

Quoted post unavailable.

Before you commented, I thought "there is nothing in this world an MBA can't make worse" was a bit too harsh. Now, I see it's actually accurate.

Flamewar comments will get you banned here, regardless of how wrong someone else is or you feel they are. If you'd please review https://news.ycombinator.com/newsguidelines.html and stick to the rules when posting here, we'd appreciate it.

Re: A/B testing gets misused to juice metrics in the short term

#206

Post by dang » Tue, Jul 12, 2022, 5:10 PM UTC

aixi wrote: post #53

jollybean wrote: post #32

Earlier quoted context omitted.

Quoted post unavailable.

You're being responded by a bunch of resentful engineers, but you're correct; it's the definition of bigotry.

Flamewar comments will get you banned here, regardless of how wrong someone else is or you feel they are. If you'd please review https://news.ycombinator.com/newsguidelines.html and stick to the rules when posting here, we'd appreciate it.

Re: A/B testing gets misused to juice metrics in the short term

#207

Post by tpoacher » Tue, Jul 12, 2022, 5:10 PM UTC

tnolet wrote: post #166

tnolet wrote: post #141

Earlier quoted context omitted.

You are correct. I will figure out two things. 1. Why our fancy expensive mailing/marketing tool (Intercom) does not spread these messages in a relaxed fashion (it should) 2. If of those five messages, maybe two were the obligatory "confirm your email" and standard "Welcome to Checkly, this is what we do" TL;DR we should not spam.

We checked our Intercom. We spammed. One user got six emails in a 5 day span. There were some separate initiatives going on. We didn't check the settings and current outgoing mail. We will change it. BTW: all of this was done without any bad intent. It's 100% us being stupid and not coordinating and being diligent enough.

Send a seventh to apologize! /s

(on a serious note, good on you for taking action though. +1)

Re: A/B testing gets misused to juice metrics in the short term

#208

Post by oxfordmale » Tue, Jul 12, 2022, 5:17 PM UTC

jerf wrote: post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

I love to sign-up to news letters and get a discount. Of course I am giving you my spam account I set up for this exact purpose.

Re: A/B testing gets misused to juice metrics in the short term

#209

Post by jollybean » Tue, Jul 12, 2022, 5:19 PM UTC

Miraste wrote: post #129

jollybean wrote: post #56

Earlier quoted context omitted.

Quoted post unavailable.

> MBA's are generally better communicators and much easier to get along with. This is exactly the problem. The MBA skill set is climbing corporate hierarchies. Not product design, customer service, empathy, or long-term thinking; only promotions. That's the whole reason anyone gets an MBA. MBAs often lack domain knowledge, but even when they have it, the MBA problem remains because their priorities are unrelated to p…

"You can tell the witches by their propensity to gather with one another - which is proof indeed they are witches, and must be burned, lest they infect the rest of us good people!"

My. God. Man.

The amount of misunderstanding, arbitrarily 'made up stuff' and naive speculation on this thread ... it's just ridiculous.

It's kind of general professional designation, and valuable education for those with no business background (though likely not useful unless from one of the better schools). I'll gather that more than 1/2 of them come from hard professional backgrounds (i.e. CFA, Finance, Eng, Science, Econ) and that they may not be directly oriented towards a specific 'role or trade' frankly is not that important.

Almost everything that has been specifically said in this thread is a bit ridiculous (almost comical), and a lot of it kind of petty as well.

"Those sneaky manipulators!"

"Their priorities are this or that!"

And really, really weird the amount of hyperbole here.

I'm going to guess that maybe you don't really know MBA is (that's ok, it's not an insult, most people don't), and because you don't speak the language of finance, macro/micro economics, marketing, you may be inclined to assume that those things 'are not irrelevant', which is and oddly peculiar form of ironic glibness.

It's a generalist designation.

Some of them are good, some are not.

It's not that big of a deal, and it's not at all like what some people are implying here.

Re: A/B testing gets misused to juice metrics in the short term

#210

Post by corrral » Tue, Jul 12, 2022, 5:20 PM UTC

selfhoster11 wrote: post #187

corrral wrote: post #73

Earlier quoted context omitted.

> Every app update was a risk of the app rearranging itself, or features appearing/disappearing. This shit drives my parents insane. Me too, when I have to help them. I've had to spend tens of seconds looking at a major screen in the phone app , of all things, to figure out WTF I'm looking at so I could help them figure out what was up. Re-arranged every update (or new phone) for absolutely no reason, terrible afford…

At one point I was trying to set up my grandma with a popular video calling app on a dedicated device so we could stay in touch. Before I could give her the freshly grandparent-proofed device, said video calling app upgraded on my parents' PC first and changed literally every single element of the UI beyond recognition. To someone the age of my grandma, that would be literally like bricking the device remotely, becau…

> STOP CHANGING THINGS! Even if the new UI is better (debatable), some people just like or rely on a particular layout to operate the device or app. Don't rearrange without giving a ~permanent setting to use the old layout.

The fact that we're 15 years into smartphones being popular and that phones & computerized address books were basically fully-solved interfaces long before those took off, and they're still fucking around with phone app UI in big ways, is a sign of some kind of institutional or industry-market failure to me. Or both. Allowed, I suppose, because Google's market position (i.e. monopoly across several markets) both gives them the surplus profit (i.e. rents) to fuck around and waste money on harmful to users crap like phone app redesigns, and insulates them from any actual threat to their profit due to those bad choices.