Live data from Hacker News

A/B testing gets misused to juice metrics in the short term

zumsteg.net

341–346 of 346 posts

Re: A/B testing gets misused to juice metrics in the short term

#341

If you're upset about internet retail, I hope you're also upset about milk being in the back of the store to get you to walk through the whole thing, because this has been merchandising's bread and butter for a very long time.

This is a long-standing and oft-repeated myth.

Milk is in the back of the store because that's where it makes sense to have a refrigerated wall.

Milk is increasingly available in smaller quantities in compact refrigeration units at the front of the store.

Re: A/B testing gets misused to juice metrics in the short term

#342
post #21

"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page , and some…

What you describe could actually be an artifact of "flaw of averages" (an article was posted here just few days ago, TLDR: air force discovered that nobody fit the body dimensions of an average pilot and therefore created adjustable seats for jets). If promotion banners are just segmented to general millions of people then that's exactly what will happen. But not sure if better segmenting is even possible with data available.

Re: A/B testing gets misused to juice metrics in the short term

#343
The "Hotel Tonight" example hit home with me, recently.

I used to use that app all the time, then kids happened and spontaneous hotel reservation became rare. Fast forward a few years and a circumstance came up that made me think "Hotel Tonight". I discovered it wasn't installed on my new phone so I grabbed it. It was unrecognizable. Maybe the prices were as good, maybe it could still be used the way I used it previously, but it looked like it turned into a hotel booking app when what I wanted to see was a small selection of good hotels nearby with unusually low prices. One of the features was the lack of choice.

Re: A/B testing gets misused to juice metrics in the short term

#344

If you're upset about internet retail, I hope you're also upset about milk being in the back of the store to get you to walk through the whole thing, because this has been merchandising's bread and butter for a very long time.

This is a long-standing and oft-repeated myth. Milk is in the back of the store because that's where it makes sense to have a refrigerated wall. Milk is increasingly available in smaller quantities in compact refrigeration units at the front of the store.

It's probably both. But where products are placed in a store are uncontroversially designed to sell more not to make your life more convenient. https://www.npr.org/2014/08/01/337034378/everyone-goes-to-th...

Re: A/B testing gets misused to juice metrics in the short term

#345

Earlier quoted context omitted.

This is why every company gets a different address for me. If junk starts coming in, that address is blocked, and I stop doing business with that company (if I haven't already). This sometimes falls foul of spammers adding some random addresses of the form blahblah@mycatchall.domain.tld or @mycatchall.domain.tld into their lists, but that hasn't happened often enough to be a problem. That it isn't much of a problem s…

You can just use [id]-[sha1 hmac]@domain.tld The id could be anything, and the SHA1 HMAC takes 32 characters in base32 (which is an email-address-safe encoding). Then just configure your spamfilter to reject any address where the HMAC doesn’t check out. Of course, the drawback is that you’ll need a computer to generate a new address… At which point you may as well store an explicit whitelist of valid addresses.

You can do that from a phone app though, or even a static webpage that loads on phones. It doesn't have to be 32 characters either, putting ~4 characters is probably secure enough, unless a ton of people start using this exact scheme.

This is a great idea, I might implement it. The difficult part is hooking the validation in the mail server I guess.

Re: A/B testing gets misused to juice metrics in the short term

#346

Earlier quoted context omitted.

Yeah my mortgage company sent me a letter saying to opt out of affiliate marketing Emails or snail mail I have to send them a letter requesting it. This was 4 months after they bought my mortgage so the most the letter woulda done is stop them after selling my info for 4 months.

I had a coworker who sent those letters as a side hustle. He had a few different ones and would send the letters certified mail. Companies are very poor at compliance, and certain violations allow you to sue the company.

Everything in that except the unspoken end point where I end up with money sounds awful
Post reply on HN