Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing gets misused to juice metrics in the short term

zumsteg.net

Post reply on HN
Print view View original

261–270 of 346 posts

Re: A/B testing gets misused to juice metrics in the short term

#261

Post by daniel_reetz »

teakettle42 wrote: post #120
koheripbal wrote: post #107

Earlier quoted context omitted.

Do people honestly think that Apple doesn't do A/B testing?

Apple unequivocally does not do active A/B testing on their users by changing applications or the operating system out from under them.

I can't think of an Apple engineer who would make such a definitive statement on behalf of the company, in public. I also can't think of anyone working there who would have definitive knowledge across all apps and software, because disclosure would prevent such knowledge.

Apple introduced A/B testing to Testflight in 2017 and more A/B testing to the App store this year.

Source: I worked at Apple, but not in software.

Re: A/B testing gets misused to juice metrics in the short term

#263

Post by donmcronald »

forgotmypw17 wrote: post #235

I have developed a personal strategy of ridding the Web of these things. Anytime it happens, I close the tab and move along. Very little of value is lost.

This is basically what I do. Anything that pops up or tries to grab my attention gets instant closed before I look at it and if I can't find the control to close it in 1s I just close the whole tab.

Re: A/B testing gets misused to juice metrics in the short term

#264

Post by gumby »

Such perfect timing: I just tried to place a take out lunch order with a restaurant. Opening the page popped up a modal box that said "Join Our List Subscribe to find out about new specials, community events, store openings and more." There were no buttons to click, no place to enter my email address (had I wanted to) and clicking did not dismiss it. The modal had a background that obscured the actual page.

I finally opened the inspector and deleted it, so that I could use the menu to select "order online", which took me to a page ... with the same modal.

Re: A/B testing gets misused to juice metrics in the short term

#265

Post by donmcronald »

mschuster91 wrote: post #7

My biggest issue with A/B testing isn't even mentioned here... gaslighting your customers is absolutely not OK . Particularly with older people, the constant "where the f..k did Outlook now put feature XYZ" (in the case that comes to my mind, the CC bar which used to be tab-reachable, now you have to tab+space or manually click on the tiny gray "cc" letters) onslaught is just absurd. When you change how applications…

> Outlook now put feature XYZ" (in the case that comes to my mind, the CC bar which used to be tab-reachable, now you have to tab+space or manually click on the tiny gray "cc" letters)

I can sort of understand wanting to hide stuff on mobile, but the discovery of controls to unhide things should be better. I often help people that are stuck trying to figure out how to do something in an app and not realizing they can click on something that gives no indication it's clickable is a common thing.

Desktop is another world. I often have 20+ inches of horizontal space and a hamburger menu. It's infuriating, especially when the hamburger menu is hiding one button.

Re: A/B testing gets misused to juice metrics in the short term

#266

Post by ryanmcbride »

I give an incredibly similar warning every time a company I'm working for starts trying to dip their toe into A/B testing. I have a lot of experience with it at scale (one at a fortune 100 company) and I've even built an a la carte testing framework in aws for a company that didn't like Target or Optimizely.

Every single time I warn them about how the bill of goods they've been sold with A/B testing is almost completely unattainable, especially in the way that they want to go about it. They won't magically start getting more conversions by changing a button color. Even if they start getting more clicks, they rarely start getting more complete conversions, because the increased numbers is usually from people who weren't good leads in the first place.

On top of that every company I've worked with has no idea what the real methodology for good tests is, no matter how many times I explain it or put it in a slide deck. I would constantly get requests to use A/B testing for feature rollout.

Them: "Hey, could you do an A/B test of our existing site design and our upcoming redesign?"

Me: "if the old design performs better are you going to toss out the redesign?"

Them: "No we're going with the redesign but we want metrics on how it'll affect traffic"

Me: "Those metrics are useless if you aren't going to listen to them, and if the results come back and the old design performs better, you're not even going to put it in a presentation because it's counter to your planned actions. There's literally no point in running this test"

Them: "Run it anyway"

Re: A/B testing gets misused to juice metrics in the short term

#267

Post by jamiek88 »

quickthrower2 wrote: post #161
b112 wrote: post #125

Earlier quoted context omitted.

I'm not a big fan of ipv6, but fan or not, I bet if all spammy mailchimp type provider IP ranges were confiscated and freed, we'd be in ipv4 land for another 20 years. And as a second thought, the way China amd Russia are going, maybe we should just reclaim all their ipv4 addresses, and just give each country 1 IP, they can proxy through it on their end.

Then say goodbye to the internet and hello to a mesh of country-specific networks. Finally, true decentralisation!

We already have that. It’s just we are in denial as a society about it.

Globalization is over. The post Berlin wall fall consensus undermined and ruined.

The last to realize, loses.

Re: A/B testing gets misused to juice metrics in the short term

#268

Post by ryanmcbride »

eloisius wrote: post #103
enlyth wrote: post #49

Earlier quoted context omitted.

Speaking of 'sign up to our newsletter', one of the latest dark patterns I've found that astounded me was adding a checkbox to the login form [0], where you'd normally expect the 'remember me' checkbox to be. You almost click it out of muscle memory if you don't read what it says. [0] https://i.postimg.cc/HW89hs7r/Screenshot-2022-07-12-145957.p...

Email marketing in general blows my mind. Marketers typically have absolutely no respect for consent, and the costs are completely borne by the recipient. The whole industry depends on dark patterns, shady list sharing, and scraping your email to add it to their lists despite you having no relationship with them. I know it's not simple, and it's just my frustration speaking, but I don't understand how my mail host ca…

That's why the only email subscription service on my site is completely transparent and details exactly what we're storing, and it's impossible to click by accident. Everything I do for myself I try to build like a service I'd like to use myself. But the second managers or marketers are involved it's all out the window. I remember early in my career my boss had me add every email included in a TED booklet to their marketing email list. I told him that morality issues aside he could likely catch a fine for that, especially since the type of people listed in a TED booklet are likely more litigious than the average bear. Didn't care, wanted more eyes on the marketing.

Re: A/B testing gets misused to juice metrics in the short term

#269

Post by zanecodes »

selfhoster11 wrote: post #187
corrral wrote: post #73

Earlier quoted context omitted.

> Every app update was a risk of the app rearranging itself, or features appearing/disappearing. This shit drives my parents insane. Me too, when I have to help them. I've had to spend tens of seconds looking at a major screen in the phone app , of all things, to figure out WTF I'm looking at so I could help them figure out what was up. Re-arranged every update (or new phone) for absolutely no reason, terrible afford…

At one point I was trying to set up my grandma with a popular video calling app on a dedicated device so we could stay in touch. Before I could give her the freshly grandparent-proofed device, said video calling app upgraded on my parents' PC first and changed literally every single element of the UI beyond recognition. To someone the age of my grandma, that would be literally like bricking the device remotely, becau…

In the same way that we try to use semantic versioning to prevent unintended breakages in dependents of our public-facing APIs, I think we developers should start considering the UI/UX of our apps to be their public-facing API, and use the philosophy of semantic versioning accordingly.

When we rename or remove a function from an API, that's a breaking change, any dependent software will no longer work unless it's modified to take that change into account.

Similarly, when we move, rename, or remove a UI element, keyboard shortcut, or visual affordance, that should be considered a "breaking change" for our dependents, the humans on the other side of the screen. And in the same way, we should avoid making such changes unless the long-term benefit of doing so outweighs the short-term cost. Plus, users will know that moving from 7.x.x of your software to 8.x.x will require them to relearn some aspect of it.

Re: A/B testing gets misused to juice metrics in the short term

#270

Post by londons_explore »

You can get long term results from AB tests long after the test has ended...

For example, you can see if Group A or Group B from a test are more likely to still use the site 1 year later.

You hypothesize that those ways to 'juice the metrics in the short term' hurt the user experience in the long term... Well if your hypothesis is right, these long term AB results should show it.

Post reply on HN