131–140 of 346 posts
Earlier quoted context omitted.
Do people honestly think that Apple doesn't do A/B testing?
Apple unequivocally does not do active A/B testing on their users by changing applications or the operating system out from under them.
Earlier quoted context omitted.
Sure, but in many cases, such as the example given by GP, long-term AB testing is hard or almost impossible. For the testing to have validity, you need the A and B cohorts to be stable, and have little or no overlap, and that is hard for long time spans for anything that is not account based (and somewhat dangerous even for account-based things, as people will almost certainly start to notice that they are getting a…
In online dating, at least, this is a non-issue. Using an online dating app is, ironically, a solitary enough activity that people don’t go around comparing whether their UI is different from their friends’ UI. You of course can’t let the same user see two versions, but that just means doing permanent group assignment on signup. We used to A/B test subscription prices over enormous ranges (e.g., randomly giving some…
I had a PM who pushed us to A/B test _everything_. We hired a new graphic designer who suggested that we change our product links from ALL CAPS to Title Case (a very popular idea on the team, and his first real suggestion after a few weeks with us), and she insisted that we A/B test it first. It felt like an insult to him, and a dumb test since title case looked way better. The three key outcomes I observed from the…
I say this a lot and I will keep saying it. Conversion != customer obsession. There is a place for A/B testing. It is necessary and can be extremely beneficial in helping your customers enjoy and use your product more successfully. The main issue is that people mix conversion with customer obsession! Whenever you work on a product or feature you should be asking yourself "Is this really good for my customer" - if the…
My 'favorite' silly thing PMs do is UX research studies (typically on 5-10 people) and essentially ask completely untrained people if we should go with X/Y or Z. It's a super-effective way of avoiding responsibility for product decisions ("the data suggest we should go with Y"). If only building good products were as easy as asking what customers think they want.
Earlier quoted context omitted.
Recently I got an account on a developer tool (Checkly) because the company I joined uses it. I then got 5 different emails from them in a 48 hour period. Like I'm sure many users sign up then drop out of your funnel but I'm part of an organization that's a paying customer. I'm already going to use your stuff. What possible business benefit could there be to you spamming me? If anything you're risking the inverse - i…
Hey CSMastermind, I'm founder at Checkly and I got a ping we were mentioned. We do send out some "getting started" messages on autopilot. We also did a product launch Thursday and then our regular changelog on Monday. That probably was overwhelming. If you could email me on tim -at- checklyhq -dot- com I will track down if we hit the the spam cannon too hard.
I've talked about this on HN but I'll say it again. A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a bi…
Why can't we have nice things?
Earlier quoted context omitted.
I never seen an A/B test where the goal was not to maximise profit for the company.
This comment is way too broad and cynical. It reminds me of the sentiment I sometimes hear from my teenage children before I explain to them that the world isn't a Bernie Sander/Reddit gotcha soundbite and that reality is complicated.
Earlier quoted context omitted.
What would the other advantages of A|B testing be for a corporation instead of widening the funnel, increasting lifetime revenue or other such bottom-line focused goals?
you can genuinely find out what message makes more sense. explaining what your product does is hard when you’re the product expert. A/b testing ad copy helps folks parse what your product does. If your product is actually good then having more people understand what it is, is also good.
I recall when Booking.com rolled out the false urgency features. I was amazed at how utterly trashy and desperate they were. The problem is it's not subtle at all; there's a handful of those features that, when combined, end up being overbearing and noisy: "3 people looked at this listing within the past 3 days! 12 rooms left at this rate!" I don't care. I'm looking to book business or vacation travel. If a spot fill…
But unfortunately, it works.
I've seen friends that I consider intelligent panic buy tickets/hotels, "because prices are going up since the last time I checked!"
Next time you want to book anything, browse around, ignore any of the fake urgency notifications, ignore the price (while staying broadly in your price-range, of course). Then when you found a destination you like, open the page in a private browsing window (or clear your cookies), and you'll see that prices and availability are back to normal.
Earlier quoted context omitted.
In this case, what I'm referring to is: 1. Clicked on page. 2. Took maybe 10 seconds to take in what is "above the fold". 3. Scrolled down to see what else there is. 4. BAM! Popup triggered by scrolling down. While I understand what you're getting at, they do not yet have the info to know that I'm browsing or whatever. They were so excited about their stupid popup that they didn't even get that far. I will say, gener…
In fact, you want the dismiss button to be easily discovered and used. Dismissals are an important signal about the quality of the content; just as important as clicks. When you make the button impossible to use you rob yourself of that signal while simultaneously making click data far less reliable.
Then the problem is people, and "they're just clicking close out of reflex!".
Cue hiding this, and results you wanted appear! Success! Raise! Promotion! Or, maybe more funding, due to signs of greater engagement.