Ask HN: What are the cons of experimentation driven products?

If you do A/B testing wrong it can be bad (search around that and you’ll find stuff)

If you maximize signups you may be fishing with too large a net and catch customers who aren’t appropriate. A/B won’t tell you that.

You may end up at a local maxima optimizing things when you should be fixing major real issues.

Experimentation should have a hypothesis and a test, not throwing stuff at the wall to see what sticks.

Experimentation has become a cop out in many companies I've seen. I've seen it become an easy way for product managers to show 'they're doing something' and no one would ever blame them for it. I've seen it used as an excuse for not being creative, hard working, knowing the market and competitors and making big bold bets, but rather to just throw random stuff on a jira board. Later in meetings the experiment 'working or not working' is being discussed without any repercussions for having wasted a whole engineering team's time for weeks.

I'm not against experimenting in general, but it has a place. Experiment what you truly don't know. Don't experiment what you know. If you're good at your job you know things that need to happen. Get those done.

Many a/b tests are false positives and fail A/A/B test

A/b test completely miss long term effects i.e. more ads= more money and no losses in engagement, but user is getting fed up and in 6 months resentment is going to boil over then all new tests will start showing negative outcomes but cause that triggered it occurred half a year ago.. only way to counter is have users on all possible variations all the time and have enough of them. Not feasible.

Only evolutionary ideas and efforts are optimized for not revolutionary. You will just get a faster and faster horse. Low hanging fruit type of mentality. Get caught in vot of mediocrity

Say you have N + M users and you offer feature A to N users and B to M users and collect data like churn rate on them. If N or M is too small to be statistically significant (meaning either the average is too small in either segment, or the t-test is too close to either average) then the benefit of A/B testing N vs M users was a waste of resources.

So when user count is small you need to be story driven. That is to say, you have to actually speak to users (current, future, potential) about why they are using the product, why they walked away, and why they stayed.

Basically A/B testing is only useful if you have a large enough user base to be confident in its success. That can be hundreds or thousands of users. A lot of products don't have that critical mass, so you can't effectively be data driven.