To your point, my company is doing some A/B tests and I insisted that we not just measure conversion ("it works") and additionally measure some metrics that would indicate that it works _well_. For example, if you have a carousel of products, and someone buys something from the carousel, then it "works," but it would work _better_ if the item they bought was the first thing on the carousel rather than the last. That indicates that we showed relevant products first, which is better than showing the relevant product last!
Sure, conversion would go up if it was in the first slot versus the last, but it takes effort (however little) to scroll through a carousel, so ensuring that we can measure the quality of the result and not just the quantity is really important.
This is one way I've tried to avoid the problem you describe. It's not enough that people can engage with the feature, but they need to engage with it meaningfully and in such a way that would encourage repeat behavior.
Another example of what you describe is that on our site if you search for "neon blue guitar", don't interact with the search results whatsoever, go back to the home page, click on a product on a carousel, and purchase _that_ product, it counts as a "successful search event," even though the search technically failed because they didn't interact with it in any meaningful way. To your point: PM is happy; user is not.
TL;DR it's really important to think through tests and how you measure success!