Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

If your customers don't talk, NPS is a vanity metric

elliotcsmith.com

Post reply on HN
Print view View original

51–57 of 57 posts

Re: If your customers don't talk, NPS is a vanity metric

#51

Post by nulbyte »

pflenker wrote: post #17
nulbyte wrote: post #3

I would go a step further: NPS is a garbage metric. First, you start by assuming your customers can even reasonably ascertain their likelihood to recommend. They can't; there are people who answer 10 but will never recommend, and there are people who answer 0 but already have and will again. Next, you assume your customers are idiots and don't know how an 11-point scale works by adjusting the midpoint: Instead of 5,…

Not defending NPS here - I don't use it - but some of your assumptions are wrong. > They can't; there are people who answer 10 but will never recommend, and there are people who answer 0 but already have and will again. What matters with NPS is trend over time, and getting the numbers at a scale. Yes, there are people who randomly click on one end of the scale or the other, but the assumption is that on average the p…

> Yes, there are people who randomly click on one end of the scale or the other, but the assumption is that on average the portion of these people is stable.

That assumption is predicated first on the idea that people can and will tell you their likelihood to recommend within a reasonable degree of accuracy. I don't think they do.

1: https://www.xminstitute.com/data-snippets/gap-consumer-recom... 2: https://hbr.org/2019/10/where-net-promoter-score-goes-wrong

> If you believe in that core assumption that NPS measures word of mouth, then this interpretation of "likely to snag some other customers on my way out the door" is a sensible one.

But that's not the scale given to the respondent. The scale is given as going from "not at all likely" to "very likely" to recommend. There isn't an option for likely to recommend against. The low end of the scale probably captures some, but to assume it is near 100% is a mistake.

Re: If your customers don't talk, NPS is a vanity metric

#52

Post by tinco »

superjan wrote: post #50
tinco wrote: post #32

Earlier quoted context omitted.

No, the goal of tracking NPS is to find out whether the customer enjoys the product enough that they would recommend it. Whether the customer would recommend it to other potential customers is almost completely irrelevant. The article is the author realizing what NPS is meant for, not some hot take. Asking "How did you learn about us?" is a question that helps evaluate your marketing and sales pipeline. Asking "Would…

Ok. I am clearly out of my comfort zone here. So NPS is not a proxy for organic marketing success, but for customer satisfaction. Is the article’s point then that the question is sometimes too hypothetical? I still think that it is useful to track actual reccomendations when that is expected to happen.

The article is literally just not on point. They're confused about what NPS is for, and wrote a confused article about it.

I agree that it's very useful to track where your leads come from, and I love the short radio button lists that allow you to select a source.

If you feel like you should only ask one question, and it's either that or the NPS question, then it really depends on what your goals are at that point. Are you trying to figure out if your customers are happy with your product, or are you trying to figure out which of your marketing channels is most effective?

I think tracking NPS over time is really powerful, so then you're tracking not an absolute number, but you're tracking if customers have recently changed their opinion of you.

Re: If your customers don't talk, NPS is a vanity metric

#53

Post by ggm »

mhuffman wrote: post #48
ggm wrote: post #6

I work in a not for profit functional monopoly space. A member body distributing resources according to policy. I can get why we do satisfaction scores, but NPS never made sense to me. Like, does the justice system do NPS for family law cases? It's not like you can get divorced in a 7-11 by the clerk, and even mediation is a process inside the model so NPS in a non competitive self-regulated monopoly.. what does it e…

>what does it even mean to recommend to others? I will give you a real-world example. My girlfriend works for the government and they have an initiative to do outreach to people in the local community and let them know of all the funded help available to people for different situations. However, they can't go to all 30k+ households at once and even in targeted areas (which is the way it works now) it takes a very lon…

That's a good counter example. Not "would you recommend" but "if you were trying to help somebody do you believe we've done enough to make it easy to explain how to use us"

Re: If your customers don't talk, NPS is a vanity metric

#55

Post by dredmorbius »

smitec wrote: post #24
beepbooptheory wrote: post #23

I know this is silly but for a split second I gasped when I saw the name of the author here.

One day I hope to do enough that Google doesn’t suggest my name is a typo.

NB: https://news.ycombinator.com/item?id=39086041>

https://news.ycombinator.com/newsguidelines.html>

Re: If your customers don't talk, NPS is a vanity metric

#56

Post by pflenker »

nulbyte wrote: post #51
pflenker wrote: post #17

Earlier quoted context omitted.

Not defending NPS here - I don't use it - but some of your assumptions are wrong. > They can't; there are people who answer 10 but will never recommend, and there are people who answer 0 but already have and will again. What matters with NPS is trend over time, and getting the numbers at a scale. Yes, there are people who randomly click on one end of the scale or the other, but the assumption is that on average the p…

> Yes, there are people who randomly click on one end of the scale or the other, but the assumption is that on average the portion of these people is stable. That assumption is predicated first on the idea that people can and will tell you their likelihood to recommend within a reasonable degree of accuracy. I don't think they do. 1: https://www.xminstitute.com/data-snippets/gap-consumer-recom... 2: https://hbr.org/2…

Re. your first point: When working with NPS, you would not actually assume that 1.000 people who voted with the highest possible value actually translate to 1.000 actual recommendations. I think not even the most vocal proponents of NPS would claim this. What matters is the _trend over time_, and that trend allows drawing conclusions regardless of actual individual customer behaviour, unless you have reason to believe that what customers actually do is completely independent of the score they give. If today 50% of respondents rate positively, and tomorrow 25% do, for large enough pools of respondents, something has gotten worse and it's not a stretch to assume that the total number of recommendations that go around for your product will go down.

Re. your second point: You are right that it is not explicitly asked for, but that does not imply that the core assumption would be faulty.

So much depends on the actual product being sold, and many other aspects, but being an imperfect metric or only a rough approximation would not imply that the metric itself is garbage, as the comment I replied to stated.

Here's a simplified example: Let's say I own a web shop, where growth implies a growth in sales. Someone on the positive end of the scale is more likely to shop again (contributing to keeping the current growth rate stable), and more likely to recommend my shop to others (contributing to positive growth). Someone in the middle or on the negative side is less likely to shop again (contributing to negative growth) or even actively recommend against my product (counteracting any positive growth the promoters would cause).

Does the NPS tell me anything about _actual additional sales_ I can expect? No. Does it tell me anything about _actual customers I will lose_? Also no. But it is one predictor of future growth, and as such useful.

Does my speedometer tell me if I'm driving in the right direction? No. Does it show how many traffic jams are ahead? Also no. But it is one predictor of when I will arrive at my destination, and as such useful.

Re: If your customers don't talk, NPS is a vanity metric

#57

Post by naijaboiler »

pflenker wrote: post #56
nulbyte wrote: post #51

Earlier quoted context omitted.

> Yes, there are people who randomly click on one end of the scale or the other, but the assumption is that on average the portion of these people is stable. That assumption is predicated first on the idea that people can and will tell you their likelihood to recommend within a reasonable degree of accuracy. I don't think they do. 1: https://www.xminstitute.com/data-snippets/gap-consumer-recom... 2: https://hbr.org/2…

Re. your first point: When working with NPS, you would not actually assume that 1.000 people who voted with the highest possible value actually translate to 1.000 actual recommendations. I think not even the most vocal proponents of NPS would claim this. What matters is the _trend over time_, and that trend allows drawing conclusions regardless of actual individual customer behaviour, unless you have reason to believ…

Thank you for taking time to explain this to people who are think NPS should be interpreted literally.

No, it’s really just a decent enough proxy for how your product is trending over time among your users.

Individual responses are uninterpetable. Single NPS survey is also not very interpretable. But with enough people and over enough time, it is a useful signal

Post reply on HN