Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NPS doesn’t say anything particularly useful

cranberryblog.substack.com

Post reply on HN
Print view View original

31–40 of 83 posts

Re: NPS doesn’t say anything particularly useful

#31

Post by sixdimensional »

afarrell wrote: post #20
sixdimensional wrote: post #15

Earlier quoted context omitted.

This, to me, is a general problem with a purely quantified, metrics driven approach to management, not just limited to NPS. I'm not demonizing these approaches, just saying that you have to balance humanity with data. I like to look at this kind of thing as the one of the dark sides/dark patterns of using data for decisionmaking. Qualitative measures are important, as is maintaining as much humanity as possible, to h…

https://en.wikipedia.org/wiki/McNamara_fallacy “The first step is to measure whatever can be easily measured. This is OK as far as it goes. The second step is to disregard that which can't be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what can't be measured easily really isn't important. This is blindness. The fourth step is to s…

Thank you for this very interesting share, I had not heard of this before!

Re: NPS doesn’t say anything particularly useful

#33

Post by ekianjo »

hn_throwaway_99 wrote: post #9

I see proposals against NPS like this from time to time, but they always tend to ignore the underlying science that went into making NPS in the first place, primarily that there was a very strong correlation between the calculated NPS score and business performance. Also, I find that many of these articles have somewhat of a caricaturish view of how NPS is used in the real world. "It's not 'oh no, our NPS score is do…

> primarily that there was a very strong correlation between the calculated NPS score and business performance.

There is avsolutely no science linking performance and NPS. Just pseudo science.

Re: NPS doesn’t say anything particularly useful

#34

Post by Lazare »

hn_throwaway_99 wrote: post #9

I see proposals against NPS like this from time to time, but they always tend to ignore the underlying science that went into making NPS in the first place, primarily that there was a very strong correlation between the calculated NPS score and business performance. Also, I find that many of these articles have somewhat of a caricaturish view of how NPS is used in the real world. "It's not 'oh no, our NPS score is do…

> they always tend to ignore the underlying science that went into making NPS in the first place, primarily that there was a very strong correlation between the calculated NPS score and business performance.

There is no underlying science behind NPS. It was an arbitrary measure with no research or model behind it. Also, it doesn't work very well, in large because because there is no strong correlation between NPS and anything else, because - as this article makes clear - NPS is an extremely noisy number. Look at the graphs - you can have enormous variations in NPS, with zero change in underlying customer sentiment. Obviously that can't correlate with anything useful.

I think what you're doing is conflating the idea of "asking how likely people are to recommend your product" with NPS. The former is a decent idea, and it does have some (limited) science behind it! There really was a study done that really did find that question was an excellent predictor of many key results. (...and some others that found it was not, but hey, it's something.)

So yes, asking people the question and reading the results is probably a good idea. ...but that's not what NPS is.

> an unexpected drop in NPS is something to investigate

It really, really isn't. And I would challenge you to point to any evidence to the contrary.

Re: NPS doesn’t say anything particularly useful

#35

Post by edsrzf »

hn_throwaway_99 wrote: post #9

I see proposals against NPS like this from time to time, but they always tend to ignore the underlying science that went into making NPS in the first place, primarily that there was a very strong correlation between the calculated NPS score and business performance. Also, I find that many of these articles have somewhat of a caricaturish view of how NPS is used in the real world. "It's not 'oh no, our NPS score is do…

Honest question: what science? Any good papers you could recommend?

After a quick search it seems that NPS originated in a Harvard Business Review article, which I don't consider a credible source of scientific results. The scientific papers I'm seeing mostly seem pretty skeptical, judging from the abstracts.

Re: NPS doesn’t say anything particularly useful

#36

Post by Lazare »

You know what would be vastly better than NPS?

Ask people every so often how they're feeling about the product. Let them pick a thumbs up or a thumbs down. If they pick thumbs down, say "we're sorry you hear that. Please let us know what's bothering you", and display a little text box. Carefully read every comment you receive.

Asking unhappy users for feedback is a great idea. The 0 to 10 scale and - especially - the bizarre calculation NPS does with it - is statistical malpractice that just isn't mathematically capable of doing what NPS supports claim.

Re: NPS doesn’t say anything particularly useful

#37

Post by abeyer »

hn_throwaway_99 wrote: post #9

I see proposals against NPS like this from time to time, but they always tend to ignore the underlying science that went into making NPS in the first place, primarily that there was a very strong correlation between the calculated NPS score and business performance. Also, I find that many of these articles have somewhat of a caricaturish view of how NPS is used in the real world. "It's not 'oh no, our NPS score is do…

Like sibling comment said, I'd love to see actual science behind it.

> a caricaturish view of how NPS is used in the real world. "It's not 'oh no, our NPS score is down, let's hire more people to focus on customer 'happiness'"

That sounds exactly like what responses at many companies would be. There's often much nuanced discussion about how/when/why to adopt certain metrics, but once they are they become sacrosanct and any deviation from "good" results is a crisis needing immediate action.

Re: NPS doesn’t say anything particularly useful

#38

Post by paulmooreparks »

hn_throwaway_99 wrote: post #9

I see proposals against NPS like this from time to time, but they always tend to ignore the underlying science that went into making NPS in the first place, primarily that there was a very strong correlation between the calculated NPS score and business performance. Also, I find that many of these articles have somewhat of a caricaturish view of how NPS is used in the real world. "It's not 'oh no, our NPS score is do…

I generally agree, but I think one problem is when improving the NPS score becomes the sole objective and not improving the underlying problems that caused it to go down. In other words, calling a doctor to treat the canary does not improve the conditions in the coal mine that made the canary sick.

Re: NPS doesn’t say anything particularly useful

#39

Post by ketralnis »

nhumrich wrote: post #32

The real problem with NPS is when a certain score is a goal. After all goodharts law states that is the case with any good measure.

Yep. I had an advertising-funded app that asked me to fill out a survey about why I rated it poorly in a modal I can’t skip. So I always gave it a good score, until I stopped using it because it interrupted me so often to ask. They lost me and they’ll never know why

Re: NPS doesn’t say anything particularly useful

#40

Post by joe2784 »

It's not fair to compare how "important" different metrics are to the whole business. A business is composed of and built on top of many sectors. That's why we have Finance team, Sales team, Marketing team, Product and Development team..., and not just one "we're a company" team! Free cash flow is important for business operations. Churn is important to predict sales. NPS, though not directly affecting the "money", might be a good indicator for the marketing team. There are a lot more important marketing metrics than NPS, but for a big corporation's branding team, especially in consumer goods industry, it might be useful to look at the "trend" of their NPS over a long period of time. NPS has noise just because of its nature of being collected with surveys. Like any questionnaires or surveys, if you design it well, you can reduce the noise.
Post reply on HN