Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NPS doesn’t say anything particularly useful

cranberryblog.substack.com

Post reply on HN
Print view View original

11–20 of 83 posts

Re: NPS doesn’t say anything particularly useful

#11

Post by yjftsjthsd-h »

madeofpalk wrote: post #5

My "problem" with NPS is when essential services, which I have next to no choice over using, ask me if i would recommend them to others. No, IRS, I would never recommend your service to friends or family. That's not really indicative of anything.

Ah, like the good old

> How likely are you to recommend Windows 10 to a friend or colleague? Please explain why you gave this score.

> I need you to understand that people don't have conversations where they randomly recommend operating systems to one another

Re: NPS doesn’t say anything particularly useful

#12

Post by lmeyerov »

keiran_cull wrote: post #3
yuliyp wrote: post #2

The point of NPS vs free cash flow or operational metrics is that it is in theory a leading indicator instead of a trailing indicator. It's nice (but a bit too late) to know that your business is screwed because your customers left, it's another to know that your business is about to be screwed because your customers are on the verge of leaving if an opportunity appears. The criticisms of NPS pointed out (that it's a…

What exactly would you suggest it's a leading indicator of? Churn? Ultimately if it matters, it should have to be an input to the cash flow time series -- mediated by churn -- no? (Article means to frame free cash flow as a series rather than a point-in-time value). If you're interested in forecasting, as a company, you should know the things that mean your customer is not doing well within the specific context of yo…

Hopefully, it should be a leading indicator for negative churn -- per-account and new account growth / virality -- and for more complicated products, a way to slice that across different personas and features. ('bakers at high-end restaurants have high NPS for the new precision thermometer tracker while those at regular shops ignore them and have low NPS.')

Esp in early days of B2B products, it's hard to get that because you don't have the volumes and velocity of b2c nor a good way to detect and attribute viral activation. If you are in a startup and not riding the channel of some megacorp, even more so. Alternatives like signups or other activation checkpoints, or say qualitative interviews, are also interesting, but even more spotty. (Ex: startups raising based on GitHub stars.) We don't do NPS as we have our plate full with known funnel holes through less annoying data collection methods, but as soon as we are happy with the baseline funnel, that's the simple next step.

Re: NPS doesn’t say anything particularly useful

#13

Post by xraystyle »

actually_a_dog wrote: post #7

I suspect the concept of NPS is, if not the source of, a major driver of the whole "anything less than 10/10, 5 stars, etc. is basically a fail" meme. There seem to be signifiant toxic effects from this on customer-facing employees, in my experience.

This is absolutely a massive issue with it. Apple did this with retail employees back in the day and it was terrible. Employee reviews were based on it, managers had to call customers and apologize if the customer didn't leave a 9 or 10 rating, it was absolutely awful.

One of the biggest problems with it is that an average person, unaware of what NPS is, doesn't understand that giving any rating less than a 9 is essentially giving a rating of zero.

If I have what I would consider to be an average interaction with a business, e.g. I just buy something and leave, no need for support, no problems to deal with, etc., that seems average to me. Based on a non-NPS understanding of a 0-10 scale, I'd say that's what, a 7? But the business now considers this as a failure on the part of the person that helped me at that store.

This is why sales people and phone reps are constantly now asking you to give them a 10/10 rating if you receive a survey, because even if they literally just took your money and handed you change, their jobs depend on you acting as if you had a heart attack and they saved your life by performing CPR or something.

It's honestly a terrible system that produces no meaningful feedback for the company and causes employees to do whatever they can to game the numbers. All you're measuring with NPS is how good your employees are at juking the stats, nothing more.

Re: NPS doesn’t say anything particularly useful

#14

Post by Spooky23 »

xboxnolifes wrote: post #10
madeofpalk wrote: post #5

My "problem" with NPS is when essential services, which I have next to no choice over using, ask me if i would recommend them to others. No, IRS, I would never recommend your service to friends or family. That's not really indicative of anything.

No PatientCo, I did not enjoy paying my medical bills through your system. I have no intention of recommending your bill payment system to my family. Why is this even asked?

I switched doctors when their billing became impossible to deal with.

Assuming they gave a shit, that survey response would have been useful to the practice.

Re: NPS doesn’t say anything particularly useful

#15

Post by sixdimensional »

xraystyle wrote: post #13
actually_a_dog wrote: post #7

I suspect the concept of NPS is, if not the source of, a major driver of the whole "anything less than 10/10, 5 stars, etc. is basically a fail" meme. There seem to be signifiant toxic effects from this on customer-facing employees, in my experience.

This is absolutely a massive issue with it. Apple did this with retail employees back in the day and it was terrible. Employee reviews were based on it, managers had to call customers and apologize if the customer didn't leave a 9 or 10 rating, it was absolutely awful. One of the biggest problems with it is that an average person, unaware of what NPS is, doesn't understand that giving any rating less than a 9 is esse…

This, to me, is a general problem with a purely quantified, metrics driven approach to management, not just limited to NPS. I'm not demonizing these approaches, just saying that you have to balance humanity with data.

I like to look at this kind of thing as the one of the dark sides/dark patterns of using data for decisionmaking.

Qualitative measures are important, as is maintaining as much humanity as possible, to have a balanced and healthy culture. Being solely metrics/data driven can lead to cold, heartless, damaging culture (might be efficient or make profit, but very dehumanizing).

Re: NPS doesn’t say anything particularly useful

#16

Post by riversflow »

mdorazio wrote: post #4

I didn't see discussed here the main drivers you might want NPS vs. outcome- (and company) specific metrics: 1) it's harder to game and/or bullshit. This is a massive issue with surveying customers and then using their responses to feed back into internal performance goals & rewards. 2) because it has grown to be such a common customer experience metric, it's possible to benchmark against others, which isn't really p…

> it's harder to game and/or bullshit. . . it's possible to benchmark against others,

I don't have access to A/B testing of a large business to understand how things affect responses. But I certainly intuit that there is a significance in when/how you ask someone for feedback. e.g. If you ask your question directly after the checkout screen you'll get a significantly different response than if you survey people randomly who visit your website, or email customers a set period after making a purchase/ upon transaction completion.

I've seen all these, which brings me to the second part, how do you benchmark against a competitors data, and why would you trust a competitors data?

Re: NPS doesn’t say anything particularly useful

#17

Post by hakfoo »

yuliyp wrote: post #2

The point of NPS vs free cash flow or operational metrics is that it is in theory a leading indicator instead of a trailing indicator. It's nice (but a bit too late) to know that your business is screwed because your customers left, it's another to know that your business is about to be screwed because your customers are on the verge of leaving if an opportunity appears. The criticisms of NPS pointed out (that it's a…

One of the most common contexts for NPS scoring is giving them to direct retail customers as a phone/email followup survey. This is the most dangerous context for it, because a customer who isn't familiar with the system, is sort of primed to have the wrong expectations.

They're thinking in terms of "how satisfied was I", which is a different scale entirely with "would you recommend the product". But the NPS question, even when explicitly explained, reads close enough that they'll answer it with the "how satisfied am I" answer,

NPS might have worked in a context of a professionally proctored focus group where everyone understood the question and discussed it, but I'm worried you're losing a lot of information when you turn it to one digit on a keypad.

Re: NPS doesn’t say anything particularly useful

#18

Post by foo4u »

What’s more interesting to me than the NPS scores are the free form comment section below the 0-10 rating.

Never mind that anything below X is considered neutral or a detractor. You asked a user / customer/ etc to rate you on a scale of 1-10 and then tell you why.

Want to know what you’re truly screwing up on? Take the feedback on your 1-6 scores seriously and you can find the low hanging fruit to take a product from mediocre to a great user experience.

That’s the true NPS value. It’s all about how you handle the feedback.

Re: NPS doesn’t say anything particularly useful

#19

Post by erik_seaberg »

xraystyle wrote: post #13
actually_a_dog wrote: post #7

I suspect the concept of NPS is, if not the source of, a major driver of the whole "anything less than 10/10, 5 stars, etc. is basically a fail" meme. There seem to be signifiant toxic effects from this on customer-facing employees, in my experience.

This is absolutely a massive issue with it. Apple did this with retail employees back in the day and it was terrible. Employee reviews were based on it, managers had to call customers and apologize if the customer didn't leave a 9 or 10 rating, it was absolutely awful. One of the biggest problems with it is that an average person, unaware of what NPS is, doesn't understand that giving any rating less than a 9 is esse…

Most internal surveys at my company use a scale of 1 to 6, where 1=4 and 5=6. Our managers can’t even see an improvement from “terrible” to “fairly decent” in the underlying distribution.

Sometimes they require writing something to justify a bad score (but not a good score!)

Re: NPS doesn’t say anything particularly useful

#20

Post by afarrell »

sixdimensional wrote: post #15
xraystyle wrote: post #13

Earlier quoted context omitted.

This is absolutely a massive issue with it. Apple did this with retail employees back in the day and it was terrible. Employee reviews were based on it, managers had to call customers and apologize if the customer didn't leave a 9 or 10 rating, it was absolutely awful. One of the biggest problems with it is that an average person, unaware of what NPS is, doesn't understand that giving any rating less than a 9 is esse…

This, to me, is a general problem with a purely quantified, metrics driven approach to management, not just limited to NPS. I'm not demonizing these approaches, just saying that you have to balance humanity with data. I like to look at this kind of thing as the one of the dark sides/dark patterns of using data for decisionmaking. Qualitative measures are important, as is maintaining as much humanity as possible, to h…

https://en.wikipedia.org/wiki/McNamara_fallacy

“The first step is to measure whatever can be easily measured. This is OK as far as it goes. The second step is to disregard that which can't be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what can't be measured easily really isn't important. This is blindness. The fourth step is to say that what can't be easily measured really doesn't exist. This is suicide.”

Post reply on HN