Live data from Hacker News

NPS doesn’t say anything particularly useful

cranberryblog.substack.com

71–80 of 83 posts

Re: NPS doesn’t say anything particularly useful

#71
Companies with a promoter level score during the pandemic grew their business. There is a direct correlation between a promoter NPS and revenue growth. The percentage of companies that now provide a promoter level service is now over 20%, up from 13% before the pandemic. The percentage that are rated a detractor level score also grew to nearly 50%. These org’s revenues either shrank or stagnated. This is an existential issue for organizations, not a nuisance KPI.

Companies that are leaders in NPS are leaders in employee engagement. This post sounded pretty frustrated with an employer/contract…

NPS is a measure of past experiences, it isn’t a GPS navigation system to improvement. It is also 1 question, so it is the most simple to collect. However even this gets screwed up all the time. Interfaces such as a telephone where 10 can be measured as the first digit 1, or PIN pads where they are a nuisance to a customer when in a hurry to pay and missed when paying with NFC.

Understanding negative NPS is like understanding a traffic ticket with no other data. Why did I get a traffic ticket? Well, what did or didn’t you do and how fast were you doing it? Were you a jerk to the cop? Was the cop stressed by something else that day?

To be a leader in NPS is an organizational effort, not the result of divining an insight from a few survey answers. Ideally, customer journeys maps have been developed and problem areas are self evident before an NPS score arrives.

Re: NPS doesn’t say anything particularly useful

#72

Why do everyone seem to think that management does not have actual good intentions with measuring NPS as a KPI? The companies I've worked with have all had genuinely good intentions making the best possible experience for the user, because that is what ultimately wins in the end. If someone scores 1-6, you ask them to provide additional feedback, to learn from it - problem solved and everyone has all the info they ne…

20% of companies are knocking it out of the park, and 50% are nowhere near a leaderboard. I expect such folks are correct in their observation, but work for the 50%. Perhaps you work for one that is getting a promoter score.

My advice to friends looking for a job, ask them what their NPS is. The leaders in NPS are leaders in employee engagement. If they don’t know, or it’s bad and your role isn’t to make it better, look elsewhere.

Re: NPS doesn’t say anything particularly useful

#73
This post misses that the primary benefit of an NPS program is not the metric—that's a nice byproduct—it's the dialogue with customers.

As a single metric Net Promoter Score is OK. It's extremely easy for customers to answer. So there's volume in the number of answers. You can learn more by segmenting different ways—Users vs Admins, Enterprise vs SMB, Verticals, etc..

The company I currently work for puts a lot into NPS, not necessarily the metric, but the process. Everyone that leaves feedback gets a follow-up email and interview from someone in Product or User Research. That feedback is then organized in Productboard where it is grouped with similar requests/issues. Overall those interviews heavily weigh what features we refine and build.

For some background, I previously designed an NPS tool and have written about NPS pretty extensively.

https://solomon.io/understanding-net-promoter-score/

Re: NPS doesn’t say anything particularly useful

#74
post #59

Earlier quoted context omitted.

NPS has the advantage of being concrete and simple, producing evenly sampled feedback. This is hard to get any other way. A proper NPS asks only one question and requires only a single click to answer -- eg "Scale of 1-10 how likely are you to recommend to a friend?" > click [8] > done. There's a free-text "Tell us more" field that is totally optional. This means users actually answer it. NPS feedback is the closest…

In an almost Goodhart’s Law case, now that I know how NPS is used, I answer these questions differently, being far more prone to use 6, 7, or 8 to express my opinion.

It probably depends on the survey. If I just want to get out the door of the service department at the car dealership I'll probably give a perfect score unless they really screwed something up. I don't want to explain a non-perfect score and probably shouldn't give one so long as the experience was "OK."

On the other hand, like many, I probably tend to follow something like the XKCD star rating levels (https://xkcd.com/1098/) for products. And for employee surveys, I generally answer somewhere in the mildly positive range to most questions.

I'm not sure I've ever answered an NPS survey but, for most companies, I'd probably be somewhere in a similar range, even for companies I'm perfectly fine with most of the time.

The more complex the transactions/products the more reliably you'll have areas of gripe-age. A book or movie is rarely 5 stars for me. A USB cable pretty much works or it doesn't.

Re: NPS doesn’t say anything particularly useful

#75
post #61

Earlier quoted context omitted.

If you go into the trouble of collecting it, it's reasonable to use as a KPI, as it carries useful information. I imagine you are annoyed that they optimize for it. And yes, it's not reasonable to optimize for it. Why do people optimize every KPI?

>Why do people optimize every KPI? Because if it's not worth optimizing around, how key can it be?

You can do a lot of things with a number that are not optimizing.

For example, you can satisfy it, you can alert on behavior, or you can use it as a control (but ok, maybe this one makes a two number KPI instead of two KPIs).

Re: NPS doesn’t say anything particularly useful

#76
post #56

Earlier quoted context omitted.

CSAT is not standardized and there are many ways to calculate it, one of which you have linked. I've seen firsthand many of the largest brands calculate it differently.

Yeah, I agree it doesn't have the universal agreement on implementation that NPS does. Here's an alternative that I believe does and is referenced in some of the NPS skeptical research: https://en.wikipedia.org/wiki/American_Customer_Satisfaction...

That "measures the satisfaction of consumers across the U.S. economy", very different from Brand's Customer Experience Surveys. Having worked as a software vendor in the CX field, I saw a lot of different implementations of the "same" survey metrics. NPS was only standardized one (I think because its from b school in the 80's).

Re: NPS doesn’t say anything particularly useful

#77

Earlier quoted context omitted.

The "science" is quite dubious: https://articles.uie.com/net-promoter-score-considered-harmf... The reason why NPS exists is because it's an easy number to calculate that sounds convincing, and manager-types love numbers without having to think too hard about how they got them.

That article just links to the wikipedia page. The main criticism paper[1] reasonably successfully debunks NPS as uniquely valuable, but more or less shows it as of comparable value to other measures of customer satisfaction. 1: https://web.archive.org/web/20200716065914/https://pdfs.sema...

Yes, that's correct. The broader point is that it's really hard to boil down something as complex as user satisfaction to a number. My time spent with UX professionals has taught me that all of these measures are lossy, and my time with data scientists has taught me that once you start depending too much on lossy measures, you're going to be led astray.

But management-types love little numbers like NPS, so it usually gets done anyways, especially in big orgs. And then it goes downhill, because PMs and leads are incentivized to optimize for the particular number their management chain tells them matters, and they game it because that's how people work. Later on, frustrated engineers and PMs who aren't a part of that game wonder if they're the crazy ones because they see very real customer frustrations brushed aside by an org structure that doesn't seem to care much about what users actually tell them anymore. Or they say they do, but never incentivize the rest of the org to address issues coming in through verbatim feedback.

Maybe, someone with the title of VP eventually wonders why a competitor is doing really well, looks at verbatim feedback themselves (it's usually been given to them already but they forgot about it), and then realizes they've been steering a big ship in the wrong direction, and causes chaos all kinds of chaos in the process. The sycophants in the org all line up to agree with them and declare they were right all along (and get rewarded at the end of the year), line managers and their reports are left confused because of the jolting priority shift, and people who might have felt vindicated by the direction shift are wondering if their leadership are actually fit to be leaders or not.

Is that all the fault of NPS or other numbers? Of course not! But they're an easy and generally accepted way for bad leaders to hide their bad leadership.

Re: NPS doesn’t say anything particularly useful

#78

I see proposals against NPS like this from time to time, but they always tend to ignore the underlying science that went into making NPS in the first place, primarily that there was a very strong correlation between the calculated NPS score and business performance. Also, I find that many of these articles have somewhat of a caricaturish view of how NPS is used in the real world. "It's not 'oh no, our NPS score is do…

> As a software engineer I love reading our NPS reports, because they always have great tidbits of info

My only experience of NPS is when I worked retail/sales in my first job, and it absolutely sucked in that context.

Less “great tidbits” and more “I just lost 30% of my commission this month because of something 100% outside of my control”.

Re: NPS doesn’t say anything particularly useful

#79

Earlier quoted context omitted.

This is absolutely a massive issue with it. Apple did this with retail employees back in the day and it was terrible. Employee reviews were based on it, managers had to call customers and apologize if the customer didn't leave a 9 or 10 rating, it was absolutely awful. One of the biggest problems with it is that an average person, unaware of what NPS is, doesn't understand that giving any rating less than a 9 is esse…

> giving any rating less than a 9 is essentially giving a rating of zero. Well, at least its not a negative number!

If you convert it to IEEE 754, it could be -0. ;)

Re: NPS doesn’t say anything particularly useful

#80

Earlier quoted context omitted.

This is absolutely a massive issue with it. Apple did this with retail employees back in the day and it was terrible. Employee reviews were based on it, managers had to call customers and apologize if the customer didn't leave a 9 or 10 rating, it was absolutely awful. One of the biggest problems with it is that an average person, unaware of what NPS is, doesn't understand that giving any rating less than a 9 is esse…

I remember when I worked my first job as tech support for an ISP our entire performance was measured by NPS. The company had a question asking 'how would you rate the rep' and 'how would you rate the company'. But, the rep question was a decoy, and the company question actually counted in my personal NPS stats. I would get so many 10s for the rep (which does not count at all) and 1s for the company which would oblite…

Damn, this sounds like the definition of "evil".
Post reply on HN