You Only Need to Test with 5 Users (2000)
nngroup.com
You Only Need to Test with 5 Users (2000)
1–10 of 112 posts
Re: You Only Need to Test with 5 Users (2000)
#2This is an excellent point, and the much more fascinating corollaries:
- a product designer/manager of something with 1,000,000 users won’t learn more about usability than a product designer/manager of something with 15 users. All those measurements of flows and secret at scale analytics data is sort of worthless for the purposes of usability.
- people with 15 users worth of learning about usability instead of 0 users are way undervalued, while people with 1,000,000 are way overvalued
- “we don’t know until we test it,” the #1 refrain of big company design nowadays, is intellectually bankrupt for most free software, since if it looked bad for 15 users it’s probably going to still be bad for 1,000,000
- after you’ve shown something to 15 people and they don’t like it due to usability problems, you’re extremely unlikely to find 1,000,000 more who will like it.
This also appeals to my instinct that there is something learnable about design, that great design is achieved by people and not by massive datasets.
Re: You Only Need to Test with 5 Users (2000)
#3This is an excellent point, and the much more fascinating corollaries: - a product designer/manager of something with 1,000,000 users won’t learn more about usability than a product designer/manager of something with 15 users. All those measurements of flows and secret at scale analytics data is sort of worthless for the purposes of usability. - people with 15 users worth of learning about usability instead of 0 user…
This may be true for software used when working alone, but for software used by groups, you are not going to figure out group effects without testing with different group sizes. Consider the differences testing a game with 1, 2, 5, 100, or 1000 users.
Re: You Only Need to Test with 5 Users (2000)
#4Ok, but how are you going to test for accessibility?
Re: You Only Need to Test with 5 Users (2000)
#5The author ends by saying you need five users from each distinct type of user, with the example of parent and child.
Unfortunately the author neglected to mention that the vast majority of projects already have multiple distinct groups- abled people, blind people, deaf people, and are just a start. In many places these are legally required considerations.
Re: You Only Need to Test with 5 Users (2000)
#6Anybody who studied a bit of probability theory knows that this is both false and wrong.
Statistically significant sampling can't be done with only five users.
Confidence will only be reasonably sufficient for a population of 5 (at most). This is like if you addressed your product to 5 people (I don't believe it to be likely, you probably have way more than 5 users).
See this page for assistance with computing sample size function of population and confidence https://www.surveymonkey.com/mp/sample-size-calculator/
This article probably keeps being reposted because some people try to save money on user testing
...and that's probably why we keep having sh*tty products out in the wild.
Re: You Only Need to Test with 5 Users (2000)
#7This is an excellent point, and the much more fascinating corollaries: - a product designer/manager of something with 1,000,000 users won’t learn more about usability than a product designer/manager of something with 15 users. All those measurements of flows and secret at scale analytics data is sort of worthless for the purposes of usability. - people with 15 users worth of learning about usability instead of 0 user…
Agree, as the massive datasets are telling you more about what people want instead how the product works.
That’s a big distinction, as when you look at a million people, 80% are are either the same, or don’t really understand what they are doing at all.
Modern designers usually assume that people are dolts and need simplicity. Optimal in my opinion is providing quality on-boarding and understanding behavior of someone who knows what they are doing as well.
Re: You Only Need to Test with 5 Users (2000)
#8I don't have much to do with UX, but I will add one insight. Try finding users with esoteric ways of working, and ensure the site works for them. By esoteric I mean blind users of screen readers, users of old, non js browsers, people on corp or school networks where things may be blocked, people on extremely small screens or extremely large ones (TVs with remotes), people who don't own or don't want to use a credit card (there's a widespread credit card phobia in i.e. Poland when it comes to online purchases), people using a different language and keyboard layout, particularly ltr, significant when it comes to desktop apps etc. I'm a screen reader user myself, and I find websites that might be beautiful, but are utterly inaccessible constantly. I've either encountered, or was a witness of, usability difficulties in all of those categories I outlined. For each one, I could provide an example of a website or app that I/someone had to abandon for just that reason, and this is just me. I'm sure there are more nichse I haven't thought about.
Re: You Only Need to Test with 5 Users (2000)
#9Anybody who studied a bit of probability theory knows that this is both false and wrong. Statistically significant sampling can't be done with only five users. Confidence will only be reasonably sufficient for a population of 5 (at most). This is like if you addressed your product to 5 people (I don't believe it to be likely, you probably have way more than 5 users). See this page for assistance with computing sample…
Anybody who studied a bit of probability theory knows that all this depends on the underlying probability distributions. If one draws five samples from a Gaussian distributions with an unknown mean and a standard deviation of 10, and none of the samples is below 50, then indeed it is very unlikely that the mean is 20.
Re: You Only Need to Test with 5 Users (2000)
#10Anybody who studied a bit of probability theory knows that this is both false and wrong. Statistically significant sampling can't be done with only five users. Confidence will only be reasonably sufficient for a population of 5 (at most). This is like if you addressed your product to 5 people (I don't believe it to be likely, you probably have way more than 5 users). See this page for assistance with computing sample…
Seems like for large effects, this depends on how often the effect occurs. If it's something that happens for everyone, you only need one user.