Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You Only Need to Test with 5 Users (2000)

nngroup.com

Post reply on HN
Print view View original

11–20 of 112 posts

Re: You Only Need to Test with 5 Users (2000)

#11

Post by ddebernardy »

an_opabinia wrote: post #2

This is an excellent point, and the much more fascinating corollaries: - a product designer/manager of something with 1,000,000 users won’t learn more about usability than a product designer/manager of something with 15 users. All those measurements of flows and secret at scale analytics data is sort of worthless for the purposes of usability. - people with 15 users worth of learning about usability instead of 0 user…

+1 but several of your points are incorrect or need caveats.

Your 1st point isn't correct, in that you will learn interesting things from 1m users that you won't from 15. The thing is that the 15 will tell you why, whereas the 1m won't unless you ask precisely the right question (which is a problem in itself). It basically takes experience here... (And this is something an AI may eventually become very good at.)

Your 3rd and 4th point aren't correct, in that given some sampling error you may very well find that what 15 people don't like 1m will.

I'm in full agreement with you in principle, in the sense that I firmly hold that far too many start-ups do quantitative without qualitative and end up with the wrong conclusions; but you can't just wave away quantitative data like that.

Re: You Only Need to Test with 5 Users (2000)

#13

Post by julienreszka »

skybrian wrote: post #10
julienreszka wrote: post #6

Anybody who studied a bit of probability theory knows that this is both false and wrong. Statistically significant sampling can't be done with only five users. Confidence will only be reasonably sufficient for a population of 5 (at most). This is like if you addressed your product to 5 people (I don't believe it to be likely, you probably have way more than 5 users). See this page for assistance with computing sample…

Seems like for large effects, this depends on how often the effect occurs. If it's something that happens for everyone, you only need one user.

Obviously

Re: You Only Need to Test with 5 Users (2000)

#14

Post by ivalm »

julienreszka wrote: post #6

Anybody who studied a bit of probability theory knows that this is both false and wrong. Statistically significant sampling can't be done with only five users. Confidence will only be reasonably sufficient for a population of 5 (at most). This is like if you addressed your product to 5 people (I don't believe it to be likely, you probably have way more than 5 users). See this page for assistance with computing sample…

Yes and no, people have relatively convergent views on usability. Statistically, you can think of proposition A "user from set U affirms that object O has property Q". You then sample opinions for U on whether this proposition is true. Each sampling is a Bernoulli trial parametrized by

p = Prob(A = True)

The standard error of the mean is then

sqrt(p*(1-p)/N)

where N is how many users you sampled. Suppose people are convergent in their opinion (either p=0.99 or 0.01) then even with N=5 the uncertainty in mean is less than 5%!

To make a concrete example, you only need to ask very few users if a particular object is white to be fairly confident whether the majority of people would consider a particular object to be white.

Re: You Only Need to Test with 5 Users (2000)

#16

Post by baxtr »

ddebernardy wrote: post #11
an_opabinia wrote: post #2

This is an excellent point, and the much more fascinating corollaries: - a product designer/manager of something with 1,000,000 users won’t learn more about usability than a product designer/manager of something with 15 users. All those measurements of flows and secret at scale analytics data is sort of worthless for the purposes of usability. - people with 15 users worth of learning about usability instead of 0 user…

+1 but several of your points are incorrect or need caveats. Your 1st point isn't correct, in that you will learn interesting things from 1m users that you won't from 15. The thing is that the 15 will tell you why, whereas the 1m won't unless you ask precisely the right question (which is a problem in itself). It basically takes experience here... (And this is something an AI may eventually become very good at.) Your…

Re 1st point: OP was referring specifically to usability learnings

Re: You Only Need to Test with 5 Users (2000)

#17

Post by rosser »

ivalm wrote: post #14
julienreszka wrote: post #6

Anybody who studied a bit of probability theory knows that this is both false and wrong. Statistically significant sampling can't be done with only five users. Confidence will only be reasonably sufficient for a population of 5 (at most). This is like if you addressed your product to 5 people (I don't believe it to be likely, you probably have way more than 5 users). See this page for assistance with computing sample…

Yes and no, people have relatively convergent views on usability. Statistically, you can think of proposition A "user from set U affirms that object O has property Q". You then sample opinions for U on whether this proposition is true. Each sampling is a Bernoulli trial parametrized by p = Prob(A = True) The standard error of the mean is then sqrt(p*(1-p)/N) where N is how many users you sampled. Suppose people are c…

That is to say, if all five of the users with whom you've tested your application say it's confusing, or it sucks somehow, it is diminishingly likely for that population to be the outlier [0], and if only you had tested with a few tens or dozens — let alone thousands — more, you'd see the true pattern...

Yes, statistically, it's possible for outliers to bunch like that. It's also, statistically, far less likely.

[0] Assuming, for sake of argument, a nominally representative test group.

Re: You Only Need to Test with 5 Users (2000)

#18

Post by iamaelephant »

miki123211 wrote: post #8

I don't have much to do with UX, but I will add one insight. Try finding users with esoteric ways of working, and ensure the site works for them. By esoteric I mean blind users of screen readers, users of old, non js browsers, people on corp or school networks where things may be blocked, people on extremely small screens or extremely large ones (TVs with remotes), people who don't own or don't want to use a credit c…

This is going to depend heavily on your target market. In many of the SaaS applications I have been involved in we really don't care about users with non-JS browsers, or extremely small screens, or TVs, or people without credit cards. Some of the applications I have worked on will never be translated.

Re: You Only Need to Test with 5 Users (2000)

#19

Post by visarga »

ddebernardy wrote: post #11
an_opabinia wrote: post #2

This is an excellent point, and the much more fascinating corollaries: - a product designer/manager of something with 1,000,000 users won’t learn more about usability than a product designer/manager of something with 15 users. All those measurements of flows and secret at scale analytics data is sort of worthless for the purposes of usability. - people with 15 users worth of learning about usability instead of 0 user…

+1 but several of your points are incorrect or need caveats. Your 1st point isn't correct, in that you will learn interesting things from 1m users that you won't from 15. The thing is that the 15 will tell you why, whereas the 1m won't unless you ask precisely the right question (which is a problem in itself). It basically takes experience here... (And this is something an AI may eventually become very good at.) Your…

> but you can't just wave away quantitative data like that

quantity has a quality of its own

Re: You Only Need to Test with 5 Users (2000)

#20

Post by noir_lord »

an_opabinia wrote: post #2

This is an excellent point, and the much more fascinating corollaries: - a product designer/manager of something with 1,000,000 users won’t learn more about usability than a product designer/manager of something with 15 users. All those measurements of flows and secret at scale analytics data is sort of worthless for the purposes of usability. - people with 15 users worth of learning about usability instead of 0 user…

It might even be worse still since this model (seems I may be wrong) to assume that the probability of a usability bug is constant, it might be that the share of bugs discovered by users is skewed towards the first few such that the first user finds more than the formula would predict.

It's certainly been my observation that cynical developers who test things as they go by deliberately putting in silly things into stuff they just wrote seem to get hung up less in testing.

I mean the system I inherited at work the first thing I did when I got an instance spun up was put in a negative value in the quote line quantity (which immediately broke..well almost everything) then decimal values in quantity fields where only integers made sense, then text in number fields and so on each time breaking something in a new and interesting way.

Sometimes I think it's hard not to be cynical about enterprise systems.

My old lecturer (somewhat pithily) "Almost all the testing in the world means nothing compared to 15 minutes in the hands of the 17 year old office junior"

Post reply on HN