Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You Only Need to Test with 5 Users (2000)

nngroup.com

Post reply on HN
Print view View original

31–40 of 112 posts

Re: You Only Need to Test with 5 Users (2000)

#31

Post by shkkmo »

nightcracker wrote: post #26
baxtr wrote: post #16

Earlier quoted context omitted.

Re 1st point: OP was referring specifically to usability learnings

I'd argue that usability also must include accessibility and I highly doubt that in your 15 user sample set you will cover the deaf, the (color) blind, the dyslexic, the paralyzed, the epileptic and various other groups of people with special accessibility requirements.

Usability should include accessibility when possible, but there is no must here. There are lots of tools that need usability testing that have a sufficiently small team and user-base to make accessibility a waste of time and resources.

Indeed, on teams and products of any size there is always a decision to be made about which accessibility requirements will be met and to what degree.

I would make the separate argument that what you need with accessibility requirements is separate testing expirements for each one. You don't need 100 testers to to make sure you have some special accessibility requirements, e.g. you need 15 normal vision testers and 15 impared vision testers.

Re: You Only Need to Test with 5 Users (2000)

#32

Post by awillen »

This is one of those things that takes a very abstract concept and tries to boil it down a bit too far with a mathematical model. Also, the 5 number in the headline is just misleading, since he clearly points out that the real number is 15.

The reality is that the number of people you need to test with to get the right number of insights (along with the depth of testing for any given user) is going to vary drastically across products of varying purposes and level of complexity. 5/15 users may be a reasonable average, but this is a case where an average of many different things isn't a particularly useful measure for any one of those things.

That doesn't even take into account the quality of people that you're testing with. Five experienced testers is different than five people with domain expertise but not testing experience is different than five people off the street.

Re: You Only Need to Test with 5 Users (2000)

#33

Post by HeWhoLurksLate »

noir_lord wrote: post #20
an_opabinia wrote: post #2

This is an excellent point, and the much more fascinating corollaries: - a product designer/manager of something with 1,000,000 users won’t learn more about usability than a product designer/manager of something with 15 users. All those measurements of flows and secret at scale analytics data is sort of worthless for the purposes of usability. - people with 15 users worth of learning about usability instead of 0 user…

It might even be worse still since this model (seems I may be wrong) to assume that the probability of a usability bug is constant, it might be that the share of bugs discovered by users is skewed towards the first few such that the first user finds more than the formula would predict. It's certainly been my observation that cynical developers who test things as they go by deliberately putting in silly things into st…

One of my friends was literally hired as an intern to try to break software last year. He loved it, and found a ton of bugs, which was really helpful to the company- they eventually paid him a $1,000 bonus for his help over the summer.

Re: You Only Need to Test with 5 Users (2000)

#34

Post by HeWhoLurksLate »

skybrian wrote: post #3
an_opabinia wrote: post #2

This is an excellent point, and the much more fascinating corollaries: - a product designer/manager of something with 1,000,000 users won’t learn more about usability than a product designer/manager of something with 15 users. All those measurements of flows and secret at scale analytics data is sort of worthless for the purposes of usability. - people with 15 users worth of learning about usability instead of 0 user…

This may be true for software used when working alone, but for software used by groups, you are not going to figure out group effects without testing with different group sizes. Consider the differences testing a game with 1, 2, 5, 100, or 1000 users.

So fifteen groups, then?

Re: You Only Need to Test with 5 Users (2000)

#35

Post by anbop »

Also... when you have 15 users (and it’s slow growing) it’s because you satisfy a unique need. These users are actually willing to talk to you for HOURS because they need your product, they know it’s niche, and that their feedback can actually affect the product development. Speaking from experience, I had a customer fly to ME to give me feedback.

Re: You Only Need to Test with 5 Users (2000)

#36

Post by cwyers »

nightcracker wrote: post #26
baxtr wrote: post #16

Earlier quoted context omitted.

Re 1st point: OP was referring specifically to usability learnings

I'd argue that usability also must include accessibility and I highly doubt that in your 15 user sample set you will cover the deaf, the (color) blind, the dyslexic, the paralyzed, the epileptic and various other groups of people with special accessibility requirements.

Or, for an American developer, what are the odds that your 15-user sample will find usability problems having to do with your icons only making sense to someone in the US? There's more to localization than just translation. Or if you have a product that does voice recognition, what range of accents are you covering with those 15 users?

Re: You Only Need to Test with 5 Users (2000)

#37

Post by zild3d »

awillen wrote: post #32

This is one of those things that takes a very abstract concept and tries to boil it down a bit too far with a mathematical model. Also, the 5 number in the headline is just misleading, since he clearly points out that the real number is 15. The reality is that the number of people you need to test with to get the right number of insights (along with the depth of testing for any given user) is going to vary drasticall…

> Also, the 5 number in the headline is just misleading, since he clearly points out that the real number is 15.

The 5 number isn't misleading at all, the author shows that it's the 80/20 point. The first 5 users in your usability test give you 85% of the value of testing with 15. The takeaway is to not do the same exact test with 15 users, do an iteration with 5, and rinse and repeat.

"Let us say that you do have the funding to recruit 15 representative customers and have them test your design. Great. Spend this budget on 3 studies with 5 users each!"

Re: You Only Need to Test with 5 Users (2000)

#38

Post by wwweston »

iamaelephant wrote: post #18
miki123211 wrote: post #8

I don't have much to do with UX, but I will add one insight. Try finding users with esoteric ways of working, and ensure the site works for them. By esoteric I mean blind users of screen readers, users of old, non js browsers, people on corp or school networks where things may be blocked, people on extremely small screens or extremely large ones (TVs with remotes), people who don't own or don't want to use a credit c…

This is going to depend heavily on your target market. In many of the SaaS applications I have been involved in we really don't care about users with non-JS browsers, or extremely small screens, or TVs, or people without credit cards. Some of the applications I have worked on will never be translated.

Two observations:

1) A site that isn't ready for a screen reader probably isn't ready for a voice browser or other non-visual user agent. Say, an AI/digital assistant. Or perhaps even a search engine indexing bot (though for the last 10-15 years the incentives behind this often mean people will invest heavily, even non-cooperatively, in this specifically).

2) Projects for which the engineering takes into account the possibility of serving content to non-JS or weird screens UAs from the get-go are often better technically. Not a statistically investigated theory, but my theory is that it's because if you're really doing the REST/resource-oriented thinking necessary to make a non-trivial app display responses as minimal markup, you're also doing the thinking necessary to make a good API to be consumed by a dynamic or even SPA front-end. And vice versa: if you've got a good API for a dynamic-heavy or SPA front-end, it's not difficult to represent the given resource with the media type HTML instead of JSON. Which means if it is difficult to represent a resource as HTML instead of JSON, something's probably not right with how your app is put together.

I wouldn't go so far as to say every app needs to be plain-HTML + accessibility focused, but I think there's benefits that go beyond the margins of users with direct accessibility issues.

Re: You Only Need to Test with 5 Users (2000)

#39

Post by noir_lord »

HeWhoLurksLate wrote: post #33
noir_lord wrote: post #20

Earlier quoted context omitted.

It might even be worse still since this model (seems I may be wrong) to assume that the probability of a usability bug is constant, it might be that the share of bugs discovered by users is skewed towards the first few such that the first user finds more than the formula would predict. It's certainly been my observation that cynical developers who test things as they go by deliberately putting in silly things into st…

One of my friends was literally hired as an intern to try to break software last year. He loved it, and found a ton of bugs, which was really helpful to the company- they eventually paid him a $1,000 bonus for his help over the summer.

I can believe it, the best person to test software for low hanging fruit of the "What happens if I do this thing that no sane person who knows anything about what the software is supposed to do would do?".

It's one of the reasons why I don't trust myself to test things fully, we write the software with all sorts of assumptions in our heads and subconsciously steer away from doing silly things - in that context it's really difficult to aim at a point a zero-knowledge user would hit.

Re: You Only Need to Test with 5 Users (2000)

#40

Post by imhoguy »

gdcohen wrote: post #28

There is a trend towards not testing at all! Instead builds are deployed straight to production or canary (mini-subset of production) and then very carefully and closely monitored. If a problem is uncovered, a rollback is performed. If canary is done well, then the problem can be caught before it has widespread impact.

By "not testing" you mean automated and no manual testing. Still somebody needs to manually test some cases to write them down next as a code to cover constantly changing product, but even the best automation won't resolve unknown unknowns. Canary is only a way to reduce one deployment impact.
Post reply on HN