Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You Only Need to Test with 5 Users (2000)

nngroup.com

Post reply on HN
Print view View original

61–70 of 112 posts

Re: You Only Need to Test with 5 Users (2000)

#61

Post by adrianratnapala »

cwyers wrote: post #36
nightcracker wrote: post #26

Earlier quoted context omitted.

I'd argue that usability also must include accessibility and I highly doubt that in your 15 user sample set you will cover the deaf, the (color) blind, the dyslexic, the paralyzed, the epileptic and various other groups of people with special accessibility requirements.

Or, for an American developer, what are the odds that your 15-user sample will find usability problems having to do with your icons only making sense to someone in the US? There's more to localization than just translation. Or if you have a product that does voice recognition, what range of accents are you covering with those 15 users?

That's OK, we nowadays use icons that don't make sense to anyone at all. I mean how would you know that three horizontal lines was a "Hamburger menu", and if you do know that, why would you want a Hamburger?

Re: You Only Need to Test with 5 Users (2000)

#62

Post by [deleted] »

shkkmo wrote: post #31
nightcracker wrote: post #26

Earlier quoted context omitted.

I'd argue that usability also must include accessibility and I highly doubt that in your 15 user sample set you will cover the deaf, the (color) blind, the dyslexic, the paralyzed, the epileptic and various other groups of people with special accessibility requirements.

Usability should include accessibility when possible, but there is no must here. There are lots of tools that need usability testing that have a sufficiently small team and user-base to make accessibility a waste of time and resources. Indeed, on teams and products of any size there is always a decision to be made about which accessibility requirements will be met and to what degree. I would make the separate argumen…

[deleted]

Re: You Only Need to Test with 5 Users (2000)

#63

Post by achow »

This has been questioned (5 user theory).

From 2008 research paper: http://www.simplifyinginterfaces.com/wp-content/uploads/2008...

Excerpt:

Historic reason: Both Nielsen (1993) and Virzi (1992) were writing in a climate in which the concepts of usability were still being introduced..They were striving to lighten the requirements of usability testing in order to make usability practices more attractive to those working with the strained budgets.

Conclusion: It is advisable to run the maximum number of participants that schedules, budgets, and availability allow. The mathematical benefits of adding test users should be cited. More test users means greater confidence that the problems that need to be fixed will be found; as is shown in the analysis for this study, increasing the number from 5 to 10 can result in a dramatic improvement in data confidence. Increasing the number tested to 20 can allow the practitioner to approach increasing level

Re: You Only Need to Test with 5 Users (2000)

#64

Post by eddieh »

adrianratnapala wrote: post #61
cwyers wrote: post #36

Earlier quoted context omitted.

Or, for an American developer, what are the odds that your 15-user sample will find usability problems having to do with your icons only making sense to someone in the US? There's more to localization than just translation. Or if you have a product that does voice recognition, what range of accents are you covering with those 15 users?

That's OK, we nowadays use icons that don't make sense to anyone at all. I mean how would you know that three horizontal lines was a "Hamburger menu", and if you do know that, why would you want a Hamburger?

Actually the icon looks like three menu items. It is a pretty good icon for signaling that it will open a menu. The hamburger nomenclature seems to be an ex pos facto name for something that rightfully should be called the menu icon.

Re: You Only Need to Test with 5 Users (2000)

#65

Post by piyush_soni »

anbop wrote: post #35

Also... when you have 15 users (and it’s slow growing) it’s because you satisfy a unique need. These users are actually willing to talk to you for HOURS because they need your product, they know it’s niche, and that their feedback can actually affect the product development. Speaking from experience, I had a customer fly to ME to give me feedback.

Wow. Skype/Email didn't suffice? :)

Re: You Only Need to Test with 5 Users (2000)

#67

Post by mntmoss »

an_opabinia wrote: post #2

This is an excellent point, and the much more fascinating corollaries: - a product designer/manager of something with 1,000,000 users won’t learn more about usability than a product designer/manager of something with 15 users. All those measurements of flows and secret at scale analytics data is sort of worthless for the purposes of usability. - people with 15 users worth of learning about usability instead of 0 user…

I definitely think there is a learnable aspect to design. Throwing data at the problem is expensive and does not create many deep insights. But focusing on communication and how the product communicates a coherent philosophy through features and UX yields a more systematic, standalone framework to test hypotheses against, something which data can further shape. It's more important to find "must" and "cannot" boundary conditions than preferences, and it's also easier to see if you are satisfying those conditions.

My formula is therefore: Pick a few principles to apply to the solution you're making, make sure as best you can that they agree with each other. Test to see if potential users agree with those principles. Select features based on the resulting design framework. Then develop the product as a way of discovering more about this framework.

Lots of companies are doing this implicitly through their founding teams and hiring practices: they just end up with culture that values certain principles, so they get upheld at every meeting and the results end up in the product. But it can also be communicated and imposed more explicitly, and I think that's where design becomes more visible in the process.

Re: You Only Need to Test with 5 Users (2000)

#69

Post by andy_ppp »

I find it very irritating where I am that the design team don’t interact with the programmers concerns about the designs difficulties and instead just point to user research. A lot of design issues can be fixed by talking to not just your users but any human walking by. Hallway usability testing works well and just about anyone, even a programmer, will make your design better with a fresh pair of eyes.

Re: You Only Need to Test with 5 Users (2000)

#70

Post by CodiePetersen »

Wow great article and perfect timing for me. I'm about to go into testing phase myself and this is something I've never considered. I was worried because we're and indie group and I wasn't sure how we were going to get a lot of people but looks like smaller is good enough and even optimal.

Thanks.

Post reply on HN