A friend who worked at google told me once they would invariably get better insight on usability by just following a few individual users around a task vs. analyzing their zillions and zillions of site visits.
You Only Need to Test with 5 Users (2000)
21–30 of 112 posts
Re: You Only Need to Test with 5 Users (2000)
#22Earlier quoted context omitted.
Seems like for large effects, this depends on how often the effect occurs. If it's something that happens for everyone, you only need one user.
Obviously
If the same exact thing happens to every user, you don't need sampling. You can't possibly have a relevant bias while choosing
Re: You Only Need to Test with 5 Users (2000)
#23Ok, but how are you going to test for accessibility?
By bringing in people with accessibility needs. Few ways:
1. Large cities have accessibility Meetups, go to one and strike up a conversation and offer to pay someone to user test your software while you watch.
2. Hire a company/contractor that specializes in accesabilty audits.
3. Hire someone with an accessibility need. They will be unable to do their job till you fix your accesibilty problems.
Re: You Only Need to Test with 5 Users (2000)
#24Anybody who studied a bit of probability theory knows that this is both false and wrong. Statistically significant sampling can't be done with only five users. Confidence will only be reasonably sufficient for a population of 5 (at most). This is like if you addressed your product to 5 people (I don't believe it to be likely, you probably have way more than 5 users). See this page for assistance with computing sample…
What you're missing is covered in the article: the purpose of testing is to improve the product, not accurately document all the problems with one iteration of it. Better to test with 3 sessions of 5 users, than 1 session of 15.
Re: You Only Need to Test with 5 Users (2000)
#253 users has a surprisingly decent return too. Test something for crying out loud, a minimal investment provides a great deal of insight.
Re: You Only Need to Test with 5 Users (2000)
#26Earlier quoted context omitted.
+1 but several of your points are incorrect or need caveats. Your 1st point isn't correct, in that you will learn interesting things from 1m users that you won't from 15. The thing is that the 15 will tell you why, whereas the 1m won't unless you ask precisely the right question (which is a problem in itself). It basically takes experience here... (And this is something an AI may eventually become very good at.) Your…
Re 1st point: OP was referring specifically to usability learnings
I'd argue that usability also must include accessibility and I highly doubt that in your 15 user sample set you will cover the deaf, the (color) blind, the dyslexic, the paralyzed, the epileptic and various other groups of people with special accessibility requirements.
Re: You Only Need to Test with 5 Users (2000)
#27Ha, as a web designer I long for the ability to test with even one user before launch day. Rarely do I get room in the budget for that :(
Re: You Only Need to Test with 5 Users (2000)
#28There is a trend towards not testing at all! Instead builds are deployed straight to production or canary (mini-subset of production) and then very carefully and closely monitored. If a problem is uncovered, a rollback is performed. If canary is done well, then the problem can be caught before it has widespread impact.
Re: You Only Need to Test with 5 Users (2000)
#29I did some work for a major telco and we weren't allowed to do user testing outside of the company courtyard.
It consistently blew my mind how "out there" the feedback was with about a half-a-day-worth of testing.
So I don't know about 5-users, but I can say having about 25-50 is good for getting a broad sample.
If you get the wrong 5 users... you're going to get some really skewed results. For example, if you grab a group of 5 people and all of them are in tech your results are going to be dramatically different than if they are 5 people from accounting, or 5 people from janitorial services.
Re: You Only Need to Test with 5 Users (2000)
#30Ha, as a web designer I long for the ability to test with even one user before launch day. Rarely do I get room in the budget for that :(
[deleted]