Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You Only Need to Test with 5 Users (2000)

nngroup.com

Post reply on HN
Print view View original

41–50 of 112 posts

Re: You Only Need to Test with 5 Users (2000)

#41

Post by Izkata »

noir_lord wrote: post #39
HeWhoLurksLate wrote: post #33

Earlier quoted context omitted.

One of my friends was literally hired as an intern to try to break software last year. He loved it, and found a ton of bugs, which was really helpful to the company- they eventually paid him a $1,000 bonus for his help over the summer.

I can believe it, the best person to test software for low hanging fruit of the "What happens if I do this thing that no sane person who knows anything about what the software is supposed to do would do?". It's one of the reasons why I don't trust myself to test things fully, we write the software with all sorts of assumptions in our heads and subconsciously steer away from doing silly things - in that context it's r…

Even a sane person can find a lot if they never touched the implementation of a specific feature.

On one of my teams, we had a hard rule where all features must be tested by 2 other team members (and if multiple people worked on a feature, none of them can be the testers). Something like every other case found something questionable, if not an outright broken edge case, that the developer(s) completely missed.

Re: You Only Need to Test with 5 Users (2000)

#42

Post by EGreg »

I totally disagree this is true for most sites today.

Usability these days is not just about what a single user will do. We build multi user apps. So the following things are an emergent phenomenon of actions MANY people take:

Engagement

Retention

Viral Spread

Collaboration

Notifications

Real time updates

Chicken and Egg problems

To test these things, you often need tons of real or fake accounts. You get some people trying interest X, and others interest Y. Sometimes things with the exact same interface usability take off in one country and not another. Like Orkut in Brazil!

Sites sometimes arrive at breakthroughs by A/B testing many things automatically across millions of sessions. That’s far more efficient but requires a large enough sample.

In fact, most of the reasons famous sites are famous is because they successfully got a lot of people to keep coming back and doing something. They probably got them to invite friends. And so on.

Re: You Only Need to Test with 5 Users (2000)

#43

Post by JasonFruit »

noir_lord wrote: post #20
an_opabinia wrote: post #2

This is an excellent point, and the much more fascinating corollaries: - a product designer/manager of something with 1,000,000 users won’t learn more about usability than a product designer/manager of something with 15 users. All those measurements of flows and secret at scale analytics data is sort of worthless for the purposes of usability. - people with 15 users worth of learning about usability instead of 0 user…

It might even be worse still since this model (seems I may be wrong) to assume that the probability of a usability bug is constant, it might be that the share of bugs discovered by users is skewed towards the first few such that the first user finds more than the formula would predict. It's certainly been my observation that cynical developers who test things as they go by deliberately putting in silly things into st…

I bombed a job interview by writing a piece of demo software that failed all of your "throw it garbage" tests. I couldn't have hired a professor to teach me a more useful lesson!

Re: You Only Need to Test with 5 Users (2000)

#45

Post by QuinnWilton »

noir_lord wrote: post #39
HeWhoLurksLate wrote: post #33

Earlier quoted context omitted.

One of my friends was literally hired as an intern to try to break software last year. He loved it, and found a ton of bugs, which was really helpful to the company- they eventually paid him a $1,000 bonus for his help over the summer.

I can believe it, the best person to test software for low hanging fruit of the "What happens if I do this thing that no sane person who knows anything about what the software is supposed to do would do?". It's one of the reasons why I don't trust myself to test things fully, we write the software with all sorts of assumptions in our heads and subconsciously steer away from doing silly things - in that context it's r…

I'm been building an API Security Scanner, and one of the things it does is just fuzz every endpoint with garbage in each parameter to look for stacktraces, errors, etc.

Moreso than any of the security tests I've written, that fuzzing has broken every enterprise API our customers have thrown at it.

Re: You Only Need to Test with 5 Users (2000)

#46

Post by z3t4 »

I think the key here is to find five representative customers , not just five random people. These users can be hard to find unless you have good market fit. But if you have very good market fit, my experience is that users will overcome just about every hurdle. For example digging gold and other minerals, people will do a lot of work if it's valuable. But if it's not as valuable you might need to hand it to them on a silver plate.

Re: You Only Need to Test with 5 Users (2000)

#47

Post by mclehman »

HeWhoLurksLate wrote: post #33
noir_lord wrote: post #20

Earlier quoted context omitted.

It might even be worse still since this model (seems I may be wrong) to assume that the probability of a usability bug is constant, it might be that the share of bugs discovered by users is skewed towards the first few such that the first user finds more than the formula would predict. It's certainly been my observation that cynical developers who test things as they go by deliberately putting in silly things into st…

One of my friends was literally hired as an intern to try to break software last year. He loved it, and found a ton of bugs, which was really helpful to the company- they eventually paid him a $1,000 bonus for his help over the summer.

I have a friend who does this for free. Every single thing I've ever shown him (both my software and that of others) breaks as soon as he gets his hands on it.

He's the sort to dig as far into the internals of things as he can and then start messing with it (he implemented partial function application for C, for example [1]).

My poor prototypes never stood a chance.

[1] https://github.com/zwimer/C-bind

Re: You Only Need to Test with 5 Users (2000)

#48

Post by noir_lord »

JasonFruit wrote: post #43
noir_lord wrote: post #20

Earlier quoted context omitted.

It might even be worse still since this model (seems I may be wrong) to assume that the probability of a usability bug is constant, it might be that the share of bugs discovered by users is skewed towards the first few such that the first user finds more than the formula would predict. It's certainly been my observation that cynical developers who test things as they go by deliberately putting in silly things into st…

I bombed a job interview by writing a piece of demo software that failed all of your "throw it garbage" tests. I couldn't have hired a professor to teach me a more useful lesson!

Mine was brilliant, not a famous academic or anything and it was a college not a university but he'd written production systems in the 80's for telecomms and massive supermarkets so as an instructor for the real world really hard to beat - he taught me all sorts of things that stuck some of them I didn't understand at the time but 20 years later they make much more sense :).

Re: You Only Need to Test with 5 Users (2000)

#50

Post by perl4ever »

an_opabinia wrote: post #2

This is an excellent point, and the much more fascinating corollaries: - a product designer/manager of something with 1,000,000 users won’t learn more about usability than a product designer/manager of something with 15 users. All those measurements of flows and secret at scale analytics data is sort of worthless for the purposes of usability. - people with 15 users worth of learning about usability instead of 0 user…

My impression is that most people instinctively assume that to get accurate results from sampling, you need a sample size proportional to the population, and that isn't generally the case.

Maybe for many purposes, 5 won't cut it and you need 100 or 1,000, but you don't need 1%.

Post reply on HN