Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You Only Need to Test with 5 Users (2000)

nngroup.com

Post reply on HN
Print view View original

91–100 of 112 posts

Re: You Only Need to Test with 5 Users (2000)

#91

Post by bluGill »

adrianratnapala wrote: post #60
HeWhoLurksLate wrote: post #33

Earlier quoted context omitted.

One of my friends was literally hired as an intern to try to break software last year. He loved it, and found a ton of bugs, which was really helpful to the company- they eventually paid him a $1,000 bonus for his help over the summer.

It sounds like this guys employer has taken first step to inventing QA. There's whole classes of highly paid engineers whose job is to do this. But they work for old fashioned, boring, companies.

The problem is QA tends to get to scripted. You need to exercise each corner of the product, thus you test A, B, C in that order - so you never find cases where testing C, A, B breaks, or any other permutation. (to be fair with any complexity it is impossible to test all permutations)

Re: You Only Need to Test with 5 Users (2000)

#92

Post by bluGill »

an_opabinia wrote: post #2

This is an excellent point, and the much more fascinating corollaries: - a product designer/manager of something with 1,000,000 users won’t learn more about usability than a product designer/manager of something with 15 users. All those measurements of flows and secret at scale analytics data is sort of worthless for the purposes of usability. - people with 15 users worth of learning about usability instead of 0 user…

If you have 1,000,000 users you need to ask very different questions, so you better learn different things.

At 15 users your training budget is to teach everybody how to use your mess of a UI is cheaper than the cost of fixing the mess of a UI. At 1,000,000 users your cost of training is higher, but you also have more users to spread the cost of preparing the training among.

Re: You Only Need to Test with 5 Users (2000)

#93

Post by StavrosK »

noir_lord wrote: post #39
HeWhoLurksLate wrote: post #33

Earlier quoted context omitted.

One of my friends was literally hired as an intern to try to break software last year. He loved it, and found a ton of bugs, which was really helpful to the company- they eventually paid him a $1,000 bonus for his help over the summer.

I can believe it, the best person to test software for low hanging fruit of the "What happens if I do this thing that no sane person who knows anything about what the software is supposed to do would do?". It's one of the reasons why I don't trust myself to test things fully, we write the software with all sorts of assumptions in our heads and subconsciously steer away from doing silly things - in that context it's r…

I remember I implemented 2FA for a previous company, and our QA person managed to lock himself out of the account while enabling it. I asked him how that's possible, and he said "well I went to enable 2FA, it gave me the recovery codes, then it asked for a 2FA code so I entered one of the recovery ones, but now I ran out".

I thought this was so dumb it was brilliant QA.

Re: You Only Need to Test with 5 Users (2000)

#94

Post by Jeff_Brown »

This takes into consideration the benefits of more users, but not the cost. If the cost of adding one more test user is sufficiently small, it can be optimal to test with lots of them.

(Roughly, the economic rule here is that you keep doing something until the marginal cost is greater than the marginal benefit.)

Re: You Only Need to Test with 5 Users (2000)

#95

Post by saagarjha »

adrianN wrote: post #89
saagarjha wrote: post #80

Earlier quoted context omitted.

> If our JavaScript doesn't work in your browser, you are a security risk and we don't want our site to work on your browser. Sorry, what? Why am I security risk for not wanting to run the arbitrary code that your website sends me? (I browse with JavaScript turned on, FWIW, so this is a hypothetical question for me.)

Some javascript is used to detect bots.

Client-side JavaScript to detect bots seems doomed to fail, unless it's some sort of proof-of-work kind of thing.

Re: You Only Need to Test with 5 Users (2000)

#96

Post by adrianN »

saagarjha wrote: post #95
adrianN wrote: post #89

Earlier quoted context omitted.

Some javascript is used to detect bots.

Client-side JavaScript to detect bots seems doomed to fail, unless it's some sort of proof-of-work kind of thing.

All bot detection is doomed to fail, but you can make it more expensive for the bot authors.

Re: You Only Need to Test with 5 Users (2000)

#98

Post by lucb1e »

bravura wrote: post #49
Kluny wrote: post #27

Ha, as a web designer I long for the ability to test with even one user before launch day. Rarely do I get room in the budget for that :(

You don't have $10? Come on.

I think it's rather about their own time. If I waste a day that could have been customer work, my company misses out on well over a thousand bucks.

Re: You Only Need to Test with 5 Users (2000)

#99

Post by ebiester »

bluGill wrote: post #91
adrianratnapala wrote: post #60

Earlier quoted context omitted.

It sounds like this guys employer has taken first step to inventing QA. There's whole classes of highly paid engineers whose job is to do this. But they work for old fashioned, boring, companies.

The problem is QA tends to get to scripted. You need to exercise each corner of the product, thus you test A, B, C in that order - so you never find cases where testing C, A, B breaks, or any other permutation. (to be fair with any complexity it is impossible to test all permutations)

We call it exploratory testing.

Re: You Only Need to Test with 5 Users (2000)

#100

Post by nojvek »

an_opabinia wrote: post #2

This is an excellent point, and the much more fascinating corollaries: - a product designer/manager of something with 1,000,000 users won’t learn more about usability than a product designer/manager of something with 15 users. All those measurements of flows and secret at scale analytics data is sort of worthless for the purposes of usability. - people with 15 users worth of learning about usability instead of 0 user…

It’s not exclusive, usability is a duality. At the micro and macro levels. At the micro level you want to watch a couple of individual users and watch whether every little thing added to the UI makes sense to them. Whether the product solves their problem.

If micro fails, macro will most likely fail.

At the macro level you’re tracking higher level metrics like how many users engage with the different parts of the UI, where the fallout is? What is the retention of users, what do power users do differently to new users e.t.c

This kind of wholistic view about the product at micro and macro levels gives you a much better understanding about the bottle necks in the product.

Post reply on HN