You Only Need to Test with 5 Users (2000) • Hacker News Board

Re: You Only Need to Test with 5 Users (2000)

#51

Post by dredmorbius » Mon, Jul 15, 2019, 1:23 AM UTC

skybrian wrote: post #3

an_opabinia wrote: post #2

This is an excellent point, and the much more fascinating corollaries: - a product designer/manager of something with 1,000,000 users won’t learn more about usability than a product designer/manager of something with 15 users. All those measurements of flows and secret at scale analytics data is sort of worthless for the purposes of usability. - people with 15 users worth of learning about usability instead of 0 user…

This may be true for software used when working alone, but for software used by groups, you are not going to figure out group effects without testing with different group sizes. Consider the differences testing a game with 1, 2, 5, 100, or 1000 users.

There's usability testing and there's scale testing.

Basic UI/UX will be significantly revealed be small-group testing.

Don't let the perfect be the enemy of the good.

Re: You Only Need to Test with 5 Users (2000)

#52

Post by dredmorbius » Mon, Jul 15, 2019, 1:42 AM UTC

baxtr wrote: post #15

Reminds me of the small “sample size misconception” https://www.dailykos.com/story/2008/11/9/656465/-

Very true, though this can cut several different ways.

I've done several measurements of various aspects of use and engagement on Google+, as an independent outsider.

From a randomised sample of fewer than 100 randomly selected profiles (of 2.2 billion total), it was clear that the active fraction was about 10%, and the highly active fraction a minuscule portion of that.

I ended up checking 50k profiles, and another (fully independent) analysis by Stone Temple Consulting of 500k profiles largely re-demonstrated the initial 9% finding. But, with more profiles, it was possible to dial in on the very few (about 0.16%) highly active users. Which is another sampling problem -- you're looking for the 1 in 1,000 users who are active, and need to get a sufficient sample (typically 30-300) of those. Let's call it 100, for round numbers.

Which means you're looking for a sample of a one-in-a-thousand subpopulation, meaning that 100 rare high-use users requires sampling 100 * 1000 = 100k of the total population.

(This presumes no other way of subsetting the population.)

I ran into this looking at G+ Communities -- 8 million in total, with again a very small fraction of highly-active ones. Initial samples of 12k and 36k subsets were useful (and tractable on a residential DSL connection), but it was being gifted with a full 8-million record population summary of community activity that allowed full and detailed statistics to be calculated.

https://social.antefriguserat.de/index.php/Migrating_Google%...

Re: You Only Need to Test with 5 Users (2000)

#53

Post by empath75 » Mon, Jul 15, 2019, 1:44 AM UTC

JasonFruit wrote: post #43

noir_lord wrote: post #20

Earlier quoted context omitted.

It might even be worse still since this model (seems I may be wrong) to assume that the probability of a usability bug is constant, it might be that the share of bugs discovered by users is skewed towards the first few such that the first user finds more than the formula would predict. It's certainly been my observation that cynical developers who test things as they go by deliberately putting in silly things into st…

I bombed a job interview by writing a piece of demo software that failed all of your "throw it garbage" tests. I couldn't have hired a professor to teach me a more useful lesson!

That seems like a brutally unfair interview practice unless you were told in advance they’d be doing that.

Re: You Only Need to Test with 5 Users (2000)

#54

Post by soup10 » Mon, Jul 15, 2019, 2:02 AM UTC

hammerbrostime wrote: post #25

3 users has a surprisingly decent return too. Test something for crying out loud, a minimal investment provides a great deal of insight.

most of what you need to know to fix usability issues will be discovered pretty quickly just by watching a single "regular" person use it.

Re: You Only Need to Test with 5 Users (2000)

#55

Post by JasonFruit » Mon, Jul 15, 2019, 2:02 AM UTC

empath75 wrote: post #53

JasonFruit wrote: post #43

Earlier quoted context omitted.

I bombed a job interview by writing a piece of demo software that failed all of your "throw it garbage" tests. I couldn't have hired a professor to teach me a more useful lesson!

That seems like a brutally unfair interview practice unless you were told in advance they’d be doing that.

They gave me a small project to do, and told me to do it as though I were building it for a customer. I think that was warning enough that it ought to gracefully handle bad input — any real-world program needs to do the same.

Re: You Only Need to Test with 5 Users (2000)

#56

Post by pkamb » Mon, Jul 15, 2019, 2:07 AM UTC

Kluny wrote: post #27

Ha, as a web designer I long for the ability to test with even one user before launch day. Rarely do I get room in the budget for that :(

Sit in Starbucks and offer $5 gift cards...

Re: You Only Need to Test with 5 Users (2000)

#57

Post by derp_dee_derp » Mon, Jul 15, 2019, 2:08 AM UTC

wwweston wrote: post #38

iamaelephant wrote: post #18

Earlier quoted context omitted.

This is going to depend heavily on your target market. In many of the SaaS applications I have been involved in we really don't care about users with non-JS browsers, or extremely small screens, or TVs, or people without credit cards. Some of the applications I have worked on will never be translated.

Two observations: 1) A site that isn't ready for a screen reader probably isn't ready for a voice browser or other non-visual user agent. Say, an AI/digital assistant. Or perhaps even a search engine indexing bot (though for the last 10-15 years the incentives behind this often mean people will invest heavily, even non-cooperatively, in this specifically). 2) Projects for which the engineering takes into account the…

Ux Engineering manager here for a fortune 100 company.

We don't care about any of the use cases in your comment. If our JavaScript doesn't work in your browser, you are a security risk and we don't want our site to work on your browser.

Please call our 1-800 number to talk to a representative.

Re: You Only Need to Test with 5 Users (2000)

#58

Post by itsmurat » Mon, Jul 15, 2019, 2:12 AM UTC

80/20 rule apply here you will get 80-90% problems with usability when you tested with 5 people today’s world though we need to have predictive analytics to increase this coverage and also Start creating customized journeys

Re: You Only Need to Test with 5 Users (2000)

#59

Post by wwweston » Mon, Jul 15, 2019, 2:55 AM UTC

derp_dee_derp wrote: post #57

wwweston wrote: post #38

Earlier quoted context omitted.

Two observations: 1) A site that isn't ready for a screen reader probably isn't ready for a voice browser or other non-visual user agent. Say, an AI/digital assistant. Or perhaps even a search engine indexing bot (though for the last 10-15 years the incentives behind this often mean people will invest heavily, even non-cooperatively, in this specifically). 2) Projects for which the engineering takes into account the…

Ux Engineering manager here for a fortune 100 company. We don't care about any of the use cases in your comment. If our JavaScript doesn't work in your browser, you are a security risk and we don't want our site to work on your browser. Please call our 1-800 number to talk to a representative.

> We don't care about any of the use cases in your comment

Hey, everybody has to make tradeoffs, and it's certainly conceivable that the margins of disability don't matter to you, or that your phone support is your non-visual UA, or even that you don't care about search.

That's only half the contents of my comment, though, the rest is about engineering benefits. Perhaps you don't care about those either?

> If our JavaScript doesn't work in your browser, you are a security risk

What? JavaScript fantastically useful, but it's a vulnerability surface, not a security feature.

Maybe you meant to say HTTPS/TLS? Because if you meant to use JS as a proxy for TLS support, oh, man, I have some bad news for you. Both about security and about the relative utility of engineering around proxy tests vs engineering around tests for specific feature support.

> Ux Engineering manager here for a fortune 100 company.

I'd love to know the name of the company. Don't worry, I won't try to get you in trouble or anything, it's just clear that there's opportunities to rise well above one's competence there.

Re: You Only Need to Test with 5 Users (2000)

#60

Post by adrianratnapala » Mon, Jul 15, 2019, 3:06 AM UTC

HeWhoLurksLate wrote: post #33

noir_lord wrote: post #20

Earlier quoted context omitted.

It might even be worse still since this model (seems I may be wrong) to assume that the probability of a usability bug is constant, it might be that the share of bugs discovered by users is skewed towards the first few such that the first user finds more than the formula would predict. It's certainly been my observation that cynical developers who test things as they go by deliberately putting in silly things into st…

One of my friends was literally hired as an intern to try to break software last year. He loved it, and found a ton of bugs, which was really helpful to the company- they eventually paid him a $1,000 bonus for his help over the summer.

It sounds like this guys employer has taken first step to inventing QA.

There's whole classes of highly paid engineers whose job is to do this. But they work for old fashioned, boring, companies.