Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

QA = Time and Money. How much should you invest?

blog.rainforestqa.com

Post reply on HN
Print view View original

41–50 of 57 posts

Re: QA = Time and Money. How much should you invest?

#41

Post by zachrose »

ThatMightBePaul wrote: post #32
calinet6 wrote: post #24

QA is interesting, sure. I'm more interested in the Q part, and you don't get that just by having a role or department checking work before it hits the customer. This article does a great job of describing the minimum level of quality allowable to scrape past certain stages of growth. The question you should be asking is: what level of quality, built in from the beginning, intentionally and with full support, will cr…

You mentioned 'level of quality'. That can be pretty difficult to quantify. In the absence of well defined metrics, a lot of thinking about QA is qualitative. "I feel like we have a high level of quality here". That makes conversations about QA pretty challenging. Metrics is the one area I'd LOVE to see move forward.

Heh, maybe that's because the nature of quality is qualitative rather than quantitative.

There are things that can be measured that might be correlated to quality ("bugs", sales, support, etc) but ultimately, classical quality seems to be to be a marriage of what was expected or desired with what was actually there. As a developer, my take is that quality usually stems from "developing" those expectations as much as making the relatively concrete thing to compare them too. When anyone can quantify the first half of this equation, please let me know at once.

Re: QA = Time and Money. How much should you invest?

#42

Post by calinet6 »

ThatMightBePaul wrote: post #32
calinet6 wrote: post #24

QA is interesting, sure. I'm more interested in the Q part, and you don't get that just by having a role or department checking work before it hits the customer. This article does a great job of describing the minimum level of quality allowable to scrape past certain stages of growth. The question you should be asking is: what level of quality, built in from the beginning, intentionally and with full support, will cr…

You mentioned 'level of quality'. That can be pretty difficult to quantify. In the absence of well defined metrics, a lot of thinking about QA is qualitative. "I feel like we have a high level of quality here". That makes conversations about QA pretty challenging. Metrics is the one area I'd LOVE to see move forward.

That's a really good question, and it requires some more context.

First, you have to understand your company from a systemic viewpoint. Break it down, and understand the inputs and outputs of every part of your company. You've got your individuals, your various teams, the people they work with, the systems they build, and the product itself. You can measure the input and output of each of these systems in meaningful ways: number of tickets opened, number of defects reported, raw performance metrics, uptime, availability, quantitative user feedback like NPS, and engagement metrics like product usage or feature usage.

But there are caveats: you can't turn these metrics into goals. You need to lead toward the improvement of the systems responsible for those metrics, with the hypothesis of improving them and ultimately the end goal of providing a high quality product of value to your customers. The metrics are valuable information about how the system operates, not the end goal.

There's another problem: you're absolutely right about quality being fundamentally qualitative. It is, after all, the root of the word "qualitative." That's significant. Quality is the humanity of the engineering game: while you can measure a lot of things toward it, ultimately, as W. Edwards Deming said, "The most important things cannot be measured." It sucks, but it is closer to reality than defining any metric or set of metrics. Instead, it requires leadership and a comfort with doubt and complexity. Here's my answer to a Quora question expressing discomfort with this paradox: http://qr.ae/f5iUg

So it's a bit of a Bermuda triangle. First, there are significant metrics you can measure, and you need to decide which ones mean quality to you and your customers—I promise they exist. Second, you can't aim directly for your metrics, since any large enough organization is highly complex and over-optimizing for a single metric or set of metrics can cause significant unintended side-effects. Third, the most important things critical to quality actually can't be measured; or measuring them would cause more harm than good. This makes for a challenge, and it's why this is such a difficult problem to solve in an organization.

What I'd LOVE to see moving forward is instead a profound comfort with complexity, and an understanding of why numeric metrics aren't always the most vital ingredient to success.

Re: QA = Time and Money. How much should you invest?

#43

Post by geoelectric »

School-Cotton wrote: post #35
geoelectric wrote: post #26

Earlier quoted context omitted.

What you describe is QA. Testing is technically QC: Quality Control. It's only one facet of QA, though the terms are usually conflated. Most companies only have a QC department.

You're proposing a definition that is out of line with how 99.99% of people use it. Even if your definition is more logical, this isn't how language works.

I'm not proposing anything. This is a well-known concept among quality professionals. I'm going to guess you have a different specialty.

Google "quality assurance control" and you'll find a lot more, but here's one source:

http://whatis.techtarget.com/definition/quality-control-QC

Re: QA = Time and Money. How much should you invest?

#44

Post by calinet6 »

geoelectric wrote: post #9
EliRivers wrote: post #7

This article, as is typical around here, addresses a very small part of the industry. Here's a view from a different part; the part where a project has a lifecycle of a decade or so. I used to work for a software consultancy that had been acquired by a US defence company (second tier). Over the previous 15 years, the software consultancy had elevated their QA to the point that bugs in the delivered product were liter…

So, this I agree with too, in contrast with my other comment. There are particular industries (engineering, banking, aerospace, defense, for some examples) where it -has- to be correct. These are also the industries where at least the final product should probably be waterfall or spiral, and generally you need formalism all the way down. The problem comes when people bring lessons learned from that to companies that…

Quality is defined in terms of customer expectations. Certain customers demand more, or are willing to pay more, for higher quality; certain ones aren't.

However, I think most consumer or enterprise software companies greatly undervalue quality and its impact on their competitive value proposition, and especially its impact on development efficiency and speed.

Even if the customer isn't willing to pay for higher quality, your developers are still paying for it with their time and ability to innovate; and your competitors will happily eat your lunch while you frantically hire "better developers" and use up all your time putting out fires.

There is a diminishing return on level of quality and level of QA done; you need to do what's appropriate for your market and your customer expectations. I just don't believe that most companies are anywhere near the appropriate level—not just of after-the-fact testing and best practices, but also in up-front built-in quality, design investment, and debt management.

Re: QA = Time and Money. How much should you invest?

#45

Post by MichaelCrawford »

ThatMightBePaul wrote: post #5
MichaelCrawford wrote: post #3

As a consultant, I used to ask my clients to QA my deliveries. Few of them were willing to, to the extent they did they could not or would not provide meaningful bug reports. This led to my adoption of test-driven development - mostly but not all automated testing - with the eventual result that I advertised that my final deliverables would be ready to ship to end-users, without any requirement for QA by my clients.

TDD is great. It should be a given on any project that isn't just a spike. I'm curious what kind of products you were building? Simple websites, or full fledged applications? 100% unit test coverage can still miss a host of gnarly bugs. A lot of the ugliness of QA comes from how A interacts with B. Or how A + B + C + D work together. Or at least, that's my 2 cents :D

I've done a wide variety of projects - cross-platform GUI, a windows database kernel implemented as a dll - that is, not a server - embedded storage firmware, mac os x device drivers (kernel extensions).

For some but not all of my projects, failure was not an option. Imagine I lost a disk sector in my storage firmware.

Re: QA = Time and Money. How much should you invest?

#46

Post by geoelectric »

enraged_camel wrote: post #37
geoelectric wrote: post #9

Earlier quoted context omitted.

So, this I agree with too, in contrast with my other comment. There are particular industries (engineering, banking, aerospace, defense, for some examples) where it -has- to be correct. These are also the industries where at least the final product should probably be waterfall or spiral, and generally you need formalism all the way down. The problem comes when people bring lessons learned from that to companies that…

>>The problem comes when people bring lessons learned from that to companies that don't need that level of assurance. Documentation that's never read again is a waste. Spending time to make tests replayable if they'll never be replayed again is a waste. And so forth. To me, writing thorough documentation and solid tests from the beginning serves a very important purpose that is often overlooked: it establishes the ri…

After doing quality for 15 years after another 5 as an application developer, I'm prepared to say that it's an awfully expensive and ineffective way to try to set tone. All it mostly does it get the rest of the company to assume you've got quality all handled and start chucking stuff over the wall at you. It doesn't inspire them to a higher standard.

If QA groups spent half as much time actually testing or helping stabilize development processes as they did maintaining brittle docs they'd probably be considerably more effective. At the end of the day most verbose test docs should have been one set of canonical product docs and a list of inputs/strategies. Instead we repeat and fragment the same info across a bunch of disconnected test docs and then let them rot under the weight of maintenance.

The problem is everyone thinks they have a "one true way." We know that concept is bullshit in software dev and that approach should be tailored to the problem at hand. We understand how important YAGNI and SPOT is, and why sometimes insisting on a perfect architecture is a bad thing in the real world.

Why people think it's any different for QA is beyond me. The same principles largely apply.

Re: QA = Time and Money. How much should you invest?

#47

Post by geoelectric »

calinet6 wrote: post #44
geoelectric wrote: post #9

Earlier quoted context omitted.

So, this I agree with too, in contrast with my other comment. There are particular industries (engineering, banking, aerospace, defense, for some examples) where it -has- to be correct. These are also the industries where at least the final product should probably be waterfall or spiral, and generally you need formalism all the way down. The problem comes when people bring lessons learned from that to companies that…

Quality is defined in terms of customer expectations. Certain customers demand more, or are willing to pay more, for higher quality; certain ones aren't. However, I think most consumer or enterprise software companies greatly undervalue quality and its impact on their competitive value proposition, and especially its impact on development efficiency and speed. Even if the customer isn't willing to pay for higher qual…

I basically agree, but in many cases I know it's not so much a question of amount of effort as it is spending effort doing things that either sound "tried and true" or happen to be quantifiable, at the expense of the things that would actually make quality better.

It's another type of taking the easy way out in that a potentially more effective context-driven strategy takes more effort to justify and maintain faith up the chain. But not adopting a strategy specific to your context can easily result in spending a lot of effort to get only marginally better results than unit testing, dogfooding, and nothing else.

Re: QA = Time and Money. How much should you invest?

#48

Post by geoelectric »

EliRivers wrote: post #22
geoelectric wrote: post #9

Earlier quoted context omitted.

So, this I agree with too, in contrast with my other comment. There are particular industries (engineering, banking, aerospace, defense, for some examples) where it -has- to be correct. These are also the industries where at least the final product should probably be waterfall or spiral, and generally you need formalism all the way down. The problem comes when people bring lessons learned from that to companies that…

The problem comes when people bring lessons learned from that to companies that don't need that level of assurance. It goes the other way as well. Bringing agile methods to a situation in which having a requirement changed can take three months. Situations in which your "customer" is a faceless defence ministry (or two) with the level of responsiveness you'd expect. Processes are tools; use the right tools for the jo…

Yeah, totally agree. That's what I meant when I said certain industries should be waterfall. It's turned into a bad word, but waterfall or cyclic/spiral is tailor-made for situations where requirements are both known up front and necessarily rigid.

Re: QA = Time and Money. How much should you invest?

#49

Post by geoelectric »

ThatMightBePaul wrote: post #34
geoelectric wrote: post #13

Earlier quoted context omitted.

You also have to qualify 100% coverage: SUT lines, branches, or paths? Those are all very different levels of rigor. The two most useful code coverage stats I've found are 0 and "less than yesterday." The former tells me something pretty important about the culture of the team who owns that code; and having rules against the latter is about the only way to ensure a test-on-checkin policy.

I love this metric, and am totally stealing it.

All yours!

Re: QA = Time and Money. How much should you invest?

#50

Post by jacques_chester »

3am wrote: post #4
MichaelCrawford wrote: post #3

As a consultant, I used to ask my clients to QA my deliveries. Few of them were willing to, to the extent they did they could not or would not provide meaningful bug reports. This led to my adoption of test-driven development - mostly but not all automated testing - with the eventual result that I advertised that my final deliverables would be ready to ship to end-users, without any requirement for QA by my clients.

Test driven development usually refers to unit testing or integration testing. It would be interesting to know if you went beyond that and the nature of contract work you were doing (testing for an internal service with a RESTful API being very different from a mobile app, or a website).

TDD, in my view, starts from the outermost layer -- the end user -- and moves inwards progressively to the unit tests.
Post reply on HN