QA = Time and Money. How much should you invest?
blog.rainforestqa.com
QA = Time and Money. How much should you invest?
1–10 of 57 posts
Re: QA = Time and Money. How much should you invest?
#2thoughtful article
Re: QA = Time and Money. How much should you invest?
#3As a consultant, I used to ask my clients to QA my deliveries. Few of them were willing to, to the extent they did they could not or would not provide meaningful bug reports.
This led to my adoption of test-driven development - mostly but not all automated testing - with the eventual result that I advertised that my final deliverables would be ready to ship to end-users, without any requirement for QA by my clients.
Re: QA = Time and Money. How much should you invest?
#4As a consultant, I used to ask my clients to QA my deliveries. Few of them were willing to, to the extent they did they could not or would not provide meaningful bug reports. This led to my adoption of test-driven development - mostly but not all automated testing - with the eventual result that I advertised that my final deliverables would be ready to ship to end-users, without any requirement for QA by my clients.
Test driven development usually refers to unit testing or integration testing. It would be interesting to know if you went beyond that and the nature of contract work you were doing (testing for an internal service with a RESTful API being very different from a mobile app, or a website).
Re: QA = Time and Money. How much should you invest?
#5As a consultant, I used to ask my clients to QA my deliveries. Few of them were willing to, to the extent they did they could not or would not provide meaningful bug reports. This led to my adoption of test-driven development - mostly but not all automated testing - with the eventual result that I advertised that my final deliverables would be ready to ship to end-users, without any requirement for QA by my clients.
TDD is great. It should be a given on any project that isn't just a spike.
I'm curious what kind of products you were building? Simple websites, or full fledged applications?
100% unit test coverage can still miss a host of gnarly bugs. A lot of the ugliness of QA comes from how A interacts with B. Or how A + B + C + D work together. Or at least, that's my 2 cents :D
Re: QA = Time and Money. How much should you invest?
#6This is probably the most honest article I've seen, ever, regarding how much QA/testing companies at various stages need.
I'm impressed, because there's a real tendency within QA culture to say "the ideal is all the things all the time, we're the gatekeepers," which has forced quality into a bad all-or-nothing situation. We all know intuitively that it's not all needed a lot of the time and the success stories are clear, so the trust in any of it is undermined. So then you get a dev culture saying "QA is unnecessary, we can handle all of it," which is generally true at the beginning but can send your company off a cliff if you don't manage transition.
The sad part is that even once quality processes get around that, the all-or-nothing attitude leads to a pyrrhic victory. Quality teams still end up only doing what can be afforded (because math) but everyone thinks they should be doing more. That leads to the pervasive opinion of ineffective quality teams. And, unfortunately, the scramble of trying to do everything at once (and therefore mastering none of it) often makes that a self-fulfilling prophecy.
And I love that it stresses that QA is about hedging bets. Bugs get out, period. The gatekeeper attitude is what leads to the do all the things attitude in the first place. "At any cost" is generally a bad way to strategize.
As for this chart, I'd have added exploratory/informal to seed B2C--exploratory is biggest bang for buck for finding new bugs, and incubating products are -all- new bugs--but I suspect they're lumping what I'd have recommended under dogfooding.
So yeah, very nice. Of course, the best thing to do in any situation is consider the context of your company, product, and market. Think about how maintainable it really needs to be and what would cause the most damage: losing customer money, leaking their data, embarrassing you in the market, eroding their trust. Are there only a few customers to lose or are you mass market and can afford a round of sufficiently obscure failure?
Those are the things to prioritize when you start picking your battles. But this is such a great set of guidelines for where to start that conversation.
(Edits for typos only)
Re: QA = Time and Money. How much should you invest?
#7This article, as is typical around here, addresses a very small part of the industry. Here's a view from a different part; the part where a project has a lifecycle of a decade or so.
I used to work for a software consultancy that had been acquired by a US defence company (second tier). Over the previous 15 years, the software consultancy had elevated their QA to the point that bugs in the delivered product were literally unknown. Nobody ever saw any in the delivered product, over a decade. Every requirement could be traced to the design that would meet it, to the code that implemented it, to the test that verified it. The test documentation was beautiful, and anyone could (and did) execute a complete set of tests, writing their own personal signature on every confirmation of test success. The signed test specs were scanned before being sealed in an envelope against the customer ever choosing to inspect. QA had (and did) veto over requirements, design, and tests.
We were working with two other major, globally known defence names, and the customer had learned through experience that when something went wrong, come to us last. Sometimes, the customer came to us to ask us to work out which of the other companies involved had screwed up (because sometimes, the other companies were themselves unable to work it out). Once, they sued one of the other names for incompetency and gave us their work because we knew how to fix it better than they did, because our interface tests highlighted everything that their software didn't do in accordance with the spec.
We could replay any test, at any version, at any time, and compare the results now with the signed results then from the sealed envelope. The time savings we had because we got things right the first time, by doing it properly, was immense.
Where this is going is that this was priceless. We had the customer's absolute trust, based on experience. Nobody ever found a bug in our delivered software (I'm not saying there weren't any; I'm saying that our QA was thorough enough that the customer was never going to do anything with it that we hadn't tested already). Code coverage, all warnings on, static analysis, valgrind, cross-platform compilation and testing, we did everything. Customers outright told us that we were worth paying extra because of the peace of mind and the reduced risk, so they paid us more and we spent less doing it.
QA is time and money. The better your QA, the more time you will have and the more money you will make.
Re: QA = Time and Money. How much should you invest?
#8This is probably the most honest article I've seen, ever, regarding how much QA/testing companies at various stages need. I'm impressed, because there's a real tendency within QA culture to say "the ideal is all the things all the time, we're the gatekeepers," which has forced quality into a bad all-or-nothing situation. We all know intuitively that it's not all needed a lot of the time and the success stories are cl…
Thanks for the good word (I'm the author). I 100% agree about setting the right culture / the difficulty of managing the transition.
It takes a ton of work to establish the right infrastructure + culture. Specifically for the reason you mentioned: a lot of folks see QA in black and white.
It's doubly hard when you're growing fast.
Re: QA = Time and Money. How much should you invest?
#9This article, as is typical around here, addresses a very small part of the industry. Here's a view from a different part; the part where a project has a lifecycle of a decade or so. I used to work for a software consultancy that had been acquired by a US defence company (second tier). Over the previous 15 years, the software consultancy had elevated their QA to the point that bugs in the delivered product were liter…
So, this I agree with too, in contrast with my other comment. There are particular industries (engineering, banking, aerospace, defense, for some examples) where it -has- to be correct. These are also the industries where at least the final product should probably be waterfall or spiral, and generally you need formalism all the way down.
The problem comes when people bring lessons learned from that to companies that don't need that level of assurance. Documentation that's never read again is a waste. Spending time to make tests replayable if they'll never be replayed again is a waste. And so forth.
As for bugs being literally unknown, I'm jealous--it'd be nice to claim I ever guided a project to that state. But keep in mind that's different than no bugs period, just means no bugs in what was tested or used.
If you're delivering a product with very specific use cases, flows and a tight scope with strong bumpers around anything unintended, it's pretty straightforward to guarantee all that. But if you're delivering a sandbox product on the internet to a bunch of randoms, much less so.
After all, ten checkboxes on a dialog are 1024 testcases. That's not even accounting for free-entry fields. You'll never cover all combinations of all uses of the whole app interface, ever, period, and equivalences and fuzz testing only get you so far. That's a simple (if slightly fantastic) example for UI testing, but the same basic thing applies to most other kinds of testing. At the end of the day you often have to pick your battles.
So I really disagree on the better is more. There is absolutely a diminishing return. It just so happens that for what -you- were doing, that point of diminishing return is pretty high.
Re: QA = Time and Money. How much should you invest?
#10This article, as is typical around here, addresses a very small part of the industry. Here's a view from a different part; the part where a project has a lifecycle of a decade or so. I used to work for a software consultancy that had been acquired by a US defence company (second tier). Over the previous 15 years, the software consultancy had elevated their QA to the point that bugs in the delivered product were liter…
> QA is time and money. The better your QA, the more time you will have and the more money you will make.
Just to caveat again that this is for your part of the industry. That level of detail gets you few benefits if you were Facebook, for example, or a very small startup still doing product validation.
Also to say that "the better your QA the more money you'll make" is a very strong position to take, even in your industry.