Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

QA = Time and Money. How much should you invest?

blog.rainforestqa.com

Post reply on HN
Print view View original

31–40 of 57 posts

Re: QA = Time and Money. How much should you invest?

#31

Post by ThatMightBePaul »

infosectosser wrote: post #23
ThatMightBePaul wrote: post #14

Earlier quoted context omitted.

Agreed. Caveat: do you need to do pen testing every release? I'm guessing, not. YMMV. If you're a defense contractor (as some other commenters mentioned), your priorities are probably quite different.

I head infosec for a “Series A - C” B2B company and a fairly standard request from a potential customer is to see not only our own penetration test reports but third party penetration test results, as well. As a result, we run automated pen tests on weekends and before major releases. We also work with an application security firm every 6-12 months. For what it’s worth, we don’t do anything nearly as intense as defen…

Np! Thanks for sharing.

Glad to hear security is taking the front seat some places. Anecdotes like this help me expand my world view. <3

Re: QA = Time and Money. How much should you invest?

#32

Post by ThatMightBePaul »

calinet6 wrote: post #24

QA is interesting, sure. I'm more interested in the Q part, and you don't get that just by having a role or department checking work before it hits the customer. This article does a great job of describing the minimum level of quality allowable to scrape past certain stages of growth. The question you should be asking is: what level of quality, built in from the beginning, intentionally and with full support, will cr…

You mentioned 'level of quality'. That can be pretty difficult to quantify. In the absence of well defined metrics, a lot of thinking about QA is qualitative.

"I feel like we have a high level of quality here".

That makes conversations about QA pretty challenging. Metrics is the one area I'd LOVE to see move forward.

Re: QA = Time and Money. How much should you invest?

#33

Post by ThatMightBePaul »

geoelectric wrote: post #27
ffn wrote: post #12

Can someone please explain why one prefers unit testing to integration / acceptance testing when you're a seed stage company? I mean, your primary objective as an early stage company should be getting an operational product out to the market, so it makes sense that you test that your users can operate your product - aka integration / acceptance test with crapybara or phantomjs. On the other hand, if you're just unit…

Unit testing is easier to do with the developer resources on hand since they probably already know how to write them, probably necessary for quick pace in dev anyway, and gives you a large portion of the risk mitigation testing in general will give you. You also check in unit tests with the code if you do it right, whereas integration or acceptance tests require additional passes once all the code covered is checked…

This. Unit testing is better for a lot of reasons. Dev familiarity is definitely a big reason. I find they're a bit more robust, too.

Integration tests are more brittle. Which, can be good ("Hey, there's a bug here..."). For an early product, your app is probably simple enough that you get a good enough feel while dogfooding. 100% correct that also includes some informal exploratory work :D

RE: Complexity & the relative need for testing, I really like the perspective 'Out of the Tar Pits' takes. http://shaffner.us/cs/papers/tarpit.pdf

Re: QA = Time and Money. How much should you invest?

#34

Post by ThatMightBePaul »

geoelectric wrote: post #13
ThatMightBePaul wrote: post #5

Earlier quoted context omitted.

TDD is great. It should be a given on any project that isn't just a spike. I'm curious what kind of products you were building? Simple websites, or full fledged applications? 100% unit test coverage can still miss a host of gnarly bugs. A lot of the ugliness of QA comes from how A interacts with B. Or how A + B + C + D work together. Or at least, that's my 2 cents :D

You also have to qualify 100% coverage: SUT lines, branches, or paths? Those are all very different levels of rigor. The two most useful code coverage stats I've found are 0 and "less than yesterday." The former tells me something pretty important about the culture of the team who owns that code; and having rules against the latter is about the only way to ensure a test-on-checkin policy.

I love this metric, and am totally stealing it.

Re: QA = Time and Money. How much should you invest?

#35

Post by School-Cotton »

geoelectric wrote: post #26
calinet6 wrote: post #24

QA is interesting, sure. I'm more interested in the Q part, and you don't get that just by having a role or department checking work before it hits the customer. This article does a great job of describing the minimum level of quality allowable to scrape past certain stages of growth. The question you should be asking is: what level of quality, built in from the beginning, intentionally and with full support, will cr…

What you describe is QA. Testing is technically QC: Quality Control. It's only one facet of QA, though the terms are usually conflated. Most companies only have a QC department.

You're proposing a definition that is out of line with how 99.99% of people use it.

Even if your definition is more logical, this isn't how language works.

Re: QA = Time and Money. How much should you invest?

#36

Post by ThatMightBePaul »

EliRivers wrote: post #7

This article, as is typical around here, addresses a very small part of the industry. Here's a view from a different part; the part where a project has a lifecycle of a decade or so. I used to work for a software consultancy that had been acquired by a US defence company (second tier). Over the previous 15 years, the software consultancy had elevated their QA to the point that bugs in the delivered product were liter…

Woah! Thanks for sharing. Love hearing about the other side. All of the projects I've been involved in operate on either a continuous release, or a one to two week release cycle.

So, I'm definitely ignorant to the standards at spots with months-long release cycles.

Re: QA = Time and Money. How much should you invest?

#37

Post by enraged_camel »

geoelectric wrote: post #9
EliRivers wrote: post #7

This article, as is typical around here, addresses a very small part of the industry. Here's a view from a different part; the part where a project has a lifecycle of a decade or so. I used to work for a software consultancy that had been acquired by a US defence company (second tier). Over the previous 15 years, the software consultancy had elevated their QA to the point that bugs in the delivered product were liter…

So, this I agree with too, in contrast with my other comment. There are particular industries (engineering, banking, aerospace, defense, for some examples) where it -has- to be correct. These are also the industries where at least the final product should probably be waterfall or spiral, and generally you need formalism all the way down. The problem comes when people bring lessons learned from that to companies that…

>>The problem comes when people bring lessons learned from that to companies that don't need that level of assurance. Documentation that's never read again is a waste. Spending time to make tests replayable if they'll never be replayed again is a waste. And so forth.

To me, writing thorough documentation and solid tests from the beginning serves a very important purpose that is often overlooked: it establishes the right type of culture. I think of it as laying the foundation of a building. If you skimp out on materials, the whole thing will be unstable and prone to collapse. And going back later to fix it is often times a lot more costly than doing it right in the first place.

You would never make the argument that a solid foundation is unnecessary if an earthquake never hits, right?

Re: QA = Time and Money. How much should you invest?

#38

Post by ThatMightBePaul »

joshdotsmith wrote: post #16

Wow, crazy timing. We interviewed Fred at Rainforest on the Talking Code podcast – and that interview launches tonight at midnight (PDT). I'm going back through and picking out my favorite quotes right this moment in preparation for the episode launch. Some of my favorites: "How did we get to the place where the generally accepted best practice is not very good?" (referring to automated testing) "The visual represent…

Look forward to hearing this. Will Venmo $5 in exchange for the outtakes of Fred cursing ;)

Re: QA = Time and Money. How much should you invest?

#39

Post by pbiggar »

EliRivers wrote: post #17
pbiggar wrote: post #10

Earlier quoted context omitted.

> QA is time and money. The better your QA, the more time you will have and the more money you will make. Just to caveat again that this is for your part of the industry. That level of detail gets you few benefits if you were Facebook, for example, or a very small startup still doing product validation. Also to say that "the better your QA the more money you'll make" is a very strong position to take, even in your in…

Also to say that "the better your QA the more money you'll make" is a very strong position to take, even in your industry. It is a bit strong, isn't it? I felt that to caveat it might detract from the effectiveness of the point. There does come a point where the software is effectively flawless from the customer's point of view, and doing anything to improve it (including more/better QA) from there doesn't make any m…

More than that, better QA or better software doesn't actually make you money. As in, you can't make your software twice as good and make twice as much money. It's a lot more nuanced and much more indirect than that.

So certainly you can say "investing X QA hours per month should help reduce customer incidents by 50%, which will reduce churn rate by 30%, increasing MRR by 100% over 1 year, given current churn rates". Or whatever other metrics affect your actual revenue.

Overall, you need to draw a much stronger connection between QA and money, esp if you want to claim a continuous function between them.

Re: QA = Time and Money. How much should you invest?

#40

Post by ecopoesis »

Zero QA, or at least zero QA by dedicated QA people.

Developers end up using QA as a crutch. The end up not testing their code completely, and sometimes they don't test it at all. Too many times I've seen code checked in, after going through code review, that just doesn't work. I'm not talking edge cases, it does not function at all.

Get rid of the crutch, and force developers to be responsible for their code. Developers should do all the QA on their code.

If you must have QA, don't waste them on checking if features are I plmented correctly, repeating a job developers should do. Instead, have them actually check quality: have them look at the whole app or site holistically. Have them try to find the bugs that developers won't be able to find.

Post reply on HN