I am not sure you want to delay penetration testing until you are post IPO.
QA = Time and Money. How much should you invest?
11–20 of 57 posts
Re: QA = Time and Money. How much should you invest?
#12Can someone please explain why one prefers unit testing to integration / acceptance testing when you're a seed stage company? I mean, your primary objective as an early stage company should be getting an operational product out to the market, so it makes sense that you test that your users can operate your product - aka integration / acceptance test with crapybara or phantomjs. On the other hand, if you're just unit testing you're bound to miss the bigger picture (usually this shows up in the form of misspellings in your html views, some CSRP bullshit, external apis misbehaving, etc.), and will wind up with all tests that pass but a product that still doesn't work.
Re: QA = Time and Money. How much should you invest?
#13As a consultant, I used to ask my clients to QA my deliveries. Few of them were willing to, to the extent they did they could not or would not provide meaningful bug reports. This led to my adoption of test-driven development - mostly but not all automated testing - with the eventual result that I advertised that my final deliverables would be ready to ship to end-users, without any requirement for QA by my clients.
TDD is great. It should be a given on any project that isn't just a spike. I'm curious what kind of products you were building? Simple websites, or full fledged applications? 100% unit test coverage can still miss a host of gnarly bugs. A lot of the ugliness of QA comes from how A interacts with B. Or how A + B + C + D work together. Or at least, that's my 2 cents :D
You also have to qualify 100% coverage: SUT lines, branches, or paths? Those are all very different levels of rigor.
The two most useful code coverage stats I've found are 0 and "less than yesterday." The former tells me something pretty important about the culture of the team who owns that code; and having rules against the latter is about the only way to ensure a test-on-checkin policy.
Re: QA = Time and Money. How much should you invest?
#14I am not sure you want to delay penetration testing until you are post IPO.
Agreed.
Caveat: do you need to do pen testing every release? I'm guessing, not. YMMV.
If you're a defense contractor (as some other commenters mentioned), your priorities are probably quite different.
Re: QA = Time and Money. How much should you invest?
#15As a consultant, I used to ask my clients to QA my deliveries. Few of them were willing to, to the extent they did they could not or would not provide meaningful bug reports. This led to my adoption of test-driven development - mostly but not all automated testing - with the eventual result that I advertised that my final deliverables would be ready to ship to end-users, without any requirement for QA by my clients.
I've been taking this approach. I'm the sole developer on my company's API so it's important to me to not waste time fixing issues that get brought up by the UI team when they start working on with the API.
To that end, I write integration tests in that all my tests call the REST endpoint then verify the result of the API call. My tests treat the API call as a black box.
A single endpoint can have dozens or hundreds of tests. By the time the API gets to the UI guys, I know it's going to work correctly (except in those cases where I missed testing certain scenarios).
Not only does it save me from bug reports, but it makes me look good because everything I ship works.
Re: QA = Time and Money. How much should you invest?
#16Wow, crazy timing. We interviewed Fred at Rainforest on the Talking Code podcast – and that interview launches tonight at midnight (PDT).
I'm going back through and picking out my favorite quotes right this moment in preparation for the episode launch. Some of my favorites:
"How did we get to the place where the generally accepted best practice is not very good?" (referring to automated testing)
"The visual representation is only somewhat semi linked to the actual underlying code that represents it."
"A really good QA person can actually start to QA the requirements themselves."
"Good QA people are the customer before the customer."
"Shipping a bug at Facebook scale is pretty catastrophic."
"For tiny company X, a YC company, shipping a bug – while painful – is not going to be the end of the world."
"For 95% of people, QA is not a competitive advantage, so it doesn’t make sense to have it in house."
This has honestly been one of my favorite interviews ever, and I'm just really lucky it happened on my own podcast.
http://talkingcode.com/podcast/episode-6-fred-stevens-smith/
Re: QA = Time and Money. How much should you invest?
#17This article, as is typical around here, addresses a very small part of the industry. Here's a view from a different part; the part where a project has a lifecycle of a decade or so. I used to work for a software consultancy that had been acquired by a US defence company (second tier). Over the previous 15 years, the software consultancy had elevated their QA to the point that bugs in the delivered product were liter…
> QA is time and money. The better your QA, the more time you will have and the more money you will make. Just to caveat again that this is for your part of the industry. That level of detail gets you few benefits if you were Facebook, for example, or a very small startup still doing product validation. Also to say that "the better your QA the more money you'll make" is a very strong position to take, even in your in…
Also to say that "the better your QA the more money you'll make" is a very strong position to take, even in your industry.
It is a bit strong, isn't it? I felt that to caveat it might detract from the effectiveness of the point. There does come a point where the software is effectively flawless from the customer's point of view, and doing anything to improve it (including more/better QA) from there doesn't make any more money. But if you reach that point, you know :)
Re: QA = Time and Money. How much should you invest?
#18This is probably the most honest article I've seen, ever, regarding how much QA/testing companies at various stages need. I'm impressed, because there's a real tendency within QA culture to say "the ideal is all the things all the time, we're the gatekeepers," which has forced quality into a bad all-or-nothing situation. We all know intuitively that it's not all needed a lot of the time and the success stories are cl…
As a QA Engineer, I had to create an account to reply to this.
Where are these people saying "the ideal is all the things all the time, we're the gatekeepers,"?
If anyone knows the limitation of QA, it's the people of QA themselves. Trust me, we know that common and non-sensical expectations for "full coverage" and "test everything" are a scourge.
Maybe I'm just lucky to not have worked with the types you describe, but in my experience a QA will be pushing a message of priority-based tradeoffs rather than an unrealistic all-or-nothing approach to quality.
Regarding the article itself, that it advocates being thoughtful about what you test it makes me think that maybe it was written by someone who actually understands QA.
Re: QA = Time and Money. How much should you invest?
#19This article, as is typical around here, addresses a very small part of the industry. Here's a view from a different part; the part where a project has a lifecycle of a decade or so. I used to work for a software consultancy that had been acquired by a US defence company (second tier). Over the previous 15 years, the software consultancy had elevated their QA to the point that bugs in the delivered product were liter…
So, this I agree with too, in contrast with my other comment. There are particular industries (engineering, banking, aerospace, defense, for some examples) where it -has- to be correct. These are also the industries where at least the final product should probably be waterfall or spiral, and generally you need formalism all the way down. The problem comes when people bring lessons learned from that to companies that…
[deleted]
Re: QA = Time and Money. How much should you invest?
#20Wow, crazy timing. We interviewed Fred at Rainforest on the Talking Code podcast – and that interview launches tonight at midnight (PDT). I'm going back through and picking out my favorite quotes right this moment in preparation for the episode launch. Some of my favorites: "How did we get to the place where the generally accepted best practice is not very good?" (referring to automated testing) "The visual represent…
"QA the requirements".
Safety-critical software does not commonly has bugs. The problem it faces, is incorrect requirements.