QA = Time and Money. How much should you invest?

I've done two government contracts, just one for defense.

I won't ever take a defense contract again because the primary contractor selected the wrong part, then hired me as a subcontractor based on my expertise with a part they should not have selected.

This actually came to the point that I spent six weeks furiously trying to find a workaround while they burned a bunch of money making prototype assemblies that did not work, then delivering them to "The Client" with the expectation that they would integrate my firmware later.

I head infosec for a “Series A - C” B2B company and a fairly standard request from a potential customer is to see not only our own penetration test reports but third party penetration test results, as well. As a result, we run automated pen tests on weekends and before major releases. We also work with an application security firm every 6-12 months. For what it’s worth, we don’t do anything nearly as intense as defense contracting or handling financial info.

That said, I liked the article - thanks for sharing.

In my line of accessibility work, I find QA fills the role of testing to educate. If the engineers don't have the experience, looping issues into the backlog, gives them the experience to train up the skill set. They might have had verified experience before hiring, but getting team collaboration helps mature new people or smooth out the rough skill sets of a dev who might be awesome in one particular area but meh in others.

I suspect this could be the case for any subject matter expert where the skill set is viewed as fringe or not expecting the team will be required to have that exact knowledge before hire.

Unit testing is easier to do with the developer resources on hand since they probably already know how to write them, probably necessary for quick pace in dev anyway, and gives you a large portion of the risk mitigation testing in general will give you. You also check in unit tests with the code if you do it right, whereas integration or acceptance tests require additional passes once all the code covered is checked in, so it's more painful to the schedule.

Integration tests also require more development to allow integration while still being hermetic (isolating out any external factors, since you might have to architect to share injected dependencies between modules) -or- more maintenance if you don't make them hermetic and they have to track external dependencies. They require more time to triage failures as well if they're not hermetic, as external failure is a possibility; you can't just trust the test results. And final acceptance tests should never be hermetic--they should accept based on realistic environments--so they're almost always a little high-maintenance.

So while the value is there, the cost is much higher and it'd be more of a distraction from primary development. Cost and benefit have to be considered.

So in this model the integration/acceptance part is handled by dogfooding, which is the lowest cost way to get a decent chunk of it, even though it only walks happy-path for the most part.

But that's why I said I'd add exploratory to B2C seed, is to get back some of the integration testing on non-happy paths and obscure paths.

Edit: find docs on the test pyramid and consider that the shrinking number of tests as you go up also implies somewhat lower technical debt as you skip going up. The shrinkage is a function of the cost vs. benefit--you're just taking it to its logical conclusion of omission.

There are a few reasons to do unit tests first, over and above my direct reply to GP.

The first is that integration tests can be (not always, but can be) too inclusive and thus fail too much, especially early on. A good test process requires tests to stay green most of the time, otherwise you learn to ignore them as "known" and distrust and eventually dismiss the results. So you turn off or xfail failing tests while you triage. But if you turn off or dismiss an integration test as a known failure, you lose a lot of coverage.

The second is isolation. Even if you don't have full unit coverage, having -something- beneath the integration tests lets you tell a lot more from the combo of failures. Integration covers ABC and fails, have unit test for B and C, must be A.

The third is that they're for different purposes. Unit tests are there to tell you what you thought you were guaranteeing with your code has been guaranteed. That's a really important step in knowing that all the other stuff you -didn't- cover with integration testing (remembering that as you add moving parts, you have the combinatorial of all the ways they move--you won't cover it all) is probably correct. And, of course, it's what will ever let you refactor while still knowing your interface is correct.

Finally, there's simple terminology differences as to what's a unit and we might be on the same side. There's debate in the community, but I'm on the side that you always unit test to a public interface, period. Don't test within something not exported or exposed. That may mean you only test a module instead of component classes if they're all private implementation classes. It's still a unit test for that module.

Also, a political reason: once you start down that route it's super-tempting (especially for your boss or your boss's boss) to say "we'll test everything full stack, that gives us all of the coverage." In reality, it only gives you some of it, and you probably have no idea what even with code coverage tools (remember that paths are ultimately what you're testing, not lines or branches). And it's incredibly indirect, since you're trying to find externally driven cases to make the internals do different things. It's usually easier just to manipulate the internals directly via the nearest interface.

One more point: YAGNI is fine and well, but we all know modules or classes usually have behavior that isn't used yet because it's part of an obviously complete API (like, D from your CRUD API when nobody's had to remove anything yet). Unit test that stuff, since there is no integration test that'll cover it. Otherwise it's landmines waiting to be found. The unit tests -are- the client code at that point.

So one or two high level acceptance/integration sanity tests, sure. But I'd never say that you should have "integration testing in place" first. That implies a level of completeness or formality that I think would probably be counterproductive in almost all cases.