QA = Time and Money. How much should you invest?

It was the replayability part on old packages I was pushing back on. It's certainly a nice to have, but you're more likely to refer back to old test results than actually run the tests again, so having to architect mechanisms to cache and restore old packages, etc, can be overkill. Just cache the results.

As for docs, it's a matter of level of detail. Most QA groups I've been part of or had visibility in overshoot, and generate a lot of docs that aren't particularly useful for any real process. YAGNI applies to docs too.

Further, the docs themselves aren't modular--they usually end up mixing concerns between product documentation and the testing, and in doing so make themselves extremely brittle to product changes whether or not they should really matter to the tests. UI step/verify docs are the absolute worst about this and the most prevalent kind of test documentation.

I've come to the conclusion that most QA groups would be better off just writing one set of canonical docs for the product, if they don't already exist, and then pointing to that with extra info about what inputs or strategy to use for a given test. It's the same SPOT argument for modularizing automation (or any other kind of code), and has the advantage that others can reuse the product docs.

Re: foundation/earthquake argument, there's something to be said for knowing what you can't live without in the case of a disaster, what you can live without with a little bit of pain, and what you can generate JIT if you need it. The former set is usually pretty small. The middle set is where your educated bets lie. The latter set, you should probably never do up front until you've found yourselves having to scramble a couple of times.

Deming said it: "The most important things cannot be measured." How right he was.

I have never done that, and no-one on my team has ever done that. All tests need to pass when we deploy, every time. The fact is that the cause of regressions are usually so trivial that having unit tests doesn't have any value. All you need to know is that something is broken, and see some error message, and you can track the problem down in a matter of minutes. Starting with high level tests pays dividends on day one. Unit tests have much longer payback times, and a much higher chance that the system will still fail, even if your tests pass.

The second is isolation. Even if you don't have full unit coverage, having -something- beneath the integration tests lets you tell a lot more from the combo of failures. Integration covers ABC and fails, have unit test for B and C, must be A.

I've never failed to identify the cause of a failure in a matter of minutes when a test fails. The concept of isolating the system under test is taken to extremes in XUnit frameworks and the amount of work you have to do to properly isolate the system under test is cost prohibitive. Given the fact that an integration test is by definition a test of any system not sufficiently isolated to XUnit standards, really what you need to have is an integration test that is "as isolated as practical". On greenfield developments, it's typically just a test that makes sure the thing runs at all. Once you start fixing bugs, you write tests that are more focused, but worrying about isolating systems under test (with mock objects and that kind of malarky) kills productivity.

Also, a political reason: once you start down that route it's super-tempting (especially for your boss or your boss's boss) to say "we'll test everything full stack, that gives us all of the coverage." In reality, it only gives you some of it, and you probably have no idea what even with code coverage tools (remember that paths are ultimately what you're testing, not lines or branches). And it's incredibly indirect, since you're trying to find externally driven cases to make the internals do different things. It's usually easier just to manipulate the internals directly via the nearest interface.

Yep, I say "we'll test everything full stack until the QC team finds bugs, then when we fix bugs we'll add specific tests" (I'm the boss). The philosophy we operate under is that it's the job of QC to find bugs, it's the job of automated tests to ensure we only fix each bug once and to reduce the cost of QC (more here: https://github.com/iaindooley/Murphy).

We use a mix of backend integration tests (using Murphy) and frontend integration tests using this: http://monkeytestjs.io/ but the main tenet is: don't waste time testing components on a system you're developing, only test components once they have been shown to fail. Even then, isolating the system under test might happen by coincidence, but is not a requirement of the tests.

If someone gets it wrong, and creates a test that passes even though the bug is still present, it's the job of QC to catch that -- the only requirement is that this happens infrequently enough that it doesn't make QC cost prohibitive.

So one or two high level acceptance/integration sanity tests, sure. But I'd never say that you should have "integration testing in place" first. That implies a level of completeness or formality that I think would probably be counterproductive in almost all cases.

I've found exactly the opposite in all cases. In fact, I've never seen productive Unit Tests (real ones) -- only integration tests that people called unit tests, and in all cases I've seen, people started with high ideals and then all testing was abandoned when the shit hit the fan, mostly because they were too ambitious and low level with test coverage to begin with.