Superior Testing: Stop Stopping

The problem is, regardless of how stable the code was, or how easy it was for the original author to test it manually, if the code hasn't been worked on for ages, then the original author is either gone or can't remember all the details of how it was supposed to work anymore. Which leaves you in a dangerous situation, because there's no good way of knowing which behaviors are by design and which ones are incidental.

I'm currently working for an insurance company, where I created services that interact with legacy systems where you can't touch anything, because nothing can be deleted once in there. I had to submit a formal request for accounts and waited months for them. Once I had them, I had to wait weeks for having the right permissions set for those accounts. There are a test and a pre production environment, but they work differently from the production one, in very subtle ways. For example, the same account cannot login from two different places at the same time, while test and pre-production allow that behaviour.

Long story short: I've mocked the service completely, well just the parts I needed to interact with, and added assertions in my code that if something changes show an error message like "something has changed we're working on a fix" and send me alerts. I could not even get the team working on the legacy system send me an email before changing stuff...

But at least I can test my app and some of the tests are about checking that nothing has been updated in the meantime. They run automatically few times a day (just the checks) and warn me if something is wrong. Still better than a call from a user who usually can't explain what's wrong and you have to repeat the complete procedure on the phone, step by step.

In flutter parlance, a whole view (page/screen) is a widget, and that's the granularity I was testing it - it's good for testing the high level patterns that the screen is meant to adhere to, e.g. Send button appears when there is some text to send.

You can also have golden images for rendering of each screen. They change a lot, but that at least forces developers to look at the changes each time and decide if they're on purpose.

How would you manually test in a situation like that? If you can't interact with the production API because it would corrupt data, and there is no dev API, would you just be guessing that everything works before you deploy your code?

In any case, whilst I think it's good and healthy to debate the different forms of testing, their merits and tradeoffs, etc. I don't think it's appropriate when the alternative is not testing at all (e.g. "customers don't pay for tests").

I've worked at three organisations and managed to introduce automated testing to two of them. Even then, the test suites were "my responsibility", since (a) nobody else was running them and (b) I had to hand-hold people whenever I spotted a breakage (which they inevitably blamed on the test being wrong).

The heuristics I've come to follow are:

- Having automated tests is better than not having automated tests

- It's easier to improve a bad test suite than it is to get a test suite added/accepted

- There usually aren't enough tests

There are exceptions to these rules, but I don't believe them without evidence. For example, "the test suite is slow" won't convince me that there are too many tests; demonstrating systematic redundancy and fragility in the test suite might convince me (e.g. exposing privates in order to test them).

Only if people are on board with the idea of automated testing, will I bother to get more opinionated about the specifics.