Superior Testing: Stop Stopping • Hacker News Board

Re: Superior Testing: Stop Stopping

#41

Post by bluGill » Thu, Jan 31, 2019, 5:33 PM UTC

tokyodude wrote: post #11

lordnacho wrote: post #8

Once you have something nearing a steady state design, you need to have tests. I can understand if something is in a very early stage it might not have tests yet. At my current work we have loads of automatic tests. Every time something is changed, there's pipelines that test the following: - Simple unit tests: Start with state 0, make and action, is state 1 what you expect? There's literally hundreds of these, and t…

I am not convinced this is always true. The first 25 years of my career was spent making games. Never had tests. We had testers but no tests. I then worked at a big company with tests. My first experience with them. Working on something important I loved them but they easily cut my velocity by 70% compared to what I used to get when working on games. The games were AAA and shipped to millions of customers on CDs and…

Anytime you are writing a product where the first version you ship will be the last manual testing is going to be cheaper/faster than automated testing. Only when you are making changes over several years do automated tests really pay off (in a few cases this can be some core libraries to a product that otherwise is ship and forget).

Re: Superior Testing: Stop Stopping

#42

Post by moring » Thu, Jan 31, 2019, 5:45 PM UTC

thomascgalvin wrote: post #38

moring wrote: post #24

The introduction made me hope for real advice on how to "stop stopping", but it only repeated the old arguments on why we want tests. Why not deal with some of the real problems that actually prevent people from writing automated tests? Real-world example 1: The system being developed talks to external system X. We don't want tests to litter the production database, especially since it's about accounting and we would…

> However, there is no possibility to open a test account on the production system, and no budget for a license for a test installation of X. The main troiuble with X is that its public API (web service) changes from version to version and there is no useful documentation about it. How would one write integration tests for that? How would you manually test in a situation like that? If you can't interact with the prod…

> How would you manually test in a situation like that?

IMHO, You can't. We did in fact just hope it doesn't break in production.

But then, the article says just do it, so maybe there was a better solution we overlooked.

Re: Superior Testing: Stop Stopping

#43

Post by moring » Thu, Jan 31, 2019, 5:48 PM UTC

z3t4 wrote: post #28

moring wrote: post #24

The introduction made me hope for real advice on how to "stop stopping", but it only repeated the old arguments on why we want tests. Why not deal with some of the real problems that actually prevent people from writing automated tests? Real-world example 1: The system being developed talks to external system X. We don't want tests to litter the production database, especially since it's about accounting and we would…

If it's an accounting service, make a new account/table named "Software test". Book-keeping was invented to spot errors. If you have a non zero balance in your "Software test" account you probably have a bug in the software.

Unfortunately, this wasn't possible for two reasons. The first one was that we would not just need one account, but several ones. Though it might have been possible to limit this for the integration test, and test all the stuff that deals with multiple accounts by mocking the accounting system.

The second problem is that even a fake account doesn't belong on production, not in accounting. They are very strict about such things.

Re: Superior Testing: Stop Stopping

#44

Post by moring » Thu, Jan 31, 2019, 5:52 PM UTC

crdoconnor wrote: post #25

moring wrote: post #24

The introduction made me hope for real advice on how to "stop stopping", but it only repeated the old arguments on why we want tests. Why not deal with some of the real problems that actually prevent people from writing automated tests? Real-world example 1: The system being developed talks to external system X. We don't want tests to litter the production database, especially since it's about accounting and we would…

1. Create a facade abstraction (it could be a microservice, API or even command line app) over the expensive to test thing. Make the abstraction really dumb and easy to test manually in isolation from the rest of the app. Integration test only against the abstraction, not the real thing and do a minimal level of end to end manual testing against the real thing. 2. Don't. If you don't have fixed requirements your test…

1. Unfortunately, the main issue was a changing interface of the third-party system. The manual testing against the real thing was the problem. I fully agree with automating everything where an abstraction is sufficient.

2. This is good to know. I might have worked with vague requirements for too long, but knowing this may help identify the parts where testing is indeed possible.

Re: Superior Testing: Stop Stopping

#45

Post by gnahckire » Thu, Jan 31, 2019, 6:10 PM UTC

bunderbunder wrote: post #32

olau wrote: post #21

Earlier quoted context omitted.

> but for 90% of projects, automated testing is the most efficient long term strategy I don't think this is a good way to think about it. I see plenty of projects that don't go anywhere, and I also see plenty of projects that do go somewhere, but where large parts change so little and are so easy to test manually that there's little point in much automation. Other parts are complex and full of contradictory business…

I've inherited one or two projects like the first one you describe before, and, when they don't have tests, it's awful . The problem is, regardless of how stable the code was, or how easy it was for the original author to test it manually, if the code hasn't been worked on for ages, then the original author is either gone or can't remember all the details of how it was supposed to work anymore. Which leaves you in a…

Adding in to your point, it's really easy to get familiar with a codebase and be confident that there are no regressions when you make changes if there are comprehensive tests.

Similarly, tests can serve as a device for asserting contributions are acceptable pre/post-merge as well as evaluating that your dependencies work as expected.

Re: Superior Testing: Stop Stopping

#46

Post by skybrian » Thu, Jan 31, 2019, 6:14 PM UTC

millstone wrote: post #13

kstenerud wrote: post #6

Regardless of all the arguments for and against tests, it's important to remember your purpose as an IT professional: Your job is to solve the customer's problem. That's it. HOW you solve their problems is entirely up to you, and some solutions will work better than others. The only thing about customers guaranteed to be consistent is that they will request changes. Some will be good changes. Some will be bad changes…

There's a case for ugly entangled tests too. If you, say, isolate your tests from the filesystem (mocks, etc), your tests can become fast, hermetic, etc. But they lose all sensitivity to differences between filesystems, and encode any assumptions about how filesystems behave. I like to have tests that exercise the real stuff. The tests are flakier but have better fidelity to the actual use.

I generally agree, but we should acknowledge that "the real stuff" can be, in its way, somewhat fake.

Portable code can be run on whatever filesystem the user chooses. We can only test with what we have. So this boils down to another way to say "works on my machine" and that may or may not be relevant depending on how well the OS and hardware we test on matches the customer's setup.

It's good to be able to run your tests against whatever filesystem you choose, though.

Re: Superior Testing: Stop Stopping

#47

Post by Izkata » Thu, Jan 31, 2019, 6:41 PM UTC

> What is better — having a test or not having a test? The answer is obvious — any test is better than no tests at all.

Nooooooo....

For most people here, this is probably true - because you (and hopefully those you work with) know how to write tests well.

Examples of badly written tests I've encountered that wasted everyone's time:

* Unexpected environment - such as, Django doesn't turn off the cache while tests are running

* Tautological tests - where the test just repeats what's in the code

* Peeking too far into the implementation - restricts refactors and can create lots of false negatives or false positives depending on what's being asserted

* Mocking out too much - tests that pass when they really shouldn't

* False assumptions/not thinking it through - why did this test start failing on New Year's Day?

* Flaky integration tests

In our case, about half of these could be updated once the issue was apparent, but the rest either scrapped or completely rewritten.

And then there's a number of issues with test coverage giving you a false sense of security, with things like reaching 100% for a given piece of code, but only thinking about the happy path.

Re: Superior Testing: Stop Stopping

#48

Post by gnahckire » Thu, Jan 31, 2019, 6:49 PM UTC

Izkata wrote: post #47

> What is better — having a test or not having a test? The answer is obvious — any test is better than no tests at all. Nooooooo.... For most people here, this is probably true - because you (and hopefully those you work with) know how to write tests well. Examples of badly written tests I've encountered that wasted everyone's time: * Unexpected environment - such as, Django doesn't turn off the cache while tests are…

I partially agree w/ you. One can always delete the test and then it's like you have no tests at all.

> And then there's a number of issues with test coverage giving you a false sense of security, with things like reaching 100% for a given piece of code, but only thinking about the happy path.

Yup, and this is why one writes tests against well-defined interfaces/boundaries.

Re: Superior Testing: Stop Stopping

#49

Post by bunderbunder » Thu, Jan 31, 2019, 7:23 PM UTC

skybrian wrote: post #46

millstone wrote: post #13

Earlier quoted context omitted.

There's a case for ugly entangled tests too. If you, say, isolate your tests from the filesystem (mocks, etc), your tests can become fast, hermetic, etc. But they lose all sensitivity to differences between filesystems, and encode any assumptions about how filesystems behave. I like to have tests that exercise the real stuff. The tests are flakier but have better fidelity to the actual use.

I generally agree, but we should acknowledge that "the real stuff" can be, in its way, somewhat fake. Portable code can be run on whatever filesystem the user chooses. We can only test with what we have. So this boils down to another way to say "works on my machine" and that may or may not be relevant depending on how well the OS and hardware we test on matches the customer's setup. It's good to be able to run your t…

Concrete example: Docker for Linux, Docker for Mac and Docker for Windows all implement both networking and mounted volumes in slightly different ways.

This has recently eaten up a nonzero amount of my time in maintaining our integration and acceptance test suites.

I'm not sure where I'm going with this, other than to observe that real-ish isn't necessarily a substitute for real. I suppose the "developer empathy" story is that maybe the more empathetic approach in the long run would have been to have developers work on the target platform rather than letting them pick whatever one they used to use at their old gig.

Re: Superior Testing: Stop Stopping

#50

Post by marcosdumay » Thu, Jan 31, 2019, 8:18 PM UTC

lordnacho wrote: post #8

Once you have something nearing a steady state design, you need to have tests. I can understand if something is in a very early stage it might not have tests yet. At my current work we have loads of automatic tests. Every time something is changed, there's pipelines that test the following: - Simple unit tests: Start with state 0, make and action, is state 1 what you expect? There's literally hundreds of these, and t…

Once you have something that keeps changing, you need tests. If it's steady, there is no rush. Before something is written tests add nothing (neither remove), and if it's changing completely (as things usually do at the beginning), you'll still need the tests, but won't be able to get them, sorry.

About tests types, unit tests are worthless unless proven otherwise. Automated analysis is just great, both at static (like types) and run time (like Valgring), do it as much as you can. Integration tests are the actual tests worth their name, everything else is just development tooling. For them, see the first paragraph.