Most Unit Testing Is Waste (2014) [pdf] • Hacker News Board

Re: Most Unit Testing Is Waste (2014) [pdf]

#41

Post by MaulingMonkey » Sat, Jun 29, 2019, 4:39 AM UTC

deathanatos wrote: post #25

sulam wrote: post #9

I got progressively more and more frustrated with this article, largely because he keeps making statements about the impossibility of covering all states a class may take on (true!) but then followed up with espousing more use of integration and system tests, which are clearly combinatorial harder to test “completely”. He also implied at the very beginning that somehow this was driven by the switch from FORTRAN to OO…

I'm going to pile onto your comment, since I heartily agree. > In a given computing context, the exact function to be called is determined at run-time and cannot be deduced from the source code [in OOP languages] as it could in FORTRAN. This is not necessarily true, and I don't think this was true when this was written, either. (The Wayback Machine first saw this in 2014, and the paper doesn't date itself.) Most memb…

Basic code coverage analysis should catch that add_these doesn't have full branch coverage.

Additionally, some fuzzing tools can actually to branch analysis and target your "rare" codepaths. Now, sure, figuring out how to hit some codepaths basically requires brute force - it's not going to magically figure out an efficient way to collide SHA1s if that's what it takes for your test to fail - and you need a way to differentiate good from bad behavior (this might be a reference implementation, or this might be as simple as "does it crash?") - but you do have to get crazier than add_these before you hit the limits of even existing tooling to catch bugs.

Re: Most Unit Testing Is Waste (2014) [pdf]

#42

Post by devin » Sat, Jun 29, 2019, 4:40 AM UTC

Absent from this discussion is the use of generative testing. At the unit level they can automatically enumerate a variety of scenarios the programmer would likely overlook.

Re: Most Unit Testing Is Waste (2014) [pdf]

#43

Post by quality_theater » Sat, Jun 29, 2019, 4:55 AM UTC

One of the things I dislike the most about unit tests is how it's used for "quality theater". Some examples:

- Trying to use coverage percentage metric as a sign of quality. As if a simple percentage means anything about the quality of the tests. It's as useless as using lines of code as a way to measure progress.

- Not recognizing that useless unit tests are harmful for the code maintenance. It makes refactoring code into better structure difficult and developers just give up because it's too much work to fix the tests that weren't even providing any value in the first place. This ignorance is expressed in statements like how there's a testing "pyramid" with unit tests at the bottom and end to end tests at the top. Which is a nice sounding soundbite and image, but is useless. Forget pyramids, just write tests where it makes sense.

- Code review comments that try to look smart with "where's the unit test?" Then the developer doesn't want a long drawn out fight about how a unit test would be useless here since there's a huge crowd just cargo cult yelling "code coverage!" "unit tests are good!" "pyramid!" So the developer just writes the stupid test to get the code merged. This is also an example of how harmful code reviews can sometimes be, when there's a popular stupid idea, code reviews perpetuate it because the developers who know better just get tired of fighting the same fight over and over again.

I really hate useless unit tests. I've seen tests that setup a mock, configure it to return a value when called, call the code using the mock, then check the returned value is the mocked value. This tested absolutely nothing! I've seen unit tests that verify every line of the method was called, completely pointless. The point is supposed to be to verify that for a given input, it has a given output, not lock it into a specific implementation by verifying every line of code ran a certain way.

There is an idea for structuring software "functional core, imperative shell". Write the software this way and the natural place for unit tests and integration tests becomes obvious. But nope, the industry is all about unit test coverage percentage, stupid pyramids, looking good in code reviews. It's all quality theater, not actual focus on quality.

Re: Most Unit Testing Is Waste (2014) [pdf]

#44

Post by kstenerud » Sat, Jun 29, 2019, 5:05 AM UTC

After 15 years of writing unit tests, I haven't reached the same conclusion as the author.

What I've found in my time is that unit testing can be good, but like anything it's not a panacea. It requires discipline, and like normal code, it has code smells.

Black box unit tests are the most likely to be good tests, and white box unit tests are the most likely to be bad tests. The more you depend on the inner workings of a function in order to test it, the more likely it is that you are coupling your test to the implementation rather than the purpose of the unit being tested. Once you tie to the implementation, refactoring becomes a LOT harder, because changes will break the tests even if they don't break the functionality.

Mocks are also a major source of trouble, and more likely to be a code smell. If your tests are using a mock to test how many times your unit called it, either your tests are bad or your architecture is wrong.

There are three main kinds of code:

- Code that fetches data

- Code that stores data

- Code that transforms data

Mocks are necessary when you mix these. If you have a function that opens a DB connection, fetches data, transforms the data, and then stores the data, you now have an extra problem to deal with (the database), when all you wanted to do was test the transformation. Things would be far easier if you separated the transformation out, tested that in isolation, and then integrated that encapsulated functionality with fetching/storing code. This also improves separation of concerns and code duplication, since now your fetching/storing code can be generalized and also tested in isolation.

Actually I lie. There is a fourth kind of code: code that modifies state. This is the evilest, smelliest code around, and it's also something that unfortunately we can't get completely away from. But we can manage it, by isolating state, reducing the need for or scope of the state, and providing "configuration object" function entry points to make testing these monstrosities less nasty.

Code coverage is not just a measure of quality, but also of waste. If your code is not being called, then one of three things is happening:

1. It's error checking code for another API it's calling, which you normally shouldn't be writing tests for (unless that API is known to be buggy and you need to guard against it).

2. It's not contributing to the goals of the program, and can be taken out.

3. It does contribute to the goals of the program, in which case you need a test for it.

You can't reach 100% code coverage because of (1). But you absolutely should check WHICH code is covered in your tests because of (2) and (3). Anything higher than 80% coverage is pure luck, and tells you nothing about quality or wastage. In many cases, even 60-70% is sufficient.

Re: Most Unit Testing Is Waste (2014) [pdf]

#45

Post by MetalGuru » Sat, Jun 29, 2019, 5:20 AM UTC

war1025 wrote: post #27

njharman wrote: post #21

I did not read the article. I did read all comments Unit tests for the most part aren't about "testing". They are developer tool. To verify rthat modifications (refactoring, additions, bug fixes, etc) doesn't break contracts etc. Oh and showing that your code is a codependent mess of poorly isolated spagehtti, if your unit tests are hard to write, the code under test is a mess. Inittests are more useful in languages…

Some of our older tests at work are an absolute nightmare to deal with. We went through a phase where we basically just mocked everything non-trivial and ended up with a rewording of the code itself. Any time you make a change to that code, even if it still behaves exactly the same from a user point of view, the tests blow up into pieces and you end up having to rewrite or throw out most of them. We moved to a behavi…

What do you mean by behavior-focused testing style?

Re: Most Unit Testing Is Waste (2014) [pdf]

#46

Post by quanticle » Sat, Jun 29, 2019, 5:22 AM UTC

chongli wrote: post #14

biesnecker wrote: post #4

My experience with testing is like that old adage about advertising, "I know I'm wasting 50% of my money but I don't know which 50%." Most of it is a waste but it's hard to know in advance which test will stop an engineer in a couple of years from altering some fundamental contract and bringing the system down.

I'll say that any unit test for a bug which would have been caught by a more sophisticated type system is a waste. I don't know how much time people spend writing such "obsolete tests", but I doubt it's insubstantial.

The problem is that sophisticated type systems only catch a subset of the bugs that a unit test can catch. For example, let's say I'm adding the ability to transfer funds from one account to the other in a banking application. I want to display a warning when the amount of money being transferred is over a certain percentage (let's say 95%) of the funds in the account. That's pretty easy to do in a unit test: create mock account, call the transferFunds() method, and verify that the warning is triggered when the value being transferred is over 95% of the amount in the account.

How would I do that with a type system?

Re: Most Unit Testing Is Waste (2014) [pdf]

#47

Post by closeparen » Sat, Jun 29, 2019, 5:36 AM UTC

0xB31B1B wrote: post #36

crdoconnor wrote: post #35

Earlier quoted context omitted.

This is my experience too. Realism in testing is criminally underrated while code coverage is criminally overrated. IME unit tests can only effectively substitute for integration tests where you're testing logical/algorithmic code with simple function inputs/outputs.

I somewhat agree, somewhat disagree. IMO the largest value I get from unit tests is the confidence that I can make changes and understand what breaks. Refactoring w/o unit tests makes me feel like I am flying blind

The units we test are almost never big enough for internal refactoring. It’s the decomposition into units that wants refactoring, and there the test suite actively fights back (mock expectations in particular).

If there were enough code involved in a test that we could meaningfully refactor it while keeping the test green, we would call it an integration test.

Re: Most Unit Testing Is Waste (2014) [pdf]

#48

Post by mrmr1993 » Sat, Jun 29, 2019, 5:49 AM UTC

quanticle wrote: post #46

chongli wrote: post #14

Earlier quoted context omitted.

I'll say that any unit test for a bug which would have been caught by a more sophisticated type system is a waste. I don't know how much time people spend writing such "obsolete tests", but I doubt it's insubstantial.

The problem is that sophisticated type systems only catch a subset of the bugs that a unit test can catch. For example, let's say I'm adding the ability to transfer funds from one account to the other in a banking application. I want to display a warning when the amount of money being transferred is over a certain percentage (let's say 95%) of the funds in the account. That's pretty easy to do in a unit test: create…

To my mind, a unit test should test a unit: something which functions independently, and which is interacted with through an abstraction layer. Type systems are good tools for clarifying and catching bugs around these abstraction layers.

Maybe my understanding is wrong, but what you described doesn't seem like a unit test. I think the appropriate term for a test like this at the junction of UI, business logic and code-level triggers is 'feature test', 'integration test', or maybe even just 'test'.

Re: Most Unit Testing Is Waste (2014) [pdf]

#49

Post by littlestymaar » Sat, Jun 29, 2019, 6:03 AM UTC

mobjack wrote: post #37

the_arun wrote: post #18

Unit Tests = The assurance/confidence that you get, if something changes logic, this test will break & you will know it. Aside from just limiting bugs - this is more powerful.

95% of the time when a unit test fails for me, it is the test that need fixing instead of my code. It hardly inspires confidence.

That's also my experience, but consider how many of the remaining 5% would have led to a regression that would have gotten shipped in production if the test didn't caught it ? In my experience, this number way above zero.

Re: Most Unit Testing Is Waste (2014) [pdf]

#50

Post by MetalGuru » Sat, Jun 29, 2019, 6:12 AM UTC

ereyes01 wrote: post #40

ed-209 wrote: post #17

"The cross product of those paths with the possible state configurations of all global data (including instance data which, from a method scope, are global) and formal parameters is indeed very large. And the cross product of that number with the possible sequencing of methods within a class is countably infinite." Sounds more like a condemnation of OOP than of unit testing, and I do genuinely feel sorry for the unit…

Yeah a lot of the 90s/early 2000s OOP stuff I learned in school seemed to always result in really tightly coupled systems and bespoke webs of tests and fixtures that strung along weird dependency chains in unwieldy spaghetti piles that did no good. Following TDD has helped me land at decoupled functional interfaces like the ones you've described, and it all scales and composes so nicely, yet stays very tractable. Rob…

I wish he gave actual examples to illustrate what he was talking about. So you have an additional API layer the tests hit to call the functions you’re testing? Do endpoints map to classes? Modules? So aren’t you just tightly coupling this API to your service? When the service changes, you still have to update the API. Can you explain to me how this solves the problem?