Live data from Hacker News

Most Unit Testing Is Waste (2014) [pdf]

rbcs-us.com

21–30 of 164 posts

Re: Most Unit Testing Is Waste (2014) [pdf]

#21
I did not read the article. I did read all comments

Unit tests for the most part aren't about "testing". They are developer tool. To verify rthat modifications (refactoring, additions, bug fixes, etc) doesn't break contracts etc. Oh and showing that your code is a codependent mess of poorly isolated spagehtti, if your unit tests are hard to write, the code under test is a mess. Inittests are more useful in languages with loose or poor type systems.

Many unit tests are not written well. testing more than the interface/contract. Or full of complexity boilerplate and mock. Which means code needs fixing.

Re: Most Unit Testing Is Waste (2014) [pdf]

#22
post #14

Earlier quoted context omitted.

I'll say that any unit test for a bug which would have been caught by a more sophisticated type system is a waste. I don't know how much time people spend writing such "obsolete tests", but I doubt it's insubstantial.

Is it a waste to keep the test around until you're ready to switch to a more sophisticated type system?

It adds to the technical debt of the project. Some teams may never decide they're "ready" for a more sophisticated type system. That's fine, but they're still on the hook for maintaining those test suites which might pose significant roadblocks to refactoring or rewriting parts of the system.

Re: Most Unit Testing Is Waste (2014) [pdf]

#23
post #13

I've never tested 100% of my code. But I've also never written a unit test that didn't expose bugs. That's mainly because I know what parts of the code will be dicey, and focus unit tests on them.

That’s funny. I only write tests to prove that the program does what it’s meant to do and not to find bugs. The goal is to prevent a future programmer, including myself, from breaking any of the declared properties.

[deleted]

Re: Most Unit Testing Is Waste (2014) [pdf]

#24

I've never tested 100% of my code. But I've also never written a unit test that didn't expose bugs. That's mainly because I know what parts of the code will be dicey, and focus unit tests on them.

In terms of TDD I can't make sense of this. Do you only write tests after a bug has been found?

Doesn't sound like TDD, but being purely responsive doesn't follow either.

If I'm doing I/O on untrusted data, of course I'm going to fuzz test it thoroughly and probably find bugs.

If I'm writing new RAII types - containers, smart pointers, etc. - I'm going to write tests in an attempt to exercise double frees, or any kind of rule-of-3+ violation I can think of, and I'll probably find at least one thing I overlooked if it's complicated enough. The people building code need to be able to trust some of their foundational tools, at least, to be nearly bug free.

If I'm abstracting system APIs into a cross platform representation, I'm going to write unit tests to compare behavior, because the abstraction is probably leaky and fails to fully abstract in some edge case - possibly due to bugs in one of the N system APIs I'm targeting, or possibly because I simply forgot to implement a codepath, or possibly due to strange edge cases.

If I'm writing SIMD abstractions, I'm going to write some basic math unit tests to catch the occasional compiler codegen bug for the less thoroughly used and tested intrinsics.

If I'm writing code that I simply know to be fundamentally brittle, I write tests to catch when it eventually breaks. For example, I wrote some unit tests for rust, to catch when the module standard types are implemented within internally change, which will in turn break the .natvis files relying on those internal type names.

If I'm writing lock-free multithreaded code, I'm going to write a metric ton of tests to try and suss out edge cases, because I know a single bug slipping through can result in weeks of time lost to chasing heisenbugs, and I know my reviewers probably can't catch everything either, and you'd better believe it's going to catch a bug at some point.

Re: Most Unit Testing Is Waste (2014) [pdf]

#25
post #9

I got progressively more and more frustrated with this article, largely because he keeps making statements about the impossibility of covering all states a class may take on (true!) but then followed up with espousing more use of integration and system tests, which are clearly combinatorial harder to test “completely”. He also implied at the very beginning that somehow this was driven by the switch from FORTRAN to OO…

I'm going to pile onto your comment, since I heartily agree.

> In a given computing context, the exact function to be called is determined at run-time and cannot be deduced from the source code [in OOP languages] as it could in FORTRAN.

This is not necessarily true, and I don't think this was true when this was written, either. (The Wayback Machine first saw this in 2014, and the paper doesn't date itself.)

Most member function calls in C++ (non-virtual ones) can be resolved at compile-time. Where idiomatic C++ could would use a virtual function would require some equally uninspectable construction in any language, because one would use it when one needed run-time switching. Rust lacks OOP in the usual sense, but if I needed a virtual function of sorts, I might reach for a trait type, which also wouldn't be inspectable (beyond whatever semantics the trait establishes).

There's Java, JS, and Python, I do admit. And I find my unit tests often fill in the static analysis I wish I had. But the truth is more nuanced than I think the author conveys.

> Unit tests are unlikely to test more than one trillionth of the functionality of any given method in a reasonable testing cycle. Get over it.

> (Trillion is not used rhetorically here, but is based on the different possible states given that the average object size is four words, and the conservative estimate that you are using 16-bit words).

This is as vapid a definition of test coverage as the line coverage the author derides in the prior paragraphs. A trivial function,

  fn add_these(a: u64, b: 64) -> u64 {
    a + b
  }
could not be adequately unit-tested, because I could never cover the many trillions of input states it has.

I think there's always an implicit line to "well, if the code under test is absolutely nuts", it's not the test's fault for not catching it, and that if the function under test (with the same contract as the function above) is errantly coded something like,

  fn add_these(a: u64, b: 64) -> u64 {
    if a == 11771008849893880921 && b == 5668622331333919113 {
      0
    } else { a + b }
  }
then what amount of testing is going to catch that?

> I define 100% coverage as having examined all possible combinations of all possible paths through all methods of a class, having reproduced every possible configuration of data bits accessible to those methods, at every machine language instruction along the paths of execution.

While this somewhat contradicts the earlier example of states. I don't think we should aspire to this, since I think many "combinations" are not interesting to test; they're just pieces of the code that really don't interact; testing them is redundant. Yes, this hurts our ability to formally define a protocol for testing or not testing, but it prevents the inevitable combinatorical explosion.

> Business people, rather than programmers, should design most functional tests.

Absolutely not. A large part of my job is coming up with the actual requirements of the system from the vague machinations of business people who barely know how a computer works, let alone can specify salient requirements. How are they to write the tests, when they can't clearly articulate the requirements?¹

> Turn unit tests into assertions.

But I want to ensure these don't fire under normal circumstances. We do this, in my current code base, and we get notified of them, and of any crashes in general. But it's a failure that was noticed, and might have downstream consequences, as opposed to one caught by a unit-test.

Then the parts about Eastern Europeans with bad Internet make better programmers because they had to think instead of using the Internet, and how he grew up under similar circumstances. Real Programmers!

¹I'm not saying business types "should" (or should not) be able to articulate requirements/specifications.

Re: Most Unit Testing Is Waste (2014) [pdf]

#26
post #14

My experience with testing is like that old adage about advertising, "I know I'm wasting 50% of my money but I don't know which 50%." Most of it is a waste but it's hard to know in advance which test will stop an engineer in a couple of years from altering some fundamental contract and bringing the system down.

I'll say that any unit test for a bug which would have been caught by a more sophisticated type system is a waste. I don't know how much time people spend writing such "obsolete tests", but I doubt it's insubstantial.

Presumably you’d need to also know that the unit tests aren’t being used locally while developers develop. It’s pretty common to run a unit test watcher while doing local development, and have that catch breakages that never make it to CI.

Re: Most Unit Testing Is Waste (2014) [pdf]

#27

I did not read the article. I did read all comments Unit tests for the most part aren't about "testing". They are developer tool. To verify rthat modifications (refactoring, additions, bug fixes, etc) doesn't break contracts etc. Oh and showing that your code is a codependent mess of poorly isolated spagehtti, if your unit tests are hard to write, the code under test is a mess. Inittests are more useful in languages…

Some of our older tests at work are an absolute nightmare to deal with. We went through a phase where we basically just mocked everything non-trivial and ended up with a rewording of the code itself. Any time you make a change to that code, even if it still behaves exactly the same from a user point of view, the tests blow up into pieces and you end up having to rewrite or throw out most of them.

We moved to a behavior-focused testing style, and it has been much more robust to non-functional changes and refactorings in the code.

Re: Most Unit Testing Is Waste (2014) [pdf]

#28

Unit Tests = The assurance/confidence that you get, if something changes logic, this test will break & you will know it. Aside from just limiting bugs - this is more powerful.

Having a robust test suite is very nice. Having an overly pedantic test suite is not. Things like getters / setters that use the same library you've used 1000 other places have no need to be tested again. Add tests for the base concepts and trust that they'll continue working based on those.

Re: Most Unit Testing Is Waste (2014) [pdf]

#29
post #14

My experience with testing is like that old adage about advertising, "I know I'm wasting 50% of my money but I don't know which 50%." Most of it is a waste but it's hard to know in advance which test will stop an engineer in a couple of years from altering some fundamental contract and bringing the system down.

I'll say that any unit test for a bug which would have been caught by a more sophisticated type system is a waste. I don't know how much time people spend writing such "obsolete tests", but I doubt it's insubstantial.

What if most type system security is a waste?

I don't personally believe this is the case, but I also can't tell you I know it to be false. What I do know is that types have a non-zero cost, and I think sometimes that's ignored, and only their benefits are acknowledged. But the efforts to bring types to historically dynamic languages show that not all common dynamic language idioms are amenable to reasonable type signatures. While some idioms (e.g. monkey patching) are problematic, that's not the case of all of them.

Re: Most Unit Testing Is Waste (2014) [pdf]

#30
post #22

Earlier quoted context omitted.

Is it a waste to keep the test around until you're ready to switch to a more sophisticated type system?

It adds to the technical debt of the project. Some teams may never decide they're "ready" for a more sophisticated type system. That's fine, but they're still on the hook for maintaining those test suites which might pose significant roadblocks to refactoring or rewriting parts of the system.

You can prove a merge sort correct with dependent types. However, a typed implementation of merge sort will arguably be more difficult to mold into a TimSort if it isn't dependently typed. What you say?
Post reply on HN