Live data from Hacker News

Most Unit Testing Is Waste (2014) [pdf]

rbcs-us.com

61–70 of 164 posts

Re: Most Unit Testing Is Waste (2014) [pdf]

#61
post #9

I got progressively more and more frustrated with this article, largely because he keeps making statements about the impossibility of covering all states a class may take on (true!) but then followed up with espousing more use of integration and system tests, which are clearly combinatorial harder to test “completely”. He also implied at the very beginning that somehow this was driven by the switch from FORTRAN to OO…

For the same number of lines of test code, an integration test case traverses through much more source code than a unit test case. Not only that, but it also checks that the units are interacting properly and passing each other the right data and returning the right data. Good integration tests can also fee us from static typing because the integration tests implicitly check that the interface contracts between all the different units are met internally. It also verifies many other factors which affect the robustness of the software. For example, they can identify race conditions and prevent them in the future.

Re: Most Unit Testing Is Waste (2014) [pdf]

#62
post #14

Earlier quoted context omitted.

I'll say that any unit test for a bug which would have been caught by a more sophisticated type system is a waste. I don't know how much time people spend writing such "obsolete tests", but I doubt it's insubstantial.

Is it a waste to keep the test around until you're ready to switch to a more sophisticated type system?

A question, coz Im not sure I follow. How is type system (as described by wiki) supposed to test domain or functionality?

Or are you talking here about completely different type system ?

Re: Most Unit Testing Is Waste (2014) [pdf]

#63

One of the things I dislike the most about unit tests is how it's used for "quality theater". Some examples: - Trying to use coverage percentage metric as a sign of quality. As if a simple percentage means anything about the quality of the tests. It's as useless as using lines of code as a way to measure progress. - Not recognizing that useless unit tests are harmful for the code maintenance. It makes refactoring cod…

> I really hate useless unit tests. I've seen tests that setup a mock, configure it to return a value when called, call the code using the mock, then check the returned value is the mocked value. This tested absolutely nothing!

I've had a similar disagreement about the validity of such a test.

The other mocking issue I've seen, is people mocking blackbox third-party APIs over which they have absolutely no control, which sometimes leads to passing unit tests, failing integration tests and head scratching all around.

Re: Most Unit Testing Is Waste (2014) [pdf]

#64

I did not read the article. I did read all comments Unit tests for the most part aren't about "testing". They are developer tool. To verify rthat modifications (refactoring, additions, bug fixes, etc) doesn't break contracts etc. Oh and showing that your code is a codependent mess of poorly isolated spagehtti, if your unit tests are hard to write, the code under test is a mess. Inittests are more useful in languages…

Unit tests don't guarantee that your code units are well designed. You can still write unit tests for spaghetti code. In fact, sometimes it encourages it because it encourages antipatterns like dependency injection; because developers get the urge to inject mock objects into the units from the unit tests.

Just think about it at a high level. If your module depends on another module, is it really always necessary that this module be substitutable for another module that has the same interface (e.g. another similar module or a mock)? 99% of the time, the answer is no! In fact it's often a problem because this wrongly assumes that if a class exposes the required interface, then it is compatible. There is more to compatibility than just interface; if the submodule maintains its own state (OOP), then its behaviour can change in a way which could break the dependent logic. You can't have Dependency Injection for everything because it's never so simple; these customizable parts of the code have to be designed very carefully.

Re: Most Unit Testing Is Waste (2014) [pdf]

#65
post #37

Unit Tests = The assurance/confidence that you get, if something changes logic, this test will break & you will know it. Aside from just limiting bugs - this is more powerful.

95% of the time when a unit test fails for me, it is the test that need fixing instead of my code. It hardly inspires confidence.

To me this suggests that there is no clear understanding what your units are supposed to do.

My second theory would be that the unit tests are written by the inexperienced developers while the good developers write the other code.

Re: Most Unit Testing Is Waste (2014) [pdf]

#66

My experience with testing is like that old adage about advertising, "I know I'm wasting 50% of my money but I don't know which 50%." Most of it is a waste but it's hard to know in advance which test will stop an engineer in a couple of years from altering some fundamental contract and bringing the system down.

I have a 3 part article on testing that I need to post...

Regarding wasting time on 50% tests... it depends what your system is. In most of the systems (the no mission critical ones, no general libraries) I like to use the 80%/20% rule. I always ask: what are the 20% of the tests that deliver 80% of the value?

You learn with experience but if you do not have it, this is a simple way to learn (as a team).

During the sprint, have people write down:

1 - The tests that actually failed and found a real problem before the code was shipped.

2 - The tests you wrote for the bugs that were found in production during the sprint. If you haven't written the test yet, ask: what kind of test would have found this bug?

Have people in the team share in the retrospective meeting. If you need more time at the beginning you can hold a meeting just for bug sharing. Discuss about the patterns. Commit to write more of these kind of tests.

In a few sprint, you should see regression bugs going down. If you also take a look at the tests than never fail, you will also have a sense of the bugs you should not write.

Let me know how it goes!

Re: Most Unit Testing Is Waste (2014) [pdf]

#67
post #14

Earlier quoted context omitted.

I'll say that any unit test for a bug which would have been caught by a more sophisticated type system is a waste. I don't know how much time people spend writing such "obsolete tests", but I doubt it's insubstantial.

The problem is that sophisticated type systems only catch a subset of the bugs that a unit test can catch. For example, let's say I'm adding the ability to transfer funds from one account to the other in a banking application. I want to display a warning when the amount of money being transferred is over a certain percentage (let's say 95%) of the funds in the account. That's pretty easy to do in a unit test: create…

I think the point is that if that was written in JS you would have a test to ensure your method behaves correctly with characters passed in, or strings, or just nil(/nul?) Whereas in Java et Al, you don't need any of those tests, you just make your parameter some sort of numeric and that just isn't a problem any more.

Re: Most Unit Testing Is Waste (2014) [pdf]

#68
post #14

Earlier quoted context omitted.

I'll say that any unit test for a bug which would have been caught by a more sophisticated type system is a waste. I don't know how much time people spend writing such "obsolete tests", but I doubt it's insubstantial.

The problem is that sophisticated type systems only catch a subset of the bugs that a unit test can catch. For example, let's say I'm adding the ability to transfer funds from one account to the other in a banking application. I want to display a warning when the amount of money being transferred is over a certain percentage (let's say 95%) of the funds in the account. That's pretty easy to do in a unit test: create…

At the risk of veering off topic, I'm genuinely curious what kind of bug that unit test is designed to catch?

Given that transferFunds shows a warning over that 95% threshold, what type of change is going to break that logic? Some sort of botched code rewrite where values doesn't retain their meanings? That batchMode shouldn't trigger the warning? That race condition between the check and the actual transfer?

Most of the unit testing I see is more like test-of-defintions (is fullName still 80 characters wide and doesn't accept null bytes?) which confuses me because a definition can only be correct or incorrect in context.

I accept the usefulness in dynamic languages in the absence of types. My Personal preference is for tests to be a bit higher level (does login with a username containing null still cause an exception?) but I have come to terms with the fact that few people agree with me across multiple organizations, so there must be some point to this trivial testing that is completely lost on me.

Re: Most Unit Testing Is Waste (2014) [pdf]

#69

My experience with testing is like that old adage about advertising, "I know I'm wasting 50% of my money but I don't know which 50%." Most of it is a waste but it's hard to know in advance which test will stop an engineer in a couple of years from altering some fundamental contract and bringing the system down.

My take on that is to make tests using a unit test lib, but a bit more macro. Midway between unit tests and e2e. It's not pure, it's not best practices, but it's great value.

Re: Most Unit Testing Is Waste (2014) [pdf]

#70
post #14

Earlier quoted context omitted.

I'll say that any unit test for a bug which would have been caught by a more sophisticated type system is a waste. I don't know how much time people spend writing such "obsolete tests", but I doubt it's insubstantial.

The problem is that sophisticated type systems only catch a subset of the bugs that a unit test can catch. For example, let's say I'm adding the ability to transfer funds from one account to the other in a banking application. I want to display a warning when the amount of money being transferred is over a certain percentage (let's say 95%) of the funds in the account. That's pretty easy to do in a unit test: create…

This is very much not the kind of bug that the parent was talking about. He's talking about stuff like passing an int value to a (supposedly) string parameter. Basic type stuff.

I also agree, as years ago I wrote a large amount of javascript which had type assertions pretty much everywhere. They didn't take long to write but I'm very glad I did them, and I'd rather have the language do it to save me time, clutter and maintenance.

Lax typing is a real cost. I've been working on some SQL I inherited and even SQL's not-too-bad typing allowed the original coder to mix types where I wouldn't, and create potential runtime errors eg. to assign the contents of a 64-bit integer field to an 32-bit integer field. That's legal and gives no warning. It's also suddenly a runtime error when it exceeds 2^31 (and there are other possible consequences such as screwing up the optimiser). I'd much rather it forced me to match types exactly, or explicitly cast.

Post reply on HN