Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most Unit Testing Is Waste (2014) [pdf]

rbcs-us.com

Post reply on HN
Print view View original

81–90 of 164 posts

Re: Most Unit Testing Is Waste (2014) [pdf]

#81

Post by will4274 »

xorcist wrote: post #68
quanticle wrote: post #46

Earlier quoted context omitted.

The problem is that sophisticated type systems only catch a subset of the bugs that a unit test can catch. For example, let's say I'm adding the ability to transfer funds from one account to the other in a banking application. I want to display a warning when the amount of money being transferred is over a certain percentage (let's say 95%) of the funds in the account. That's pretty easy to do in a unit test: create…

At the risk of veering off topic, I'm genuinely curious what kind of bug that unit test is designed to catch? Given that transferFunds shows a warning over that 95% threshold, what type of change is going to break that logic? Some sort of botched code rewrite where values doesn't retain their meanings? That batchMode shouldn't trigger the warning? That race condition between the check and the actual transfer? Most of…

> Some sort of botched code rewrite where values doesn't retain their meanings?

Yes, exactly. It's supposed to catch the rewrite where somebody turns (a * 100) / b > 95 into (a / b) * 100 > 95 and suddenly, the code doesn't work anymore.

Showing a warning is full scenario - it has a whole UX. If this warning is important enough to have a PM, a UX designer, and be translated into 20 languages, it's important enough for the engineer to make sure it actually shows up when its supposed to.

Re: Most Unit Testing Is Waste (2014) [pdf]

#82

Post by ChrisMarshallNY »

aey wrote: post #13
ChrisMarshallNY wrote: post #12

I've never tested 100% of my code. But I've also never written a unit test that didn't expose bugs. That's mainly because I know what parts of the code will be dicey, and focus unit tests on them.

That’s funny. I only write tests to prove that the program does what it’s meant to do and not to find bugs. The goal is to prevent a future programmer, including myself, from breaking any of the declared properties.

For me, it’s all about maximizing quality; which is usually about bugs. For example, I’m currently writing an ONVIF driver. This needs to address hundreds of devices. I only have about ten to test against. They feature a wide range of implementation, but they are, by no means, full coverage. I’m gonna need to mock up some crazy, nonexistent devices to stress a few particular parts of the library. If I can make it work with them, then chances are better than even that it will work with devices I can’t directly test. I use test harnesses to ensure functionality. My test harnesses tend to be huge; often, a lot bigger than actual shipping implementations.

Re: Most Unit Testing Is Waste (2014) [pdf]

#83

Post by chongli »

iso-8859-1 wrote: post #30
chongli wrote: post #22

Earlier quoted context omitted.

It adds to the technical debt of the project. Some teams may never decide they're "ready" for a more sophisticated type system. That's fine, but they're still on the hook for maintaining those test suites which might pose significant roadblocks to refactoring or rewriting parts of the system.

You can prove a merge sort correct with dependent types. However, a typed implementation of merge sort will arguably be more difficult to mold into a TimSort if it isn't dependently typed. What you say?

I don't have any experience with dependently typed languages. I hope (schedule permitting) to take a course in dependently typed programming (using Agda) and formal verification in the fall. I'll have much more to say on the topic after completing the course.

Re: Most Unit Testing Is Waste (2014) [pdf]

#84

Post by jlg23 »

quanticle wrote: post #46
chongli wrote: post #14

Earlier quoted context omitted.

I'll say that any unit test for a bug which would have been caught by a more sophisticated type system is a waste. I don't know how much time people spend writing such "obsolete tests", but I doubt it's insubstantial.

The problem is that sophisticated type systems only catch a subset of the bugs that a unit test can catch. For example, let's say I'm adding the ability to transfer funds from one account to the other in a banking application. I want to display a warning when the amount of money being transferred is over a certain percentage (let's say 95%) of the funds in the account. That's pretty easy to do in a unit test: create…

With dependent types: https://en.m.wikipedia.org/wiki/Dependent_type

Re: Most Unit Testing Is Waste (2014) [pdf]

#85

Post by walshemj »

acjohnson55 wrote: post #29
chongli wrote: post #14

Earlier quoted context omitted.

I'll say that any unit test for a bug which would have been caught by a more sophisticated type system is a waste. I don't know how much time people spend writing such "obsolete tests", but I doubt it's insubstantial.

What if most type system security is a waste? I don't personally believe this is the case, but I also can't tell you I know it to be false. What I do know is that types have a non-zero cost, and I think sometimes that's ignored, and only their benefits are acknowledged. But the efforts to bring types to historically dynamic languages show that not all common dynamic language idioms are amenable to reasonable type sig…

Apart from a bit more typeing up front I cant see many costs to typing.

Re: Most Unit Testing Is Waste (2014) [pdf]

#86

Post by desc »

When I started programming, I'd think up lots of clever ways to avoid repeating things. Ten years on, that code is a nightmare to maintain because changing the behaviour for just one call site of hundreds is next to impossible, because everything gets funnelled through one extremely DRY group of modules.

Redundancy vs. Dependencies: it's the dependencies which kill you. Redundancy is often a good thing.

If you state an algorithm only once, as the implementation, then the next programmer only knows what it does, not whether it's correct.

This is, in my opinion, the main value of unit tests: state the algorithm twice, once as implementation and once as expectations, and if they don't agree then something's wrong. While the odds of a bug existing in any line of code haven't changed, the odds that the exact same bug exists in both sides are much lower.

Any bugs which survive that are probably in design rather than implementation, ie. my mental model of what the module should achieve is wrong somehow. Catching that is a job for integration tests.

I definitely agree with the 80/20 rule here. 100% code coverage is neither necessary nor desirable, and 20% is fine if it's the most valuable 20%.

Re: Most Unit Testing Is Waste (2014) [pdf]

#87

Post by walshemj »

sulam wrote: post #38
pknopf wrote: post #34

Earlier quoted context omitted.

> espousing more use of integration and system tests, which are clearly combinatorial harder to test “completely” The point is that when you do integration tests, you will test the underlying classes the way in which it actually matters . You can't test every possible condition a unit can have, but you can test for the most likely.

This is no more or less likely in an integration test vs a unit test. One of the first things you learn in writing unit tests is that you should be selective in picking your inputs. This is even taught in those “trade schools” he subtly derides. You regularly see both public test cases and hidden test cases run against problem sets, at least that’s what my interns tell me.

Interesting so they don't teach testing by deliberately trying to break things, pasting Arabic text into an address field for example.

When testing NAPI like to use edge cases like the company with the longest name eg "Donaudampfschiffahrtsgesellschaft" or the Famous Welsh location "Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch"

I am pretty sure I crashed an A17 clearpath mainframe by doing aggressive testing

Re: Most Unit Testing Is Waste (2014) [pdf]

#88

Post by nickjj »

xorcist wrote: post #68
quanticle wrote: post #46

Earlier quoted context omitted.

The problem is that sophisticated type systems only catch a subset of the bugs that a unit test can catch. For example, let's say I'm adding the ability to transfer funds from one account to the other in a banking application. I want to display a warning when the amount of money being transferred is over a certain percentage (let's say 95%) of the funds in the account. That's pretty easy to do in a unit test: create…

At the risk of veering off topic, I'm genuinely curious what kind of bug that unit test is designed to catch? Given that transferFunds shows a warning over that 95% threshold, what type of change is going to break that logic? Some sort of botched code rewrite where values doesn't retain their meanings? That batchMode shouldn't trigger the warning? That race condition between the check and the actual transfer? Most of…

> My Personal preference is for tests to be a bit higher level (does login with a username containing null still cause an exception?).

These tests are great, but what if username has a bunch of validations on it?

For example, it's reasonable to think a username field might be validated with:

- Must be required

- Within 1-32 characters that match a certain pattern (let's say a regex to limit it to lowercase letters, numbers and -)

- Must not be a blacklisted word (admin, administrator, etc.)

- Must be unique (enforced with a database index)

Pretty standard stuff. Are you going to write 5 integration tests for this? 4 to test each validation and then the success case? These would be tests that exercise your entire web framework's routing stack from request to response (ie. the user visiting a /register URL and then submitting the form).

Personally I would not. I would write 1 unit test for each of those things (4 unhappy cases where I assert a specific validation error for each invalid input and 1 happy case where with valid input I expect 0 validation errors). In this case, the "unit" would likely be a `register_user` function that accepts params as input and either aborts with validation errors, or succeeds by writing the record to the DB.

Then, for an integration test I would have 2 tests. One to make sure with invalid input I end up with some type of error displayed in the HTML response (it doesn't matter which one), and another test with the success case to make sure things work when they should (such as the user is registered and a new record was created in the DB).

So I end up with a tiny bit of overlap in tests. Technically the unit test for the success case doesn't need to be there since the integration test covers it but I usually include it for the sake of completeness because it's usually like 4 lines of code to make that test but I'm not 100% opposed to someone saying it should be left out.

Re: Most Unit Testing Is Waste (2014) [pdf]

#89

Post by nrook »

njharman wrote: post #21

I did not read the article. I did read all comments Unit tests for the most part aren't about "testing". They are developer tool. To verify rthat modifications (refactoring, additions, bug fixes, etc) doesn't break contracts etc. Oh and showing that your code is a codependent mess of poorly isolated spagehtti, if your unit tests are hard to write, the code under test is a mess. Inittests are more useful in languages…

Surprisingly, the article doesn't engage with this argument! Unless I missed it, the author never mentions that unit tests can make refactoring other code easier.

In my experience, after code is checked in, unit tests have two main purposes:

1. Checking that functions aren't completely broken in dynamically typed languages (i.e. a check so basic that a static type checker can do its job) 2. Allowing programmers to refactor code without being terrified it will break something far away in the execution path

And like, that's it. That's enough to make them useful to have around.

Re: Most Unit Testing Is Waste (2014) [pdf]

#90

Post by seppel »

quanticle wrote: post #46
chongli wrote: post #14

Earlier quoted context omitted.

I'll say that any unit test for a bug which would have been caught by a more sophisticated type system is a waste. I don't know how much time people spend writing such "obsolete tests", but I doubt it's insubstantial.

The problem is that sophisticated type systems only catch a subset of the bugs that a unit test can catch. For example, let's say I'm adding the ability to transfer funds from one account to the other in a banking application. I want to display a warning when the amount of money being transferred is over a certain percentage (let's say 95%) of the funds in the account. That's pretty easy to do in a unit test: create…

While I agree that this is hard to implement with a type system, I never want to see this implemented as a unit test.

What you describe is a real use case that I personally want to have tested on a real system with all the stuff in place (but test database instead of the real one). It has nothing to do with one unit you can test on its own.

And the crucial problem here is that you tie your testing code on the inner workings of transferFunds, when you are mocking stuff.

> and verify that the warning is triggered when the value being transferred is over 95% of the amount in the account.

Please also note that you cannot verify this property using testing. You can only verify that this works for one particular set of values (or a big but finite set of values). Testing never can verify that something always works. That's the wrong tool for this job.

Post reply on HN