Live data from Hacker News

The tragedy of 100% code coverage (2016)

labs.ig.com

301–310 of 346 posts

Re: The tragedy of 100% code coverage (2016)

#301

The worse the developer, the more tests he'll write. Instead of writing clean code that makes sense and is easy to reason about, he will write long-winded, poorly abstracted, weird code that is prone to breaking without an extensive "test suite" to hold the madness together and god forbid raise an alert when some unexpected file over here breaks a function over there. Tests will be poorly written, pointless, and give…

> The worse the developer, the more tests he'll write.

Some of the worst developers I've worked with never wrote any tests. Like zero tests. Actually in one case negative number of tests; the engineer deleted tests I wrote because they were failing the build because of this engineer's (breaking) changes.

Every good developer I know writes tests.

Re: The tragedy of 100% code coverage (2016)

#302

There are a few relevant facts that should be known to everyone (including managers) involved in software development, but which probably are not: 1) 100% path coverage is not even close to exhaustively checking the full set of states and state transitions of any usefully large program. 2) If, furthermore, you have concurrency, the possible interleavings of thread execution blow up the already-huge number of cases fr…

Partly related: Is equivalence partitioning [1] used much in the industry in software testing?

[1] https://en.wikipedia.org/wiki/Equivalence_partitioning

I came across the concept some years ago, IIRC, in the classic book The Art of Software Testing by Glenford Myers [2], and used it in a few projects. But have not really heard or read of it being used much, from people or on forums.

[2] https://en.wikipedia.org/wiki/Glenford_Myers

I think it can be complementary to some of the other approaches mentioned in this thread, which, BTW, is interesting.

Re: The tragedy of 100% code coverage (2016)

#303

Earlier quoted context omitted.

I think there's an argument to be made that if the desire for testing is a main driver for your architecture then your tests are too granular, and you aren't testing any internal integration points. In my experience that means that your tests are so tied to your current architecture that you can't even refactor without having to rewrite tests.

mocking / interaction / expect breaks encapsulation to perform "testing". Thus it is often a test of the implementation's assumptions when first written, and even worse, when the code is maintained/edited, the test is merely changed to get it to pass, because unit tests with mocks are usually: 1) fragile to implementation 2) opaque as to intent Whereas input/output integration points are more reliable, transparent, a…

Exactly. I've seen tests where some code calls:

  printf("hello world");
And the test is:

  mock_printf(string) {
    if string != "hello world" then fail;
  }
Which is basically just duplicating your code as tests.

Re: The tragedy of 100% code coverage (2016)

#304
post #265
post #221

Earlier quoted context omitted.

You can test stuff manually.

You really shouldn't, or if you do it should be an extreme minority of tests and automated tests should do the heavy lifting. Automated tests can do so much more than manual tests that shops still living by manual tests either have a damn good reason or are just as wrong as people who argued against Revision Control Systems (or people who argued for gotos instead of functions). Automated tests will be executed identi…

Maybe you're thinking about some specific applications/pieces of code? Probably not a video game?

I think it should be a mix. I disagree with the "extreme minority" portion for most projects. There are times where a manual test is the right answer and there's exploratory manual testing.

Obviously though running through a long sequence of testing manually for every code change is crazy. And some sort of CI setup like you describe is a must for every project this day and age.

Then there's also the question of unit tests vs. system/integration tests...

Re: The tragedy of 100% code coverage (2016)

#305
post #275
post #178

Earlier quoted context omitted.

By far the best codebase I ever worked with had zero test cases, was nearly 30 years olds, been worked on by not just several teams but several companies, had basically zero documentation, and was still easy to read and understand. Unit tests only really catch a tiny fraction of bugs relating to defective code, not poor design, poor market fit, poor understanding of requirements, or any of the other bugs that most of…

Survivorship bias. What about all the shitty 30 year code bases with no tests and no documentation that got dropped because they were garbage. It's not impossible to write good code without tests, it is just more likely that code with tests and docs will be good enough to survive 30 years. Your project with no tests or docs and was good might be 1 in a million, while software that is good with tests and docs might be…

The test that are written today will not cover future cases 30 years down the road. Code bases 30 years ago are not the same shitty code bases you see today. Programming was more difficult and required better programmers who wrote better code on average. TDD is needed today because the bar has been lowered and it acts as a safety net.

Re: The tragedy of 100% code coverage (2016)

#306
post #71

Earlier quoted context omitted.

I'd recommend aiming for 10-20% on those projects, and also for startups trying to rapidly push an MVP out. Tests have diminishing returns. You want to hit the absolute most crucial ones that give you plenty of bang for buck and even save you time. That means finding the (usually small handful) of functions that implement your most crucial and most complicated business logic, and writing tests for them. Anything past…

How do you know what 10-20% to test? This sort of basic level of decision-making for testing is something I wish I had, but all the tutorials and guides are about 100% code-coverage TDD so it's hard to find a path to to learn reasonable, high ROI testing.

Since what to test is a sliding scale, determining when and what to test is perhaps something that comes with exposure to good testers..

For me, a few references I use are "tests are a thinking help for specifying behavior", "tests hold behavior in place" and "test until you feel comfortable".

The first is a note that TDD thoughts are written by developers that write APIs, not by developers that write applications. TDD is a fantastic tool for an API designer, because they force you to think about the experience of using the API. So, whenever I design APIs, I like TDD. This is also a good argument for why you should minimize setup code that "goes behind the scenes" - if you're testing, say, a REST API, do as much of the setup and assertions as you can via the REST API as well!

The second helps me remember to think "a year from now, are all the behaviors this code needs to have obvious, or is someone likely to unintentionally break it?". I try to write tests that will flag if someone broke an important edge case - or the main use case! Tests can be used as the programmers equivalent of a carpenters' clamps, kind of.

The third is why I don't write as many tests anymore. I normally try to write one workflow-oriented feature test up front, like "When a user creates a new invoice, then that invoice should show up in the users list of invoices with the values they entered, plus x,y,z auto-generated parameters". As I implement the feature, if I come across a piece of logic that makes me feel uncomfortable - lots of branching, or code that's very important that it stays intact - I'll write a unit test or two to hold that in place; sometimes I don't write any unit tests, meaning I'd have written just one or two tests over the course of two or three days of implementation.

Re: The tragedy of 100% code coverage (2016)

#307

The worse the developer, the more tests he'll write. Instead of writing clean code that makes sense and is easy to reason about, he will write long-winded, poorly abstracted, weird code that is prone to breaking without an extensive "test suite" to hold the madness together and god forbid raise an alert when some unexpected file over here breaks a function over there. Tests will be poorly written, pointless, and give…

> The worse the developer, the more tests he'll write. This is obviously wrong. > Tests will be poorly written, pointless, and give an overall false sense of security to the next sap who breaths a sigh of relief when "nothing is broken". Of course, that house of cards will come down the first time something is in fact broken. That's a generalization that does not apply everywhere.

It's rather apparent to me anyway, what the OP meant. If I see someone expending an enormous effort writing tests, I'd probably conclude the same, unless I had knowledge of their expertise, in which case, I'd respect their judgement. The OP meant to draw a line which in their mind, distinguishes an expert from a novice. I don't see why so many people are getting all bent out of shape for.

>This is obviously wrong.

What makes it obviously wrong? It seems to be a personal opinion.

>That's a generalization that does not apply everywhere.

Does a comment necessarily have to contain statements that apply to every single person in every single situation at every point in time? I think we'd have to delete pretty much all comments on this website then.

Re: The tragedy of 100% code coverage (2016)

#308
post #302

There are a few relevant facts that should be known to everyone (including managers) involved in software development, but which probably are not: 1) 100% path coverage is not even close to exhaustively checking the full set of states and state transitions of any usefully large program. 2) If, furthermore, you have concurrency, the possible interleavings of thread execution blow up the already-huge number of cases fr…

Partly related: Is equivalence partitioning [1] used much in the industry in software testing? [1] https://en.wikipedia.org/wiki/Equivalence_partitioning I came across the concept some years ago, IIRC, in the classic book The Art of Software Testing by Glenford Myers [2], and used it in a few projects. But have not really heard or read of it being used much, from people or on forums. [2] https://en.wikipedia.org/wiki…

In the form discussed in [1], it gets pretty complicated when more than one operand is a variable, and they are derived from prior operations on the original data.

Alternatively, identifying equivalence partitions from the semantics of the input data faces the problem that a faulty program may create invalid partitions, so some test cases for a given equivalence partition pass, while others fail.

Re: The tragedy of 100% code coverage (2016)

#309

Earlier quoted context omitted.

I use pitest to run mutation coverage on most of my Java code bases. pitest implements a timeout to check for infinite loops introduced by changing the code. http://pitest.org/faq/

Pitest has solved the halting problem?

In the real world, you can't wait five minutes for the server to return a response. If the server returns the correct response after ten minutes, it's still wrong, unless the programmer has explicitly acknowledged that the procedure in question is a long-running procedure and has lengthened (but not eliminated) the time-out accordingly.

The halting problem is an irrelevant, Ivory-tower distraction in production code.

Re: The tragedy of 100% code coverage (2016)

#310
post #193

Earlier quoted context omitted.

No, inexperienced developers need dogmatisms. Subtle difference perhaps, but it's worth remembering that every competent developer started out being inexperienced, and that it's nothing to be ashamed of.

No. We fetishize experience too much. I've known a great many competent inexperienced developers who could weigh tradeoffs and use the right approach to a given task, and a far too many highly experienced developers who destroyed productivity through dogmatism.

Productivity is not the sole measure of competency, nor necessarily even a very good one. Productivity suffers when time is spent making software robust, secure, and performant, and the reverse is obviously also true. Likewise, short term productivity sometimes suffers when effort is spent to ensure long term productivity. Again the reverse also holds true. The problem is that short term productivity is often the only measure of competency that is immediately visible. Nobody gets recognized for the problems they don't cause.
Post reply on HN