Live data from Hacker News

The tragedy of 100% code coverage (2016)

labs.ig.com

41–50 of 346 posts

Re: The tragedy of 100% code coverage (2016)

#41
post #38

Earlier quoted context omitted.

Curious as to why you're blaming a language (ruby) for a bad practice (writing layers of useless tests). Excuse my pedantry, but it wouldn't it be more accurate to blame a culture that you feel has grown up around that language?

In Scala, "does it compile/typecheck" will generally catch typos in code that isn't frequently executed, and raise warnings for unhandled cases. For Ruby, you pretty much need to execute the code to have any inkling if it goes kaboom -- the language (and how people write it) have so much magic that short of executing things, the computer can't tell you anything useful.

Valid point.

I don't think I could write low-bug count code faster in statically typed scala than I could in unit tested ruby though. I mean I am well aware of what you're telling me, and it's obvious to me that having the compiler automatically check certain properties is a win. And yet, when push comes to shove, to get something done I'm more likely to reach for ruby.

It's something I've never come up with a good explanation for. Does static typing stunt prototyping and exploration? Do unit tests capture high level goals better?

I guess I don't agree with the assertion that ruby tests are useless. Because you test higher level things than you do with scala types.

Re: The tragedy of 100% code coverage (2016)

#42
95% coverage is pretty good balance I find.

There's something magical about being pretty sure things still work after any kind of refactor or new requirement.

I have been in situations like the author presents... decided not to test it and lo and behold it breaks production and causes $10,000 worth of revenue loss.

Re: The tragedy of 100% code coverage (2016)

#43
post #18

I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases. Just because you ran a function or ran a line doesn't mean it will work for the range of inputs…

Which why I really love things like MSR's PEX[1].

"Pex uses a constraint solver to produce new test inputs which exercise different program behavior. "

So this is not lines, but I guess.. branches and numerical limits. Either way it's cool and I wish it was integrated somewhere. They've got a online demo thing http://www.pexforfun.com/

[1]https://www.microsoft.com/en-us/research/publication/pex-whi...

Re: The tragedy of 100% code coverage (2016)

#44
post #3

I have 100% code coverage on a couple of projects. It has two benefits: Behaviour is completely covered by tests, so changes in APIs which might break consumers of the library will at least be detected. New work on the library tends to follow the 100% coverage by convention, so it's somewhat easier to maintain. Apps that have 90% coverage, for example, tend to slip and slide around. Having 100% coverage projects the…

The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of security. So if I want to contribute to your project all I have to do is write some pointless tests that are sure to execute every single getter and setter method.(yes I have seen tests that exist solely to execute getters and setters). I don't have to actually t…

> a false sense of security

Yes. Code coverage should not be the primary measure of quality of your tests. That thinking leads to tests designed to cover lines, not use cases.

Re: The tragedy of 100% code coverage (2016)

#45

My main issue with unit testing is what defines a unit? Throughout my career I find tests that tests the very lowest implementation detail, like private helper methods, and even though a project can achieve 100% coverage it still is no help avoiding bugs or regression. Given a micro service architecture I now advocate treating each service as a black box and focus on writing tests for the boundaries of that box. That…

I agree with you. That's called functional testing, and it is very useful, but it is not unit testing.

Unit testing: Test all methods and paths of a class, even private ones.

Functional Testing: Test the public api of a class/service only. If something is wrong internally, it will be caught without having to write countless of little tests.

ROI of functional testing is high, as it is usually done with real data. In my opinion unit testing is a huge waste of time. Most of the tests devolve int mock objects, calling mock methods, and doing this that really don't help to find real world bugs, where two unit tests pass, but their methods produce the wrong output.

Re: The tragedy of 100% code coverage (2016)

#46
We've almost stopped unit testing. We still test functionality automatically before releasing anything into production, but we're not doing a unit test in most cases

Our productivity is way up and our failure rates haven't changed. It's increased our time spent debugging, but not by as much as we had estimated that it would.

I won't pretend that's a good decision for everyone. But I do think people take test-driven-development a little too religiously and often forget to ask themselves why they are writing a certain unit test.

I mean, before I was a manager I was a developer and I also went to a university where a professor once told me I had to unit test everything. But then, another professor told me to always use the singleton pattern. These days I view both statements as equally false.

Re: The tragedy of 100% code coverage (2016)

#47
post #36
post #32

Earlier quoted context omitted.

It's strange to me that we continue to make languages that force us to make certain architectural choices to help facilitate testing.

Indeed. It's almost as if testing should be built into the language itself. (I'm always a big fan of including self-test code in projects)

What are the languages that have testing not as afterthought?

Re: The tragedy of 100% code coverage (2016)

#48
post #43
post #18

I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases. Just because you ran a function or ran a line doesn't mean it will work for the range of inputs…

Which why I really love things like MSR's PEX[1]. "Pex uses a constraint solver to produce new test inputs which exercise different program behavior. " So this is not lines, but I guess.. branches and numerical limits. Either way it's cool and I wish it was integrated somewhere. They've got a online demo thing http://www.pexforfun.com/ [1] https://www.microsoft.com/en-us/research/publication/pex-whi...

IntelliTest[0] is the productized version of PEX. The first release was in Visual Studio 2015 Enterprise edition.

[0] https://blogs.msdn.microsoft.com/visualstudio/2015/09/30/int...

Re: The tragedy of 100% code coverage (2016)

#49
post #36
post #32

Earlier quoted context omitted.

It's strange to me that we continue to make languages that force us to make certain architectural choices to help facilitate testing.

Indeed. It's almost as if testing should be built into the language itself. (I'm always a big fan of including self-test code in projects)

> testing should be built into the language itself

I wonder what that would look like...

Re: The tragedy of 100% code coverage (2016)

#50

If you can prove that your testing process is perfect, then your entire development process can then be reduced to the following, after the test suite is written: cat /dev/random | ./build-inline.sh | ./test-inline.sh | tee ./src/blob.c && git commit -Am "I have no idea how this works, but I am certain that it works perfectly, see you all on Monday!" && git push production master --force When presented like this, rel…

It seems to me that such a "perfect" testing process would basically amount to declarative programming.
Post reply on HN