Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The tragedy of 100% code coverage (2016)

labs.ig.com

Post reply on HN
Print view View original

291–300 of 346 posts

Re: The tragedy of 100% code coverage (2016)

#291

Post by BeetleB »

rothron wrote: post #158

I don't think I know anyone that do TDD. Uncle Bob has indoctrinated a few zealots into that mindset, but it all comes off as crazy to me. A germ of a good idea taken way too far. People of that school tend to write tests that test implementation rather than functionality. As a result you get fragile tests that break not telling you what went wrong but how the implementation has changed. Good tests should test behavi…

>Uncle Bob has indoctrinated a few zealots into that mindset

Even though Uncle Bob's advice agrees with this submission:

https://news.ycombinator.com/item?id=14301466

>People of that school tend to write tests that test implementation rather than functionality. As a result you get fragile tests that break not telling you what went wrong but how the implementation has changed.

Also another thing Uncle Bob does not advocate. From his stance, if most of your failing tests are due to refactors (changing implementation), and not bugs, you need to redesign your tests. Abstract out your interfaces so tests are not too dependent on the actual implementation. Tests are first class code, not to be treated differently from your main code base - they should abide by code standards, and they need to be architected as much as your regular code.

(Easier said than done).

The thing is, I'm not even an Uncle Bob fan. But it's crazy how neither his fans nor his detractors see the nuances in what he says - be it his videos or his blog posts.

What is it about programmer types that makes them see everything in binary?

Re: The tragedy of 100% code coverage (2016)

#292

Post by realusername »

vincnetas wrote: post #47
cbanek wrote: post #36

Earlier quoted context omitted.

Indeed. It's almost as if testing should be built into the language itself. (I'm always a big fan of including self-test code in projects)

What are the languages that have testing not as afterthought?

Definitely Ruby in general, testing is a core of the language culture.

Re: The tragedy of 100% code coverage (2016)

#293

Post by candiodari »

cowardlydragon wrote: post #270
mannykannot wrote: post #253

There are a few relevant facts that should be known to everyone (including managers) involved in software development, but which probably are not: 1) 100% path coverage is not even close to exhaustively checking the full set of states and state transitions of any usefully large program. 2) If, furthermore, you have concurrency, the possible interleavings of thread execution blow up the already-huge number of cases fr…

Also: the fundamental undecidability of the Halting Problem

If you actually know what the halting problem is, you know that if your code triggers that, it is pretty much broken. If you're not writing a programming language.

Re: The tragedy of 100% code coverage (2016)

#294

Post by dom0 »

cowardlydragon wrote: post #270
mannykannot wrote: post #253

There are a few relevant facts that should be known to everyone (including managers) involved in software development, but which probably are not: 1) 100% path coverage is not even close to exhaustively checking the full set of states and state transitions of any usefully large program. 2) If, furthermore, you have concurrency, the possible interleavings of thread execution blow up the already-huge number of cases fr…

Also: the fundamental undecidability of the Halting Problem

... is really not relevant to software testing.

Re: The tragedy of 100% code coverage (2016)

#295

Post by ska »

threatofrain wrote: post #264
ska wrote: post #234

Earlier quoted context omitted.

We are talking about software testing here. To a first approximation, nobody is being empirical, and there is no science being done.

When you are writing tests, that's an empirical, rather than a theoretical approach to software correctness. When programmers change a factor to "see what breaks", that is very much an empirical activity, and it is part of the programmer's theory-building of a phenomena. If a young child takes a gear from a watch and observes its breaking, that is very much an empirical activity. It is also the beginning of theory-bu…

I should have been more clear, I see now my wording was ambiguous.

Approximately nobody is being empirical about what works well and does not in testing. Use this approach vs. that approach, do this, no - do this instead. It's all largely heuristic.

Individuals performing testing and debugging are usually at least most empirical, I agree (although occasionally the rubber chickens come out)

Re: The tragedy of 100% code coverage (2016)

#296

Post by knodi123 »

I recently broke a unit test by adding one entry to a hash constant (a list of acceptable mime types and their corresponding file extensions). I looked at the test, and it was just comparing the defined constant, to a hardcoded version of itself.

I rewrote the test by converting the constant to a string, taking a checksum of it, and comparing _that_ to a short hardcoded value. Now the test is just 1 line of code, instead of 41! Then I put it through code review, and my team said "What a ridiculous test." But they didn't see any problem in the previous version that compared it to a 40-line hardcoded hash.

It's a weird world.

Re: The tragedy of 100% code coverage (2016)

#297

Post by sqeaky »

sidlls wrote: post #247
scottschulthess wrote: post #140

Earlier quoted context omitted.

So the worst of us developers out there write tons of tests? We wish. Who cares if it's 80% or 100%, i'd rather inherit a codebase with lots of test coverage than without.

And what about test coverage for the tests? You do realize that tests are also code, right? They have bugs, maintenance overhead, and all the other things code has. The worst code bases I've seen have more test code than code being tested. People try to write tests to cover every condition or input and in systems of even modest complexity that isn't possible, or at least not feasible in a finite time.

The actual code is the check on the tests. They cover each other, acting as though they are and there is some infinite recursion dishonest.

Tests do have a maintenance cost, but it is much lower than the maintenance cost of code without tests. Without manual tests if there a bug it must be caught manually, this costs human time every time the tests are run. That also presumes the humans do the tests correctly, have a new team, QA person out sick, what is the cost now?

The cost of not having tests is bugs in production. If I write code and there a bug that impacts production many millions of dollars are on the line, and this is true for many developers. Write software for any airplane? Crashes can causes crashes so they are dangerous and expensive. How about something more common, Write software software for an online store? If the shopping cart drops 1 in 1000 items that is a huge amount of money, not just because of lost sales, but also angry customers. You will want to run the tests at least a few thousands times to catch that, a human won't do this, an automated unit test suite is shell script and VM away from doing that.

Automated tests don't make this impossible but it reduces the amount of bugs that can make it through. If a test costs 10 hours of developer time at $500/hour and stop one bug that would have alienated 1% of customers. Then there would only need to be 500,000 customers worth $1 each. Clearly these numbers have insanely inflated costs, yet they still make sense for many businesses. And this is only accounting for one bug. Finding that bug did not consume the test, it is still ready for more. A good test can many bugs and last many years.

Re: The tragedy of 100% code coverage (2016)

#298

Post by jcadam »

graphitezepp wrote: post #288
jcadam wrote: post #202

Earlier quoted context omitted.

I worked on one such system (written in Ada, of all things) back when I was a fairly junior developer. I was hired to work on a major effort to port the system from SPARC/Solaris to RHEL/x86 and add some new features. It was designed by a crusty (and eccentric) old architect (who was still around to manage this major refresh project) who ruled with an iron fist, and the project was managed in classic waterfall fashio…

That would be an ... interesting work environment. Probably the dev version of benevolent dictator, works great until the leader leaves, and the team is left with weaker control structures.

Yea, it was only a 9 month contract for me. I was definitely glad not to be a FT employee at that place :)

Re: The tragedy of 100% code coverage (2016)

#299

Post by V-2 »

cbanek wrote: post #36
sbov wrote: post #32

Earlier quoted context omitted.

It's strange to me that we continue to make languages that force us to make certain architectural choices to help facilitate testing.

Indeed. It's almost as if testing should be built into the language itself. (I'm always a big fan of including self-test code in projects)

http://cobra-language.com/ had it.

Re: The tragedy of 100% code coverage (2016)

#300

Post by sidlls »

sqeaky wrote: post #297
sidlls wrote: post #247

Earlier quoted context omitted.

And what about test coverage for the tests? You do realize that tests are also code, right? They have bugs, maintenance overhead, and all the other things code has. The worst code bases I've seen have more test code than code being tested. People try to write tests to cover every condition or input and in systems of even modest complexity that isn't possible, or at least not feasible in a finite time.

The actual code is the check on the tests. They cover each other, acting as though they are and there is some infinite recursion dishonest. Tests do have a maintenance cost, but it is much lower than the maintenance cost of code without tests. Without manual tests if there a bug it must be caught manually, this costs human time every time the tests are run. That also presumes the humans do the tests correctly, have a…

I don't disagree that tests are good and necessary. I take issue with the view that tests are somehow free, that more testing is better, or that tests are of such marginal cost that more effort developing tests than the product is considered a reasonable use of resources.

Also, the notion that "they test each other" is likely to be dangerous.

Post reply on HN