Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Coverage is not strongly correlated with test suite effectiveness

neverworkintheory.org

Post reply on HN
Print view View original

141–150 of 178 posts

Re: Coverage is not strongly correlated with test suite effectiveness

#141

Post by 4140tm »

optymizer wrote: post #118
jacobsenscott wrote: post #20

It is easy to write a test that executes code without actually testing anything. I use coverage to find code with no tests all at, and write tests for that code. But once it is "covered" the coverage report is useless. In interpreted languages (ruby/python/etc) coverage at least tells you if there's a syntax error before running it in production, which is useful. Test first also improves the quality of the tests just…

I found TDD useful for a well defined problem or an agreed-upon API. For apps for example, especially those not well defined and designed as-you-go, where the designer and PM might change their minds frequently after toying around with the app or getting user feedback, TDD is a lot of overhead and tests after writing the code are primarily useful for preventing regressions when somebody else changes your code.

You don't need precise specs to practice TDD. If you have idea of the code you need to write, you can write the test for it beforehand - no matter how often someone might change their mind about the app.

Doing so would actually make your life a lot easier when it's time to alter functionality, because now you have well tested and testable code. Code that is written to be tested is usually a lot easier to reason about, to change and extend.

If you do it in concise manner and test behavior rather than implementation, what you previously thought of overhead will actually speed you up.

Re: Coverage is not strongly correlated with test suite effectiveness

#142

Post by chriswarbo »

hsn915 wrote: post #103
bronson wrote: post #87

Earlier quoted context omitted.

3. Your code base may start to become contorted. I've seen good programmers create bogus classes to allow test-time mocking, or add oddball env vars and configurations to let the test harness manually reach every last line. Even if that line is not worth testing: if(!(x=malloc(BUF_SIZ)) || ENV[TEST_MEM_FAIL_12]) { exit(1); } Tying code and tests this tightly discourages refactoring. Another example: a different code…

"Mocking" is such a weird thing to me and I don't think it serves a good purpose. It's the kind of thing that would only arise if you assume a-priori that 100% test coverage is a non-negotiable must. If you have a function A that calls B to get some data (by doing I/O) then process it using C, then the 100% cov rule would force you to mock B when you test A. But then what is the value of this test? What guarantees is…

Mocking and coverage aren't that closely related, but I agree that mocking makes tests less realistic, and therefore less useful.

Mocking can be helpful when that downside is still better than the alternative, e.g. if 'real' calls have significant latency, or a significant monetary cost, etc. In those cases mocking lets us do a whole lot more testing for the same amount of time/money/etc., which can make up for the loss of realism.

Mocking can be especially useful when adding tests to a legacy system. Greenfield work should aim to minimise the amount of mocking required, ideally to zero.

Re: Coverage is not strongly correlated with test suite effectiveness

#143

Post by brabel »

jacobsenscott wrote: post #134
optymizer wrote: post #118

Earlier quoted context omitted.

I found TDD useful for a well defined problem or an agreed-upon API. For apps for example, especially those not well defined and designed as-you-go, where the designer and PM might change their minds frequently after toying around with the app or getting user feedback, TDD is a lot of overhead and tests after writing the code are primarily useful for preventing regressions when somebody else changes your code.

I hear this a lot, but the result is usually an untested, and usually nearly untestable (because it was written without tests), prototype with a few characterization tests that gets pushed to production and haunts you for the rest of the life of the product. Pototyping to define the problem or API is fine, but most people don't have the discipline to tear it out and start over when they finally do have a well defined…

> the result is usually an untested, and usually nearly untestable

No way... if you ever wrote any tests, you can easily know how to write code that will be testable even if you do it later. Doing it before is just going to be a big waste of time if you throw the code away later, which happens a lot when you need to experiment with things before actually choosing what works best. Yes, you can do that with TDD as well but it will absolutely slow you down. If you end up not writing tests later, when the implementation has been chosen, it's just because you don't care about the code quality... I find it hard to believe TDD will fix you in that case.

Re: Coverage is not strongly correlated with test suite effectiveness

#145

Post by ParkerPierre »

satisfice wrote: post #71

Ever since I developed code coverage tools at Apple in 1989, and tested them for Borland in the early 90’s, I knew and have been telling people in MY conference slides that code coverage is a nearly useless metric. Anyone who thought critically about it for ten minutes knows it’s nonsense. The one thing code coverage tells you that is of any significant value is what you haven’t tested. You still know very little abo…

> Anyone who thought critically about it for ten minutes knows it’s nonsense

Can you explain a bit more? I have only done coverage on my home projects. I get 100% every time because otherwise why bother. I started experimenting with 100% branch coverage. I understand sometimes there's a few lines you can't test like if there's a fork() but in my cases I was lucky enough to not need it and didn't need to exclude anything (I was thinking about excluding an impossible to execute default case since I did val&7 and tested all 8 cases but I ended up using if statements instead for that one off case)

It seems to me as long as your testing your own code its great. It gets annoying when testing an API that you can't ask to fail. I never mock anything. It seems like coverage is a great solution

In one project I used fuzzing data and another I specified valid date and say anything invalid is not a bug and may report incorrect results (I think I detected most/all bad data and rejected it which people hated)

Re: Coverage is not strongly correlated with test suite effectiveness

#146

Post by brabel »

amw-zero wrote: post #52

You don’t even need to do a study to arrive at this conclusion. Has no one read Djikstra’s writings? https://www.cs.utexas.edu/users/EWD/ewd02xx/EWD249.PDF “Program testing can be used to show the presence of bugs, but never to show their absence!” Everybody in our industry wants to bash math and say that anyone can write programs, but programs are logical systems and can only be fully understood with math and logic.…

> We have the tools for understanding and reasoning about infinitely large structures, programmers just refuse to use them, and even deride them.

Could you point out which tools you're talking about?

Re: Coverage is not strongly correlated with test suite effectiveness

#147

Post by tpxl »

namelosw wrote: post #126

Testing is an art that takes quite a while to master. People are able to write tests but many don't know "how" to write tests. Testing is extremely valuable but at the same time very easy to get burned. When testing done wrong (like abusing tests for coverage): 1. It makes the code too inflexible to refactor, leaving the system too rigid to grow. 2. It makes people frustrated on how to write tests in this project. 3.…

> 1. Only test the behavior from the user/consumer's perspective, do not test implementations. (Therefore the internal can be refactored without deleting tests.)

Sometimes you still need to test implementations. We had database writers/readers that automatically retried on failure/disconnect/failover/whatever and you can't test those by testing the API.

Re: Coverage is not strongly correlated with test suite effectiveness

#148

Post by porker »

wonnage wrote: post #137

My personal experience working on frontend is that it's often fastest to get everything looking right in the browser, and then go back and write tests. There is a ton of manual testing involved (i.e, getting UI into all its possible states) that just goes out the window when writing unit tests because you can't easily test for stuff like the correct whitespace between elements. But then react-storybook and Happo came…

> But then react-storybook and Happo came along (I'm sure the concepts have existed long before). I can actually commit the states I was trying to test to the repo (as stories), and CI can render them and take screenshot diffs with Happo; these are now the unit tests. So now the work I'm doing during development doubles as the tests as well, and there's no second pass required.

Ohhh. Thanks, I didn't know this was possible (the committing of states, screenshots I've done before). I'm still in the hell of manual-testing the front end on multiple browsers and resolutions to check that changes don't break anything.

Re: Coverage is not strongly correlated with test suite effectiveness

#149

Post by berkes »

brabel wrote: post #143
jacobsenscott wrote: post #134

Earlier quoted context omitted.

I hear this a lot, but the result is usually an untested, and usually nearly untestable (because it was written without tests), prototype with a few characterization tests that gets pushed to production and haunts you for the rest of the life of the product. Pototyping to define the problem or API is fine, but most people don't have the discipline to tear it out and start over when they finally do have a well defined…

> the result is usually an untested, and usually nearly untestable No way... if you ever wrote any tests, you can easily know how to write code that will be testable even if you do it later. Doing it before is just going to be a big waste of time if you throw the code away later, which happens a lot when you need to experiment with things before actually choosing what works best. Yes, you can do that with TDD as well…

I've always experienced that writing the actual code is but a minor part of building features (or fixing bugs). Especially when prototyping, which most often is merely ducttaping libs together.

Everyone remembers a bugfix of one line, that took hours, or days to find.

I'd stringly encourage you to check your commitlogs. You'll probably find you commit at most hundreds of lines a day, quite probably, as is my case too, av era aging under ten lines a day.

Please stop thinking that writing code which does not make it into commits, is waste. It is learning. And tests are by far the best place to learn about your, and Libs' code, APIs and behaviour.

Re: Coverage is not strongly correlated with test suite effectiveness

#150

Post by berkes »

4140tm wrote: post #141
optymizer wrote: post #118

Earlier quoted context omitted.

I found TDD useful for a well defined problem or an agreed-upon API. For apps for example, especially those not well defined and designed as-you-go, where the designer and PM might change their minds frequently after toying around with the app or getting user feedback, TDD is a lot of overhead and tests after writing the code are primarily useful for preventing regressions when somebody else changes your code.

You don't need precise specs to practice TDD. If you have idea of the code you need to write, you can write the test for it beforehand - no matter how often someone might change their mind about the app. Doing so would actually make your life a lot easier when it's time to alter functionality, because now you have well tested and testable code. Code that is written to be tested is usually a lot easier to reason about…

Tests are also meant to be continously refactored, though.
Post reply on HN