Coverage is not strongly correlated with test suite effectiveness • Hacker News Board

Re: Coverage is not strongly correlated with test suite effectiveness

#161

Post by gilbetron » Tue, Sep 28, 2021, 12:09 PM UTC

gilbetron wrote: post #75

Say we have an application covered by 1000 tests. We fast forward a few years and (amuse this crazy notion) the application is decommissioned. We look and see that 500 tests never failed; those 500 always passed. Did we waste dev time by writing those tests? It's an interesting question to think about.

It's also interesting to see the response from people when I even pose this question, or any other question that might even remotely cast doubt on the CI/TDD dogma. Another question is: how do we rectify the complexity added to the total body of our source code when we add in really complex tests? Is there a point at which we are adding more complexity and increasing the potential bug surface area?

Re: Coverage is not strongly correlated with test suite effectiveness

#162

Post by Supermancho » Tue, Sep 28, 2021, 2:19 PM UTC

throwawaylinux wrote: post #127

Supermancho wrote: post #111

Earlier quoted context omitted.

I wouldn't even write this way because the else confuses what's happening. void fn(int a, int b) { string ret = "not"; if (a == b) ret = "equal"; printf(ret); } *I understand, this is not a good optimization in every language.

How does the 'else' confuse what is happening?

The function will perform a printf, so nesting it in an if-else subconsciously implies it's optional.

This is a quirk of human text comprehension. In a small program, it seems arbitrary because you can see it all at once. Make the branches do more than simply printf and you can see why you want to frontload the else, rather than pick through the if/else to see what always happens.

Re: Coverage is not strongly correlated with test suite effectiveness

#163

Post by Supermancho » Tue, Sep 28, 2021, 2:24 PM UTC

nybble41 wrote: post #128

Supermancho wrote: post #111

Earlier quoted context omitted.

I wouldn't even write this way because the else confuses what's happening. void fn(int a, int b) { string ret = "not"; if (a == b) ret = "equal"; printf(ret); } *I understand, this is not a good optimization in every language.

IMHO the original version without the mutable local variable was much more obvious. In any language. Though I would also accept: void fn(int const a, int const b) { printf(a == b ? "equal" : "not"); } to avoid duplicating the call to printf(). If this were Rust I might suggest: fn theFn(a: isize, b: isize) { let ret; if a == b { ret = "equal"; } else { ret = "not"; } println!("{}", ret); } but only because Rust, unli…

> IMHO the original version without the mutable local variable was much more obvious. In any language.

This is not about obviousness. You can write the same expression lots of different ways, which are all obvious. This is still a question about testable paths and code hygiene.

Nesting the same function call in both branches of if/else unintentionally obscures that it's not a conditional printf call.

If you're going to write:

printf(a == b ? "equal" : "not");

you don't need the function at all, completely missing the point.

Your second example is definitely better in Rust.

Re: Coverage is not strongly correlated with test suite effectiveness

#164

Post by Supermancho » Tue, Sep 28, 2021, 2:31 PM UTC

amw-zero wrote: post #159

Supermancho wrote: post #94

Earlier quoted context omitted.

From the entry point, stub a function to throw an exception. You'll reach it. Unit testing isn't about finding every path. Nor is it about writing unit tests for every possible combination of values in a program. A java function that uses an int does not need to test -2147483648 to 2147483647 as inputs. That's not helpful.

That’s an interesting example, that’s exactly where you’re likely to find bugs - at boundary values. You should test those more than you should test the number 5. This is exactly why unit testing gives a false sense of security. You’ve done a lot, you’ve written all kinds of tests - but at the end of the day, you don’t get a proportionate amount of confidence about the code because there are infinite more cases that…

In Java, if my function doesn't modify the int, there's no reason to test the boundaries, other than 0 if the int is used in a calculation. The type system doesn't solely determine what tests to write. Tests are inferred from a combination of type system, timings, statements and variable usage.

Re: Coverage is not strongly correlated with test suite effectiveness

#165

Post by brabel » Tue, Sep 28, 2021, 2:48 PM UTC

amw-zero wrote: post #158

brabel wrote: post #146

Earlier quoted context omitted.

> We have the tools for understanding and reasoning about infinitely large structures, programmers just refuse to use them, and even deride them. Could you point out which tools you're talking about?

Well, by this I mean math (and mathematical logic). Math is the tool for reasoning about possibly infinite concepts, i.e. you can make a statement about infinite sets and still know if it’s true or not. We aren’t limited to what we can see and touch, which is good because any non-trivial software application is so large that it could never be drawn out like a building blueprint. It can only be described and reasoned…

I know about all the tools you've mentioned as I've been following the space from a distance, was just curious if you had a more "outside research" approach in mind when you claimed developers refuse to use the available "tools"... I haven't seen Coq, or even Idris or Ada+Spark, which arguably much closer-to-the-industry tools, being used for anything but very niche cases (like the case you mentioned about using TLA+ at Amazon).

I am probably one of the developers who would gladly use anything like that, but from my brief experiments and knowledge, they are very, very far from becoming usable in a general setting, so no, I don't refuse to use the tools, the tools are just not good enough to justify the high cost of using them yet.

Re: Coverage is not strongly correlated with test suite effectiveness

#166

Post by regularfry » Tue, Sep 28, 2021, 3:12 PM UTC

brabel wrote: post #157

regularfry wrote: post #152

Earlier quoted context omitted.

Swings and roundabouts: what you lose on writing tests you gain on not bothering to implement things you don't need. If the test passes, you stop. Without the tests to guide you it's very easy to waste time over-engineering, even if what you're building is well-structured.

I often write code without implementing anything, just the "surface API". That's when I find whether things will work or not. Tests are a hindrance to that. Once I figure out the design, then I will test all that I think is important. > what you lose on writing tests you gain on not bothering to implement things you don't need. I am not sure where to start... I've written so much code, applications, libraries, algori…

> I often write code without implementing anything, just the "surface API". That's when I find whether things will work or not.

There's an entire school of TDD which works this way. That's not incompatible at all.

> with TDD, I would've spent hours trying to get something working that later I would find, by exploration, that I didn't need at all.

If you're writing tests for something you don't need, the problem isn't with TDD as I understand it.

Re: Coverage is not strongly correlated with test suite effectiveness

#167

Post by nybble41 » Tue, Sep 28, 2021, 3:44 PM UTC

Supermancho wrote: post #163

nybble41 wrote: post #128

Earlier quoted context omitted.

IMHO the original version without the mutable local variable was much more obvious. In any language. Though I would also accept: void fn(int const a, int const b) { printf(a == b ? "equal" : "not"); } to avoid duplicating the call to printf(). If this were Rust I might suggest: fn theFn(a: isize, b: isize) { let ret; if a == b { ret = "equal"; } else { ret = "not"; } println!("{}", ret); } but only because Rust, unli…

> IMHO the original version without the mutable local variable was much more obvious. In any language. This is not about obviousness. You can write the same expression lots of different ways, which are all obvious. This is still a question about testable paths and code hygiene. Nesting the same function call in both branches of if/else unintentionally obscures that it's not a conditional printf call. If you're going…

> Nesting the same function call in both branches of if/else unintentionally obscures that it's not a conditional printf call.

And the version with mutation masks the fact that the initial value assigned to `ret` may or may not be used, which seems like a very similar problem to me. Plus it involves mutation, which IMHO significantly increases the complexity of any code. Unfortunately C offers limited options here; as cryptic and controversial as it may be, the ternary operator is the only (non-hackish) way to make an expression which selects between two or more values based on a boolean condition without either the possibility of failing to assign a value or requiring a "default" value which at least some paths through the function won't use. (What if there is no reasonable default value and all the required branches are expensive to compute?)

> If you're going to write: … printf(a == b ? "equal" : "not"); … you don't need the function at all, completely missing the point.

The function still encapsulates the specific comparison and the way the result is reported—exactly like the other versions—so I wouldn't say it's completely useless despite containing only one line of code. This is, of course, merely a toy example. You could assign the string to a named const variable if you wanted to make it slightly more self-documenting, though I question whether that would make this particular code any more readable.

Re: Coverage is not strongly correlated with test suite effectiveness

#168

Post by amw-zero » Tue, Sep 28, 2021, 6:31 PM UTC

brabel wrote: post #165

amw-zero wrote: post #158

Earlier quoted context omitted.

Well, by this I mean math (and mathematical logic). Math is the tool for reasoning about possibly infinite concepts, i.e. you can make a statement about infinite sets and still know if it’s true or not. We aren’t limited to what we can see and touch, which is good because any non-trivial software application is so large that it could never be drawn out like a building blueprint. It can only be described and reasoned…

I know about all the tools you've mentioned as I've been following the space from a distance, was just curious if you had a more "outside research" approach in mind when you claimed developers refuse to use the available "tools"... I haven't seen Coq, or even Idris or Ada+Spark, which arguably much closer-to-the-industry tools, being used for anything but very niche cases (like the case you mentioned about using TLA+…

It boils down to two camps: open to formal methods, and not open to formal methods. And, I would say the overwhelming majority of developers, especially here on HN, are in the "not open to formal methods" camp. I can find comments that justify that, there are also data points from speaking with coworkers and colleagues. So that is what I was talking about when saying that logs of people "refuse the tools." They just aren't even open to them to begin with.

You sound open to them, but dissatisfied with the tools themselves. I'm honestly in that camp too! We are not there yet in terms of being able to use this as the primary way that an entire application is developed. But, I believe that's the direction we should go in. You're right - we really need a tool that wins people over with the cost to value ratio. The promise of something far off in the future isn't enough for most people.

Re: Coverage is not strongly correlated with test suite effectiveness

#169

Post by tylerc230 » Tue, Sep 28, 2021, 11:02 PM UTC

I've found that architectures like redux which force the separation of logic and side effects (eg file writes or networking) allow me to write robust tests against the logic code without mocks or stubs. My approach is to write enough code that I have a good idea of what the interface to SUT should look like. Then I write the failing test and make it pass then write another test... The tests I write generally only change when the interface changes or when the requirements change. It doesn't make the code too rigid b/c I'm only writing tests against the interface of this big blob of code. I'm free to refactor away and my test will still run as long as I don't change the interface.

Re: Coverage is not strongly correlated with test suite effectiveness

#170

Post by throwawaylinux » Wed, Sep 29, 2021, 1:12 AM UTC

Supermancho wrote: post #162

throwawaylinux wrote: post #127

Earlier quoted context omitted.

How does the 'else' confuse what is happening?

The function will perform a printf, so nesting it in an if-else subconsciously implies it's optional. This is a quirk of human text comprehension. In a small program, it seems arbitrary because you can see it all at once. Make the branches do more than simply printf and you can see why you want to frontload the else, rather than pick through the if/else to see what always happens.

I can kind of see what you mean, but if you make branches do more then it's also quite possible the common actions become more tedious to factor into common code. And I don't think it's good practice to read control statements like that (obviously indented loops can't be assumed optional either).

  int pow2(int v, int pow)
  {
    int result;

    if (pow >= 0)
      result = v > -pow;

    return result;
  }

Looking at this, both sides of the if/else perform a multiply by 2^pow, you can't take the operation to possibly not happen.

So I agree factoring common operations can be a good thing, but I think it's really a case by case basis and my code is fine.

I would also caution against assuming something may not happen elsewhere in a function just because you see it inside a conditional block. This happens frequently to various degrees and I've never encountered a style policy preventing it or any compiler or linter warnings for it.