Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Software testing, and why I'm unhappy about it

nhaehnle.blogspot.com

Post reply on HN
Print view View original

61–70 of 76 posts

Re: Software testing, and why I'm unhappy about it

#61

Post by kapep »

vincnetas wrote: post #26
randomdata wrote: post #20

Earlier quoted context omitted.

Which is funny as the purpose of testing is to explain to other other developers what the code under test assumes and what should be expected of it under various conditions. It is documentation. If you have to document your documentation, you might be missing something fundamental in how you are writing your first order documentation. Not to mention that in doing so you defeat the reason for writing your documentatio…

So i understand correctly, that your position is "code is the documentation"? Over time im inclined to value human written documentation. Especially when things involve integrations of multiple systems. I had real cases, where two parties point at code and say their code is correct. And in isolation code looks correct. But when time comes to integrate these systems. It breaks. And then if you have human readable docu…

I think they mean "test code is documentation". For example if there's a unit test that expects an error for a certain input, it serves as documentation that this kind of input is not allowed.

It not always feasible to document every little edge case in natural language and keep it in sync with your code. If you "document" edge cases as tests, they _have_ to be in sync with your code. It shouldn't replace traditional documentation though and is better suited for internal components and not for public API.

Re: Software testing, and why I'm unhappy about it

#63

Post by chriswarbo »

13of40 wrote: post #17

I have a lot of bitter things to say about automated testing, having spent 14 years of my life trying to knead it into a legitimate profession, but here's the most significant: You test case is more useless than a turd in the middle of the dining room table unless you put a comment in front of it that explains what it assumes, what it attempts, and what you expect to happen as a result. Because if you just throw in s…

I agree. One nice feature of property-driven testing is that assumptions often end up causing test failures. For example (in ScalaTest):

  "Average of list" should "be within range" in {
    forAll() {
      (l: List[Float]) => {
        val avg = l.average
        assert(avg >= l.min && avg
This test will fail, since it doesn't hold for e.g. empty lists. Requiring non-empty lists will still fail, if we have awkward values like NaNs, etc. The following version has a better chance of passing:

  "Average of list" should "be within range" in {
    forAll() {
      (raw: List[Float]) => {
        val l = raw.filter(n => !n.isNaN && !n.isInfinite)
        whenever (l.nonEmpty) {
          val avg = l.average
          assert(avg >= l.min && avg
Getting this test to pass required us to make those assumptions explicit. Of course, it doesn't spot everything; here's an article which explores this example in more depth (in Python) https://hypothesis.works/articles/calculating-the-mean

Re: Software testing, and why I'm unhappy about it

#64

Post by sirsinsalot »

BlackFly wrote: post #60
sirsinsalot wrote: post #45

Earlier quoted context omitted.

Unit tests assert implementation behaviour to aid refactoring. If developers misunderstand the spec, the unit tests can be valid. They don't assert developer understanding. Say it with me, unit tests are to aid refactoring. If we mix QA and implementation details just because both sides use the word "test" it ends in trouble. QA should be blind to unit test coverage or even usage at all, they're totally independent c…

Unit tests (class or method as the unit) hinder refactoring by binding to low level implementation details. When you refactor, by definition you are changing what the factors (units) are. Generally, your unit tests will then be testing implementation details that no longer exist. By strongly coypling to implementation details, unit test suites suffer an extremely large ripple effect on refactoring. Tests in general c…

If a unit test covers function f(a,b) to ensure it always returns the right answer in the domain of a,b

But a developer looks at f(a,b) and realises a new implementation could be 10x faster.

The developer re-writes the function, the tests still pass. Without that test they couldn't be sure their rewrite didn't break the expected behaviour.

What you're talking about is changing interfaces and structure when refactoring. Yes, unit tests can make that more painful. But you bin your old tests and write new ones.

If your unit tests are not cheap to dispose of, run or rewrite ... that's your core problem.

If your unit tests are not testing discrete units, instead testing the combined behaviour of many units (the re-factoring of structure pain) ... then you have bigger issues than the tests; namely understanding the difference and applicability between unit and integration tests.

Re: Software testing, and why I'm unhappy about it

#65

Post by geraldwhen »

midasz wrote: post #38
13of40 wrote: post #17

I have a lot of bitter things to say about automated testing, having spent 14 years of my life trying to knead it into a legitimate profession, but here's the most significant: You test case is more useless than a turd in the middle of the dining room table unless you put a comment in front of it that explains what it assumes, what it attempts, and what you expect to happen as a result. Because if you just throw in s…

I always use (if the scenario is simple enough, which most are): @Test public void myTestMethod_Scenario_ShouldReturnThis() {....

Jest makes this far more straightforward.

It(“throws when the object belongs to another user”)

It(“does a business thing when thing is in state BLAH”)

Re: Software testing, and why I'm unhappy about it

#66

Post by randomdata »

vincnetas wrote: post #26
randomdata wrote: post #20

Earlier quoted context omitted.

Which is funny as the purpose of testing is to explain to other other developers what the code under test assumes and what should be expected of it under various conditions. It is documentation. If you have to document your documentation, you might be missing something fundamental in how you are writing your first order documentation. Not to mention that in doing so you defeat the reason for writing your documentatio…

So i understand correctly, that your position is "code is the documentation"? Over time im inclined to value human written documentation. Especially when things involve integrations of multiple systems. I had real cases, where two parties point at code and say their code is correct. And in isolation code looks correct. But when time comes to integrate these systems. It breaks. And then if you have human readable docu…

No. Your documentation is your documentation. It documents your code. It is not your code.

If the documentation can also be interpreted by machine to validate what it claims is true you have a nice side benefit, but not the reason for writing your documentation.

Re: Software testing, and why I'm unhappy about it

#67

Post by randomdata »

charcircuit wrote: post #27
randomdata wrote: post #20

Earlier quoted context omitted.

Which is funny as the purpose of testing is to explain to other other developers what the code under test assumes and what should be expected of it under various conditions. It is documentation. If you have to document your documentation, you might be missing something fundamental in how you are writing your first order documentation. Not to mention that in doing so you defeat the reason for writing your documentatio…

>the purpose of testing is to explain to other other developers what the code under test assumes and what should be expected of it under various conditions No, the point of automated testing is to verify that what is under test behaves correctly and to be able to scale this verification cheaper than having humans do it. Documenting what it verifies and under what conditions is just a side effect.

That is a common falsehood. Testing does not verify that the code under test behaves correctly. It only verifies that the what the documentation asserts correctly matches what the code does. Indeed, enabling the machine to verify that the documentation is true is cheaper than having humans do it. Also less error prone. Humans are notoriously bad at keeping documentation properly up to date.

Re: Software testing, and why I'm unhappy about it

#68

Post by hbrn »

sirsinsalot wrote: post #64
BlackFly wrote: post #60

Earlier quoted context omitted.

Unit tests (class or method as the unit) hinder refactoring by binding to low level implementation details. When you refactor, by definition you are changing what the factors (units) are. Generally, your unit tests will then be testing implementation details that no longer exist. By strongly coypling to implementation details, unit test suites suffer an extremely large ripple effect on refactoring. Tests in general c…

If a unit test covers function f(a,b) to ensure it always returns the right answer in the domain of a,b But a developer looks at f(a,b) and realises a new implementation could be 10x faster. The developer re-writes the function, the tests still pass. Without that test they couldn't be sure their rewrite didn't break the expected behaviour. What you're talking about is changing interfaces and structure when refactorin…

> If your unit tests are not testing discrete units, instead testing the combined behaviour of many units

This is a very naive idea of unit tests. In the real world applications have a dependency hierarchy with more than two levels. If you want to test anything other than the leaves in that hierarchy you are by definition testing the behaviour of many units.

Sounds like what you're saying is that unit tests are only applicable to leaves in the dependency hierarchy. I could agree with that, but that's not what the world describes as unit tests. That would also make unit tests quite useless.

Re: Software testing, and why I'm unhappy about it

#69

Post by debug-desperado »

13of40 wrote: post #17

I have a lot of bitter things to say about automated testing, having spent 14 years of my life trying to knead it into a legitimate profession, but here's the most significant: You test case is more useless than a turd in the middle of the dining room table unless you put a comment in front of it that explains what it assumes, what it attempts, and what you expect to happen as a result. Because if you just throw in s…

So, "given when then" style tests (e.g. Spock) plus a descriptive test name. Or more than that?

Re: Software testing, and why I'm unhappy about it

#70

Post by eevilspock »

alpaca128 wrote: post #54
vincnetas wrote: post #26

Earlier quoted context omitted.

So i understand correctly, that your position is "code is the documentation"? Over time im inclined to value human written documentation. Especially when things involve integrations of multiple systems. I had real cases, where two parties point at code and say their code is correct. And in isolation code looks correct. But when time comes to integrate these systems. It breaks. And then if you have human readable docu…

Code as documentation feels like a good idea because code is the only reliable source of truth. But it also assumes that code can comprehensively express all assumptions and other info, which sounds more like wishful thinking. Auto-generated API docs combined with handwritten documentation that covers what can't be expressed in code and includes some useful examples seems like the right approach to me. In practice th…

I'm not sure if you're saying that rust stdlib docs do this but documentation where all the examples are themselves runnable as tests and included in the CI test suite solves so many problems.
Post reply on HN