I think this can be generalised into saying that the purpose of tests is to fail. I've seen far too many tests that are written to pass. You need to write tests to fail.
The purpose of continuous integration is to fail
21–30 of 52 posts
Re: The purpose of continuous integration is to fail
#22> Whenever a CI run fails, we can re-run it. If it passes the second time, we are sure it was flaky. Or you have a concurrency issue in your production code?
I thought that line was kind of funny: When a CI run fails, you don't rerun it and wait for the result, you rerun it and check why the original run failed in the meantime. Is it flaky? Is it a pipeline issue? Connectivity issue? Did some Key expire?
If you just rerun and don't go to find out what exactly caused CI to fail, you end up at the author's conclusion:
> (but it could also just have been flaky again).
Re: The purpose of continuous integration is to fail
#23[dead]
Re: The purpose of continuous integration is to fail
#24Earlier quoted context omitted.
> IMO, good tests are relatively immutable. You should be able to have multiple valid implementations. You should add new tests to describe the new functionality of that implementation, however, the old tests should remain relatively untouched. Taken to extreme this would mean getting rid of unit tests altogether in favor of functional and/or end-to-end testing. Which is... a strategy. I don't know if it is a good or…
> Taken to extreme this would mean getting rid of unit tests all together in favor of functional and/or end-to-end testing. The dirty little secret in CS is that unit, functional, and end-to-end tests are all the exact same thing. Watch next time someone tries to come up with definitions to separate them and you'll soon notice that they didn't actually find a difference or they invent some kind of imagined way of tes…
Yep, the 'unit' is size in which one chooses to use. The exact same thing happens when trying to discuss micro services v monolith.
Really it all comes down to agreeing to what terms mean within the context of a conversation. Unit, functional, and end-to-end are all weasel words, unless defined concretely, and should raise an eyebrow when someone uses them.
Re: The purpose of continuous integration is to fail
#25Earlier quoted context omitted.
> IMO, good tests are relatively immutable. You should be able to have multiple valid implementations. You should add new tests to describe the new functionality of that implementation, however, the old tests should remain relatively untouched. Taken to extreme this would mean getting rid of unit tests altogether in favor of functional and/or end-to-end testing. Which is... a strategy. I don't know if it is a good or…
> Taken to extreme this would mean getting rid of unit tests all together in favor of functional and/or end-to-end testing. The dirty little secret in CS is that unit, functional, and end-to-end tests are all the exact same thing. Watch next time someone tries to come up with definitions to separate them and you'll soon notice that they didn't actually find a difference or they invent some kind of imagined way of tes…
> The dirty little secret in CS is that unit, functional, and end-to-end tests are all the exact same thing.
I agree that the boundaries may be blurred in practice, but I still think that there is distinction.
> visible, public interface
Visible to whom? A class can have public methods available to other classes, a module can have public members available to other modules, a service can have public API that other services can call through network etc
I think that the difference is the level of abstraction we operate on:
unit -> functional -> integration -> e2e
Unit is the lowest level of abstraction and e2e is the highest.
Re: The purpose of continuous integration is to fail
#26Earlier quoted context omitted.
> Taken to extreme this would mean getting rid of unit tests all together in favor of functional and/or end-to-end testing. The dirty little secret in CS is that unit, functional, and end-to-end tests are all the exact same thing. Watch next time someone tries to come up with definitions to separate them and you'll soon notice that they didn't actually find a difference or they invent some kind of imagined way of tes…
> The dirty little secret in CS is that unit, functional, and end-to-end tests are all the exact same thing. I agree that the boundaries may be blurred in practice, but I still think that there is distinction. > visible, public interface Visible to whom? A class can have public methods available to other classes, a module can have public members available to other modules, a service can have public API that other ser…
> Visible to whom?
The user. Your tests are your contract with the user. Any time there is a user, you need to establish the contract with the user so that it is clear to all parties what is provided and what will not randomly change in the future. This is what testing is for.
Yes, that does mean any of classes, network services, graphical user interfaces, etc. All of those things can have users.
> Unit is the lowest level of abstraction and e2e is the highest.
There is only one 'abstraction' that I can see: Feed inputs and evaluate outputs. How does that turn into higher or lower levels?
Re: The purpose of continuous integration is to fail
#27Earlier quoted context omitted.
This ultimately is what shapes my view of what a good test is vs a bad test. An issue I have with a lot of unit tests is they are too strongly coupled to the implementation. What that means is any change to the implementation ultimately means you have to change tests. IMO, good tests are relatively immutable. You should be able to have multiple valid implementations. You should add new tests to describe the new funct…
It took me a bit of time (and two or three different view) to finally get this. That is mostly why I hardcode my values in the tests. Make them simpler. If something fails, either the values are wrong or the algorithm of the implementation is wrong.
Comparing actual outputs against expected ones is the ideal situation, IMHO. My own preference is for property-checking; but hard-coding a few well-chosen values is also fine.
That's made easier when writing (mostly) pure code, since the output is all we have (we're not mutating anything, or triggering other processes, etc. that would need extra checking).
I also think it's important to make sure we're checking the values we actually care about; since those might not be the literal return value of the "function under test". For example, if we're testing that some function correctly populates a table cell, I would avoid comparing the function's result against a hard-coded table, since that's prone to change over time in ways that are irrelevant. Instead, I would compare that cell of the result against a hard-coded value. (Rather than thinking about the individual values, I like to think of such assertions as relating one piece of code to another, e.g. that the "get_total" function is related to the "populate_total" function, in this way...).
The reason I find this important, is that breaking a test requires us to figure out what it's actually trying to test, and hence whether it should have broken or not; i.e. is it a useful signal that requires us to change our approach (the table should look like that!), or is it noise that needs its incidental details updated (all those other bits don't matter!). That can be hard to work out many years after the test was written!
Re: The purpose of continuous integration is to fail
#28[flagged]
Re: The purpose of continuous integration is to fail
#29The premise of the article has some weight, but the final conclusion with the suggestion to change the icons seems completely crazy. Green meaning "to the best of our knowledge, everything is good with the software" is well understood. Using green to mean "we know that this doesn't work at all" is incredibly poor UI (EDITED from "beyond idiotic" due to feedback, my bad). And whilst flaky tests are the most problemati…
Hey, author here: I completely agree, that's why I also haven't used those strange colours for https://nix-ci.com.
I just thought they would make for a cool visual representation of the point of the blog post.
Re: The purpose of continuous integration is to fail
#30> Whenever a CI run fails, we can re-run it. If it passes the second time, we are sure it was flaky. Or you have a concurrency issue in your production code?
Then the test is still flaky. If there's a bug you want the test to consistently fail, not just sometimes.