And, if not for testing, how would the xz hack have been exposed?
Vast amounts of projects with tests >>> an occasional blog post with some quibble.
Unit Tests Considered Harmful
71–76 of 76 posts
Re: Unit Tests Considered Harmful
#72Enough with the hyperbole. The argument is that unit tests alone are insufficient to ensure your software is functional and to detect regressions, and likewise unit test coverage is not an incontrovertible measure of quality or reliability. We can just present that argument on its own merit without the hot take. My observations: - developers need a fast feedback loop. While you’re in the process of changing things, w…
Excellent point. I think you really get to the heart of the issue.
Different kinds of tests offer different levels of confidence at different levels of time and investment.
A fully integrated shop should think this through carefully and design their own comprehensive end-to-end process. Which feedback is most helpful at which point of the idea -> deployed product pipeline? What value does it offer to the larger process? Feed in org-specific variables like release timelines, team structure, customer expectations, money, compute resources. Use appropriate tools for each piece and remember the largest goals to keep the company reliably delivering whatever makes that company valuable.
Re: Unit Tests Considered Harmful
#73Earlier quoted context omitted.
>If you test to a function, or a class, then your tests imply that function or class must be present, with that specific API. So what? Every implementation will have some basic structure. That structure can be modified if needed. The point of a unit test is not to posit that any particular thing exists, but that the things that do exist actually work. >In my experience, most people write unit tests for internal imple…
> Every implementation will have some basic structure. Setting aside functional vs OO paradigms, even if you have a simple helper function like: def removeprefix(s, prefix): if s.startswith(prefix): return s[len(prefix):] return s do you write tests for it? Or do you write tests for the higher level routines which call it? I think most write tests for the function. If you write tests for the function then you hinder…
>do you write tests for it? Or do you write tests for the higher level routines which call it? I think most write tests for the function.
I try to not test things that are that simple. But I wouldn't fault anyone for writing a small number of test cases for it.
>For example, in red-green-refactor TDD, you are not supposed to change the tests when you refactor. > >What you've ended up with are tests for the specific structure, and not the general goals.
I'm not familiar with this methodology. But if your refactoring requires changing the interfaces then it must require changing the tests. It may be ideal to have such rigid interfaces in some cases, but I don't think you have to be so committed to one methodology. Big changes require big testing.
>If you write tests for the function then you hinder future refactoring from removeprefix(s, prefix) to s.removeprefix(prefix) once you switch to a Python which implements s.removeprefix (3.9, I think?)
How? Is the old code supposed to stop working because a new function was added? You aren't obligated to delete perfectly working and tested code. And if the new version of Python breaks it somehow, the tests will tell you quickly compared to finding out in a bigger test.
>Which is why I use coverage-based method to identify what need more tests. Coverage is a powerful tool, and easily misused.
Coverage is fine. But I mean, coverage is for people who already committed to testing every line of code. Not every project has unit tests in the first place so coverage is a moot point.
>The biggest problem is that you decide to not refactor because there are so many tests to delete, and you have to figure out if the failing test really is okay to delete (because it was implementation specific) vs. one that needs to be updated (eg, because it handles a regression case).
This work is very easy compared to fixing the real product in most cases. If you didn't write a bunch of frivolous tests it isn't that much work.
>>Those types of tests are not unit tests > >Correct! And no one said they were.
Well this is a discussion about unit tests. If you want to talk about other types of tests you need to be clear about that.
>That's fine. You do you.
I was just pointing out that unit tests are not always feasible to do. But I wish they were. The stuff I see at work was not designed by me, so it's not my fault it is infeasible to test at least.
Re: Unit Tests Considered Harmful
#74Earlier quoted context omitted.
> Every implementation will have some basic structure. Setting aside functional vs OO paradigms, even if you have a simple helper function like: def removeprefix(s, prefix): if s.startswith(prefix): return s[len(prefix):] return s do you write tests for it? Or do you write tests for the higher level routines which call it? I think most write tests for the function. If you write tests for the function then you hinder…
>do you write tests for it? Or do you write tests for the higher level routines which call it? I think most write tests for the function. I try to not test things that are that simple. But I wouldn't fault anyone for writing a small number of test cases for it. >For example, in red-green-refactor TDD, you are not supposed to change the tests when you refactor. > >What you've ended up with are tests for the specific s…
> You aren't obligated to delete perfectly working and tested code.
You aren't obligated to write unit tests either.
If your goal includes long-term maintainability then should refactor for simplicity and consistency. Removing unneeded code and unneeded tests is a good thing.
> coverage is for people who already committed to testing every line of code.
Much as I would like it, I don't have 100% coverage. I use coverage to identify and prioritize which areas of code need more tests. Some are more important than others.
I also use it to identify code which is no longer needed, like workarounds for third-party-package V2021 which have been fixed in late releases, and I have no need to support V2021 any more.
> If you didn't write a bunch of frivolous tests it isn't that much work.
My observation is that people write a bunch of frivolous tests.
> If you want to talk about other types of tests you need to be clear about that.
I am talking about how to design unit tests.
Someone else here pointed to Robert Martin's essay on exactly this topic, at http://blog.cleancoder.com/uncle-bob/2017/10/03/TestContrava... .
"Design the structure of your [unit] tests to be contra-variant with the structure of your production code. ... The structure of your tests should not be a mirror of the structure of your code."
The issue I'm describing is what Martin characterizes as "the Fragile Test Problem" with some unit test approaches.
Re: Unit Tests Considered Harmful
#75Earlier quoted context omitted.
Very good advice. Additionally, I'd add: - What sort of testing can be achieved given limited time and resources? ...because nearly all organizations, no matter how quality-oriented they claim to be, will prioritize non-test code delivery over test code, leaving scarce time for test creation before shipment deadlines.
Not to pick on you specifically, but I tend to agree with other posters that testing (automated or otherwise) is just an element of programming. Like all elements it needs to be done to taste, but it's pretty essential. A line of questioning: Do you have time to write clear code? Time for comments? Time to manually test your changes hundreds of times? Time to refactor existing code when adding new code? Time to remov…
I fully agree with all your points. My comment was based in my observations that, even given the truth of everything you said, "time to prepare automated tests" is usually one of the first things to shrink as unexpected events cause projects to slip on their timelines.
That's not a good thing, but it is a real thing: healthy organizations should respond by protecting that time, extending deadlines, and assessing what about their process and environment is causing planned timelines to not match reality ... but that doesn't always happen. When it doesn't happen, testing discipline can slip from (for example) "we need unit tests for complex logic modules, and integration tests for real networked service interactions, and at least 70% coverage" to "just get the happy path tested however you can so this feature makes it into the next release".
Given the reality that this will sometimes occur, engineers should remember to always prioritize the highest-value (that is, the maxima between time spent creating tests and defects that those tests can catch) testing work and methodologies. In good conditions, that will result in them writing the most important tests first and then writing any other tests they feel they need. But in bad conditions, this approach will still ensure that the most important automated verification is present.
Re: Unit Tests Considered Harmful
#76Earlier quoted context omitted.
>do you write tests for it? Or do you write tests for the higher level routines which call it? I think most write tests for the function. I try to not test things that are that simple. But I wouldn't fault anyone for writing a small number of test cases for it. >For example, in red-green-refactor TDD, you are not supposed to change the tests when you refactor. > >What you've ended up with are tests for the specific s…
> You aren't obligated to delete perfectly working and tested code. You aren't obligated to write unit tests either. If your goal includes long-term maintainability then should refactor for simplicity and consistency. Removing unneeded code and unneeded tests is a good thing. > coverage is for people who already committed to testing every line of code. Much as I would like it, I don't have 100% coverage. I use covera…
>If your goal includes long-term maintainability then should refactor for simplicity and consistency. Removing unneeded code and unneeded tests is a good thing.
I agree with this in principle, but I think people disagree about what is needed. One person might look at a good set of tests and think it's a waste of time, but if it actually hits the right points then it's probably worth keeping. I think if unit tests are feasible and can hit a bunch of cases that would be hard to recreate otherwise, then unit tests are far superior.
>"Design the structure of your [unit] tests to be contra-variant with the structure of your production code. ... The structure of your tests should not be a mirror of the structure of your code."
That's a nice ideal. But as with the refactoring issue, it may be impossible to test a thing without recreating some of that structure. That's why we have mocks.