Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unit Tests Considered Harmful

shaiyallin.com

Post reply on HN
Print view View original

51–60 of 76 posts

Re: Unit Tests Considered Harmful

#51

Post by eesmith »

naasking wrote: post #44
eesmith wrote: post #39

Earlier quoted context omitted.

I'm pretty convinced that test-first is best seen as a negotiating technique to get the time to write the tests in the first place. "Okay, the code is done, time to write the tests" results in pressure to shorten testing, since if the golden path works then surely everything else works, right? Your management and your sales people then say "You're a great developer, it looks like it works, and we need to deploy that…

I can see that, but only because the dev has arguably already made the mistake of showing sales working features before tests are written. I've always been clear to say when something is a partial mockup to verify that I've understood what they're asking for.

Yes, but many places don't have a healthy work environment, and developers are generally younger, less confrontational, and less experienced at negotiating than management and sales.

Re: Unit Tests Considered Harmful

#52

Post by marcosdumay »

naasking wrote: post #16
potherca wrote: post #9

It keeps coming back to the same stupid erosion of paradigms. Once upon a time, some developers that were smarter than others started testing their code. They described ways to test smaller parts of large software systems separately before being integrated together. For instance "parameter testing" (to validate component subprograms against their specification) and "assembly testing" (for parts put together)[^1] As a…

Studies on TDD has failed to show a benefit over writing tests after the fact. The only factor that seems to matter is writing tests , the more tests the better. If you can write your code so it's amenable to more testing, or such that it requires fewer tests because it's simpler/has fewer cases to handle, great. If you have a framework that can generate tests (fuzzing, property-based testing), awesome.

Studies on software engineering are all low reliability ones full of confusing factors.

(What's the matter with SE as a discipline?)

Re: Unit Tests Considered Harmful

#53

Post by bluGill »

rTX5CMRXIfFG wrote: post #35
taylodl wrote: post #33

Earlier quoted context omitted.

Depends on how your unit tests were designed. You need to avoid the Fragile Test Problem. Bob Martin wrote about this a while back: http://blog.cleancoder.com/uncle-bob/2017/10/03/TestContrava...

There needs to be a rule against having to read entire articles or YT videos just so that a reader can understand the point that a commenter is trying to make, but which they cannot explain in their own words, no? It’s very anti-discussion, and how do we even know to what extent commenters are quoting or agreeing with their sources?

While the link in question isn't a good article, for complex subjects like this a long article is much better. The article linked should take several days to create if you are a great writer (again, I would create in a different way, but the content deserves days or even weeks of work to create), and thus cannot fit in a comment box except in a reduced form that doesn't do the idea justice.

Re: Unit Tests Considered Harmful

#54

Post by bluGill »

pc86 wrote: post #28
osigurdson wrote: post #13

Earlier quoted context omitted.

Agree. The "what about the client" trope is a complete non-argument. If unit tests make code better, the client maybe wants that. The argument should purely be based on if they are helpful.

I'll take it a step further than this. It doesn't matter if the client wants unit tests or not. If the developer has been hired to do the job, it's going to be up to that developer whether or not they feel unit tests will make the code better. If they do, they should do them. If they don't, they shouldn't. Caring about the client's opinions re: unit tests is a little like caring about their opinion re: the interior c…

Most clients should care about code that is long term maintainable even if it isn't in the contract. Thus tests are good. As a professional engineer (I'm intentionally invoking professional in context of other industries where the term has legal implications on quality, even though it doesn't exist for software it should!) you should care about that

Re: Unit Tests Considered Harmful

#55

Post by zbentley »

ramesh31 wrote: post #40

The author conflates unit testing with integration testing, although it seems the terminology is a huge source of confusion for articles like this. Anything with mocks is an integration. Units are pure functions only . Generally a mixture of unit and e2e is the best way to go. Mocked out integrations tend to be a nightmare to maintain and provide little value.

> Anything with mocks is an integration.

That's ... somewhere between "flat out wrong" and "I have never seen that terminology in use in my career so maybe I'm just extremely unlucky".

Sure, when things are on the line between integration and other types of tests the definition is often subjective/debatable. However, in my and my colleagues' experience, most integration tests are distinguished by reduced amounts of mocking, especially as pertains to: dependencies provided by other teams/business units; dependencies provided by third-party code; dependencies provided by networked services (including databases, caches, etc.).

In my experience, this tends to result in tests that take a bit longer to run and, due to their lower level of granularity, require slightly longer to understand failures. In return, they provide significantly higher prod-representativeness and business-value-delivered-per-hour-spent-test-writing.

Re: Unit Tests Considered Harmful

#56

Post by superchris »

Once again, missing the point of unit testing. Unit testing is about improving the design of the software, and allowing the design to evolve through refactoring (which is effectively impossible without unit testing). E2E or integration tests are for catching bugs

Re: Unit Tests Considered Harmful

#57

Post by taylodl »

rTX5CMRXIfFG wrote: post #35
taylodl wrote: post #33

Earlier quoted context omitted.

Depends on how your unit tests were designed. You need to avoid the Fragile Test Problem. Bob Martin wrote about this a while back: http://blog.cleancoder.com/uncle-bob/2017/10/03/TestContrava...

There needs to be a rule against having to read entire articles or YT videos just so that a reader can understand the point that a commenter is trying to make, but which they cannot explain in their own words, no? It’s very anti-discussion, and how do we even know to what extent commenters are quoting or agreeing with their sources?

Your unit tests should be contravariant to the code its testing so that way your code can be refactored and your existing unit tests can be used AS-IS to test the refactored code.

The Fragile Test Problem arises when your unit tests are covariant to the code its testing, such that a significant refactor breaks the tests - not because the code is necessarily wrong, but the structure of the code is no longer what the unit test was expecting.

Re: Unit Tests Considered Harmful

#58

Post by luisgvv »

nu11ptr wrote: post #17

Everyone talking about unit tests, and testing in general as a thing to achieve are missing it entirely. Tests aren't a thing to achieve, and talking about whether unit testing, mocks etc. are better or worse without context is pointless. The thing to achieve is having a code base that 1) works 2) can be changed easily and proved to be still working. Now that we know that we can start deciding "how" we can achieve th…

I wish management knew this... It's a nightmare when some manager that hasn't coded in years thinks it's a good idea to push unit testing into the CI builds so that it fails if it doesn't meet code coverage criteria etc.

Re: Unit Tests Considered Harmful

#59

Post by ramesh31 »

zbentley wrote: post #55
ramesh31 wrote: post #40

The author conflates unit testing with integration testing, although it seems the terminology is a huge source of confusion for articles like this. Anything with mocks is an integration. Units are pure functions only . Generally a mixture of unit and e2e is the best way to go. Mocked out integrations tend to be a nightmare to maintain and provide little value.

> Anything with mocks is an integration. That's ... somewhere between "flat out wrong" and "I have never seen that terminology in use in my career so maybe I'm just extremely unlucky". Sure, when things are on the line between integration and other types of tests the definition is often subjective/debatable. However, in my and my colleagues' experience, most integration tests are distinguished by reduced amounts of m…

>In my experience, this tends to result in tests that take a bit longer to run and, due to their lower level of granularity, require slightly longer to understand failures. In return, they provide significantly higher prod-representativeness and business-value-delivered-per-hour-spent-test-writing.

You are describing E2E tests here though. And this is the general problem with even discussing these things. It seems that literally everyone has a different mental model of the boundaries between each.

Re: Unit Tests Considered Harmful

#60

Post by moribvndvs »

Enough with the hyperbole. The argument is that unit tests alone are insufficient to ensure your software is functional and to detect regressions, and likewise unit test coverage is not an incontrovertible measure of quality or reliability. We can just present that argument on its own merit without the hot take.

My observations:

- developers need a fast feedback loop. While you’re in the process of changing things, waiting many seconds or (ugh) minutes for feedback not only wastes time but risks pulling you out of a flow state.

- e2e and integration tests are often much slower than unit tests and often require more coordination, putting pressure on the prior point. If you’re able to get e2e/integration tests working just as fast, good for you, do what you want (but you’re probably mocking a lot to do so, isn’t that what the author was complaining about)?

- It’s usually difficult to build e2e or integration tests until all the components are at least stubbed out. This is fine for finding regressions and providing evidence you’ve met your acceptance criteria, but they can be awkward to use during the active development process where you are refactoring considerably.

- e2e/integration tests are harder and a bit more expensive to build, and this is made worse when you have lots of edge cases you’d like to cover.

- the slower, the more expensive, the less immediately useful your testing approach is, the less likely a developer is to follow it, either by avoiding it entirely or by creating low value tests. Sure, the former can be managed by mandate, the latter is squishier.

So what’s my point? It doesn’t have to be all or nothing. I usually end up with a lot of fast unit tests to cover tricky edge cases of all my more critical components, and a smaller number of integration tests aligned with acceptance criteria to provide regression coverage. And then smoke tests. And then load and stress tests. And then… well, anyway, use the ones that are appropriate to your project and available resources, but defense in depth is a strategy that also works for quality (e.g. a coordinated combination of several types of tests gives you more assurance, if you’ve got the time and money)

There’s a reason we’ve developed a lot of different testing techniques/types; they have different purposes and deal with problems. Trying to roll back the clock and thinking you can solve all your testing problems in one shot is IMO naive.

Post reply on HN