Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Composable Tests

newsletter.kentbeck.com

Post reply on HN
Print view View original

41–50 of 67 posts

Re: Composable Tests

#41

Post by crabbone »

1. I'm not sure what CodeRabbit has to do with the article... It's obviously an advertisement, but it's merged into the text of the article, which I find bizarre. As an aside, I used CodeRabbit at work, and I have mixed feelings about it. I'm not entirely against using it or a similar tool, but people often treat the comments from CodeRabbit as a gospel, and they can harm their code as a result. Also, I've never seen CodeRabbit being anything more than a superficial reviewer: fixing typos, other unintended errors, but it never comments on the substance of the change, which implicitly validates it for the author.

2. On test composition. Unit tests are called "unit" because they are supposed to test one thing. If a test is testing more than one thing, it's an integration test. Ideally, people writing unit tests are the developers themselves and people writing integration tests are test (automation) people. This matters for administrative reasons: in the development cycle, the unit test is the basic check that validates a particular piece of code, probably, submitted for review or for merging. Passing such a test might be a necessary condition to progress the changeset along the designed workflow path. Integration tests, on the other hand, are more of a retrospective tool that is meant for detection of problems in the entire product. A failure of an integration test should, normally, schedule new task for the developers, not reject the one being worked on. Integration tests, typically, will require a more elaborate system under test setup and a more elaborate, perhaps involving multiple teams, investigation of the failure. They can be also a lot more expensive to run in terms of equipment used.

Finally, the author touched on a contentious subject a.k.a. the number of assertions in a test. A lot of people believe (me included) that the number of assertions in the unit test should be exactly one. This is often inconvenient because it requires implementation of equality for possibly ad hoc created set of results. Even so, I believe it's still worth it.

When it comes to integration tests, I don't believe assertions are at all the way to go. The system under test should be monitored continuously and every reading should be compared against the desired state of the system that the test modifies simultaneously with the change effected to the system. This is because, in practice, it's rarely just two features that are tested together. If the test waits until the final step to compare the desired and the actual state of the system, the error as well as the context in which it happened might be long gone.

As a side bonus: the monitoring+alerts system could well be part of the product itself, or, if not, it can be used in long-running tests intended to collect mileage (i.e. tests intended to prove that the system performance doesn't degrade over substantially long periods of time).

Re: Composable Tests

#42

Post by solarengineer »

locknitpicker wrote: post #32
aleksiy123 wrote: post #12

Earlier quoted context omitted.

I don’t think that’s right? The isolation comes from the test implementation not the framework. There isn’t any framework out there that can guarantee/give you isolation. If I create a new in mem dB in the test there’s nothing stopping me from running it in parallel? Nothing about that “requires” isolation. It is isolation.

> The isolation comes from the test implementation not the framework. There isn’t any framework out there that can guarantee/give you isolation. I'm not sure what you mean by that. All the frameworks I ever used were designed with test isolation as the primary design goal. Even when you set shared test fixtures and setup/teardown code, all they provide is a way to share code across tests, which are by themselves inde…

In JUnit, tests run sequentially in a single thread by default [1].

Parallel execution must be explicitly enabled. When enabled, JUnit uses a fork-join thread pool, so tests may run concurrently on different worker threads. Because these threads are reused, a ThreadLocal value left behind by one test could be visible to a later test that happens to run on the same thread.

Setup and teardown methods can be used to create and clean up test-specific state. However, developers must still ensure that test data is unique to the test so that concurrently running tests do not interfere with each other. This uniqueness is unfortunately called "isolated", and has led to much confusion like in this thread. Certainly, the Test Execution Framework cannot guarantee data-isolation.

Parallel and randomized test execution can also help expose application-side problems involving shared or order-dependent state. Such failures may only appear when tests happen to exercise the application under the relevant ordering or concurrency conditions. When I was a junior developer teaching myself Java Servlets in 2000, I had to learn this lesson the hard way. A Test Execution Framework would not be able to guarantee any "isolation" of test data and of workflows if the server-side state is mis-managed by the tech stack and/or by the developer.

[1] https://docs.junit.org/6.1.3/writing-tests/parallel-executio...

Re: Composable Tests

#43

Post by crabbone »

pfdietz wrote: post #5

I had a test suite with thousands of tests. One way of running it was to take all the passing tests, and then run them repeatedly in random order. This found new bugs involving unintended persistent state.

While I don't doubt the veracity of your report, I don't think this is an efficient testing strategy.

Ideally, you don't want a very large number of tests, no matter how big the system is. Tests are, effectively, an interface to the program that assesses the system quality / readiness for use. Any interface with thousands of individual pieces is difficult to use.

Random combinations of tests also don't spark joy because this means both repetition (i.e. waste of resources) and testing potentially useless (unreachable or invalid) system states (both wastes resources and creates false alarms).

Ideally, the tests should be able to compose only in desired ways and rather than combining them randomly, there should be some deterministic process that creates a unique sequence or a tree of individual tests on subsequent runs. Ideally, such a test runner could also be configured to start with an existing system in a known state s.t. the tester can apply a patch and resume testing.

Re: Composable Tests

#45

Post by senderista »

crabbone wrote: post #41

1. I'm not sure what CodeRabbit has to do with the article... It's obviously an advertisement, but it's merged into the text of the article, which I find bizarre. As an aside, I used CodeRabbit at work, and I have mixed feelings about it. I'm not entirely against using it or a similar tool, but people often treat the comments from CodeRabbit as a gospel, and they can harm their code as a result. Also, I've never seen…

That's why you pepper your code with assertions and rely on depth of test coverage (including generative testing) to hit them.

Re: Composable Tests

#46

Post by npodbielski »

What I do not like about that kind of example is its abstractiveness. Yes sure you can argue about testing `doSomething()` and it all falls apart when there is an actual business scenario to test.

Re: Composable Tests

#47

Post by Jtsummers »

npodbielski wrote: post #46

What I do not like about that kind of example is its abstractiveness. Yes sure you can argue about testing `doSomething()` and it all falls apart when there is an actual business scenario to test.

It's not hard, with some modest experience (say 1-2 years of professional work or serious amateur interest), to extrapolate from his deliberately high-level discussion (it was written for a blog/newsletter, not for a book) to something more interesting and "real world". What's hard is coming up with an example that's nearly complete and fits into something the size of a newsletter or comment. He did an alright job of it, just left the readers with the need to exercise their gray matter a bit.

Re: Composable Tests

#48

Post by mrkeen »

hungryhobbit wrote: post #39
mrkeen wrote: post #13

Earlier quoted context omitted.

I think I disagree with Kent, but your explanation is clearer, so I'll object here. There's nothing wrong with hitting the same assertion multiple times, even if it doesn't sit nicely in your gut. From a purely philosophical point of view: If I have testFoo(), testBar(), and testFooAndBar(), and my Foo is plain wrong, then both testFoo() and testFooAndBar() must fail. Anything less is misleading/dishonest. From a pra…

If you have people on your team deleting valid tests because of "philosophy", I think you have much bigger problems to solve than anything Kent Beck can help with.

To be clear, the philosophy I quoted was directly from Kent Beck in TFA, i.e. this is Kent Beck's "help".

I say leave both tests as is.

Kent Beck says:

  From a purely aesthetic standpoint (& don’t discount aesthetics), leaving both tests as is offends my sensibilities. They are redundant! Something must be wrong.
It's not just philosophy, it's aesthetics apparently!

Re: Composable Tests

#49

Post by pkhuong »

crabbone wrote: post #43
pfdietz wrote: post #5

I had a test suite with thousands of tests. One way of running it was to take all the passing tests, and then run them repeatedly in random order. This found new bugs involving unintended persistent state.

While I don't doubt the veracity of your report, I don't think this is an efficient testing strategy. Ideally, you don't want a very large number of tests, no matter how big the system is. Tests are, effectively, an interface to the program that assesses the system quality / readiness for use. Any interface with thousands of individual pieces is difficult to use. Random combinations of tests also don't spark joy beca…

> Any interface with thousands of individual pieces is difficult to use.

Standard libraries considered harmful: 100 interfaces with 10 pieces each are obviously easier to use.

Re: Composable Tests

#50

Post by RHSeeger »

avensec wrote: post #40
RHSeeger wrote: post #8

Earlier quoted context omitted.

I wonder if it would work do design something that was able to say test 1: do step 1, assert step 1 test 2: requires: test 1 do step 2, assert step 2 test 3: requires: test 2 do step 3, assert step 3 test 4: requires: test 2 do step 3, assert step 3 If the assertion of each test doesn't change any state, that might make things easier to read. Though, given that I haven't spent much time pondering it, I expect it coul…

The pattern can work, but the domain matters, the test type matters (unit, integration, ui) and the trade-off associated matter. e.g. If I am running a long-running UI-test scenario, I absolutely don't want test-5 to walk through 80% of the UI that was already exercised in tests 1-4. I am creating test coupling, but I'm saving cost/time by doing so. But, you'll also hear why not to do this, because it creates test co…

The question was less about speed and more about not having the same code duplicated over and over across tests. Which is how I read the article talking about it. Allowing one test to "depend" on another makes it clear they use the same setup (presumably with the second test going "a bit further", but not necessarily).

I wouldn't have a problem with something like

    test-1:
        setup:
            do-the-thing
        verification
            assert-the-thing-happened
    test-2
        setup:
            depends-on: test-1 // tells it to run test-1's setup
            do-the-next-thing
        verification
            assert-the-next-thing-happened
The format is awful, but the idea is that most tests are of the form

    GIVEN
       Some initial setup
    WHEN
       I run command
    THEN
       The result of that command is what is expected
And, in that context, the GIVEN frequently contains noise not directly related to understanding what is being tested.

I actually use the GIVEN/WHEN/THEN keywords in my tests, to make them easier to read

Post reply on HN