Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Composable Tests

newsletter.kentbeck.com

Post reply on HN
Print view View original

31–40 of 67 posts

Re: Composable Tests

#31

Post by skydhash »

akoboldfrying wrote: post #24
skydhash wrote: post #18

Earlier quoted context omitted.

> Contra Kent and, it seems, prevailing wisdom, I think simply deleting test1 is by far the simplest, clearest and best way. Provided that your testing framework tells you which specific assertion failed (e.g., by telling you the line number in a stack trace), you do know exactly where the problem is. The issue with your approach is that codepath execution is more of a graph than a linear timeline. With one test, you…

If you need test1's logic as setup for test2, then you already have that "linear timeline" -- for test2 all by itself. Additionally (that is, redundantly) running the same setup code "prefix" by itself in test1 doesn't remove test2's linear timeline. ETA: I'm assuming your objection to a "linear timeline" is that it reduces the potential for running tests in parallel -- have I got that right? If not, what do you see…

If I know that step 1 is correct for all the combinations (N) of its input (in test 1), in test 2, I only need to test the specific combinations (M) of step 2, treating step 1 as an axiom. So no need to have NxM in one test, or NxM tests.

Like if you were testing a drone stabilization software, testing the flying state can always assume that it has indeed taken off. No need to validate that it has done so for each scenarios, I can directly put it the correct value. It’s a contrived example, but that axiomatic aspect helps greatly when designing interfaces to reduce coupling between step 1 and step 2.

Re: Composable Tests

#32

Post by locknitpicker »

aleksiy123 wrote: post #12
mrkeen wrote: post #7

> If a test runs by first setting up its own test fixture, creating from scratch all the data it will be using as input, then that test is guaranteed to be isolated. It doesn’t matter what order you run the tests, the results will be exactly the same. Completely backward. This definition requires isolation, rather than granting it. In reality, you (or your test framework) provides the isolation by hobbling along, exe…

I don’t think that’s right? The isolation comes from the test implementation not the framework. There isn’t any framework out there that can guarantee/give you isolation. If I create a new in mem dB in the test there’s nothing stopping me from running it in parallel? Nothing about that “requires” isolation. It is isolation.

> The isolation comes from the test implementation not the framework. There isn’t any framework out there that can guarantee/give you isolation.

I'm not sure what you mean by that. All the frameworks I ever used were designed with test isolation as the primary design goal. Even when you set shared test fixtures and setup/teardown code, all they provide is a way to share code across tests, which are by themselves independent and isolated. BDD-style frameworks are the notable exception in breaking away from the pattern of having isolation as a fundamental design trait. In fact, the whole reason why BDD-style tools require special support from testing frameworks is that they need to work around test isolation in order to share context across steps.

Re: Composable Tests

#33

Post by dack »

akoboldfrying wrote: post #16

> Deleting test1 loses us another property from the Test Desiderata—tests should be specific. That’s the property of tests where, when one fails, you know exactly where the problem is. Contra Kent and, it seems, prevailing wisdom, I think simply deleting test1 is by far the simplest, clearest and best way. Provided that your testing framework tells you which specific assertion failed (e.g., by telling you the line nu…

I think this could lead to the tests getting harder to reason about over time because individual test size will just grow (many iterations of deleting the simple test in favor of a more complicated test that also asserts the simple things)

and if you start simplifying the more complicated test later, you may not realize that you're accidentally deleting checks that don't exist anywhere else other than incidentally here. So it's less clear what's important without thinking it through each time.

now all that to say I don't think it's strictly that big of a deal either way - there's always tons of room for taste that can make one or the other way better in practice. But that was my gut-reaction when reading your comment.

Re: Composable Tests

#34

Post by Groxx »

Ehhh... when it truly is two tests bodged into one, then sure.

But sometimes you do this kind of thing to avoid useless test brittleness: does your test check that `doSomething()` does what you expect, or do you have another test for that and this test only checks that `nowSomethingElse()` changes the object in a predictable way, e.g. updates a calculated field?

If it's the former, then it might be two tests masquerading as one, and this might make sense.

If it's the latter, you've changed a test that only checks what it cares about, and now you have a test that depends on unrelated implementation detail and will probably break unnecessarily in the future. Plus you've removed the assert that was documenting what it expects, so it's harder to tell if fixing it should mean updating both checks, or only the second one.

Re: Composable Tests

#35

Post by jaggederest »

troupo wrote: post #15
RHSeeger wrote: post #8

Earlier quoted context omitted.

I wonder if it would work do design something that was able to say test 1: do step 1, assert step 1 test 2: requires: test 1 do step 2, assert step 2 test 3: requires: test 2 do step 3, assert step 3 test 4: requires: test 2 do step 3, assert step 3 If the assertion of each test doesn't change any state, that might make things easier to read. Though, given that I haven't spent much time pondering it, I expect it coul…

Yup. I'd love to have dependency declarations for tests like this.

That's... A thing in many frameworks. If it's not in yours, you could add it.

Forgive an old man some ruby:

    it 'relies on mobile setup defined elsewhere', :mobile => true do
      # test that relies on mobile setup here
    end

Re: Composable Tests

#37

Post by latencyharbor »

RHSeeger wrote: post #8
troupo wrote: post #3

Earlier quoted context omitted.

What he wrote is basically "don't repeat in test X what you already tested in test X-1". It's not as much composition as compounding, and can work quite well. Let's say something takes 20 steps, and you want to test all 20. Instead of this: test 1: do step 1, assert step 1 test 2: do step 1, assert step 1 do step 2, assert step 2 test 3: do step 1, assert step 1 do step 2, assert step 2 do step 3, assert step 3 ... y…

I wonder if it would work do design something that was able to say test 1: do step 1, assert step 1 test 2: requires: test 1 do step 2, assert step 2 test 3: requires: test 2 do step 3, assert step 3 test 4: requires: test 2 do step 3, assert step 3 If the assertion of each test doesn't change any state, that might make things easier to read. Though, given that I haven't spent much time pondering it, I expect it coul…

A dependency graph can help with reporting, but I’d avoid making “test 1 ran successfully” part of test 2’s fixture.

Those are two separate ideas:

1. Execution dependency: if a cheap contract test fails, skip expensive downstream tests because their results won’t be informative. 2. State dependency: test 2 consumes state left behind by test 1.

The first can be useful. The second tends to create order dependence, awkward retries, and failures that are hard to reproduce when a CI runner shards or parallelizes the suite.

A safer model is a DAG of independently reproducible tests: each node declares prerequisites for scheduling/reporting, but creates or restores its own input state. Then a downstream test can be marked “blocked by X” rather than failed, while still being runnable by itself when debugging.

That distinction also keeps the dependency metadata from becoming a hidden setup mechanism.

Re: Composable Tests

#38

Post by drdexebtjl »

RHSeeger wrote: post #8
troupo wrote: post #3

Earlier quoted context omitted.

What he wrote is basically "don't repeat in test X what you already tested in test X-1". It's not as much composition as compounding, and can work quite well. Let's say something takes 20 steps, and you want to test all 20. Instead of this: test 1: do step 1, assert step 1 test 2: do step 1, assert step 1 do step 2, assert step 2 test 3: do step 1, assert step 1 do step 2, assert step 2 do step 3, assert step 3 ... y…

I wonder if it would work do design something that was able to say test 1: do step 1, assert step 1 test 2: requires: test 1 do step 2, assert step 2 test 3: requires: test 2 do step 3, assert step 3 test 4: requires: test 2 do step 3, assert step 3 If the assertion of each test doesn't change any state, that might make things easier to read. Though, given that I haven't spent much time pondering it, I expect it coul…

The hard parts to design about this, imo, are:

- How to reconcile this with tests that execute many times with varying input data. You’d need some way to express requirements with specific inputs or shared inputs.

- Passing state between test dependencies.

- When, if ever, it’s fine to share step results between tests. If tests B and C require A, can you run A just once? Not always, but you should be able to when it’s safe.

I don’t think I’ve ever used a test framework that gets these things right.

Re: Composable Tests

#39

Post by hungryhobbit »

mrkeen wrote: post #13
troupo wrote: post #3

Earlier quoted context omitted.

What he wrote is basically "don't repeat in test X what you already tested in test X-1". It's not as much composition as compounding, and can work quite well. Let's say something takes 20 steps, and you want to test all 20. Instead of this: test 1: do step 1, assert step 1 test 2: do step 1, assert step 1 do step 2, assert step 2 test 3: do step 1, assert step 1 do step 2, assert step 2 do step 3, assert step 3 ... y…

I think I disagree with Kent, but your explanation is clearer, so I'll object here. There's nothing wrong with hitting the same assertion multiple times, even if it doesn't sit nicely in your gut. From a purely philosophical point of view: If I have testFoo(), testBar(), and testFooAndBar(), and my Foo is plain wrong, then both testFoo() and testFooAndBar() must fail. Anything less is misleading/dishonest. From a pra…

If you have people on your team deleting valid tests because of "philosophy", I think you have much bigger problems to solve than anything Kent Beck can help with.

Re: Composable Tests

#40

Post by avensec »

RHSeeger wrote: post #8
troupo wrote: post #3

Earlier quoted context omitted.

What he wrote is basically "don't repeat in test X what you already tested in test X-1". It's not as much composition as compounding, and can work quite well. Let's say something takes 20 steps, and you want to test all 20. Instead of this: test 1: do step 1, assert step 1 test 2: do step 1, assert step 1 do step 2, assert step 2 test 3: do step 1, assert step 1 do step 2, assert step 2 do step 3, assert step 3 ... y…

I wonder if it would work do design something that was able to say test 1: do step 1, assert step 1 test 2: requires: test 1 do step 2, assert step 2 test 3: requires: test 2 do step 3, assert step 3 test 4: requires: test 2 do step 3, assert step 3 If the assertion of each test doesn't change any state, that might make things easier to read. Though, given that I haven't spent much time pondering it, I expect it coul…

The pattern can work, but the domain matters, the test type matters (unit, integration, ui) and the trade-off associated matter.

e.g. If I am running a long-running UI-test scenario, I absolutely don't want test-5 to walk through 80% of the UI that was already exercised in tests 1-4. I am creating test coupling, but I'm saving cost/time by doing so.

But, you'll also hear why not to do this, because it creates test coupling / breaks atomic tests, which is generally seen as bad.

If that is a local integration test and those early steps run is millis? Then maybe we keep things uncoupled to allow the system to exercise the pathways without explicit expectations.

Post reply on HN