Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Composable Tests

newsletter.kentbeck.com

Post reply on HN
Print view View original

51–60 of 67 posts

Re: Composable Tests

#51

Post by aleksiy123 »

locknitpicker wrote: post #32
aleksiy123 wrote: post #12

Earlier quoted context omitted.

I don’t think that’s right? The isolation comes from the test implementation not the framework. There isn’t any framework out there that can guarantee/give you isolation. If I create a new in mem dB in the test there’s nothing stopping me from running it in parallel? Nothing about that “requires” isolation. It is isolation.

> The isolation comes from the test implementation not the framework. There isn’t any framework out there that can guarantee/give you isolation. I'm not sure what you mean by that. All the frameworks I ever used were designed with test isolation as the primary design goal. Even when you set shared test fixtures and setup/teardown code, all they provide is a way to share code across tests, which are by themselves inde…

All I mean that there is no way for a framework to prevent your tests from interfering with each other, or to solve isolation for you.

It’s the implementation of each test that is responsible for its isolation. Every time you write or make changes to your tests. You have to think, can they mess with each other.

Re: Composable Tests

#52

Post by aleksiy123 »

solarengineer wrote: post #42
locknitpicker wrote: post #32

Earlier quoted context omitted.

> The isolation comes from the test implementation not the framework. There isn’t any framework out there that can guarantee/give you isolation. I'm not sure what you mean by that. All the frameworks I ever used were designed with test isolation as the primary design goal. Even when you set shared test fixtures and setup/teardown code, all they provide is a way to share code across tests, which are by themselves inde…

In JUnit, tests run sequentially in a single thread by default [1]. Parallel execution must be explicitly enabled. When enabled, JUnit uses a fork-join thread pool, so tests may run concurrently on different worker threads. Because these threads are reused, a ThreadLocal value left behind by one test could be visible to a later test that happens to run on the same thread. Setup and teardown methods can be used to cre…

This

Re: Composable Tests

#53

Post by RHSeeger »

drdexebtjl wrote: post #38
RHSeeger wrote: post #8

Earlier quoted context omitted.

I wonder if it would work do design something that was able to say test 1: do step 1, assert step 1 test 2: requires: test 1 do step 2, assert step 2 test 3: requires: test 2 do step 3, assert step 3 test 4: requires: test 2 do step 3, assert step 3 If the assertion of each test doesn't change any state, that might make things easier to read. Though, given that I haven't spent much time pondering it, I expect it coul…

The hard parts to design about this, imo, are: - How to reconcile this with tests that execute many times with varying input data. You’d need some way to express requirements with specific inputs or shared inputs. - Passing state between test dependencies. - When, if ever, it’s fine to share step results between tests. If tests B and C require A, can you run A just once? Not always, but you should be able to when it’…

> Passing state between test dependencies

Actually, I wasn't even thinking about passing state. I was thinking about shared setup steps. I'm perfectly happy for the same steps to run for each test - as long as each test doesn't need to list the steps (when they're setup steps not directly related to the thing being tested)

Re: Composable Tests

#54

Post by pfdietz »

crabbone wrote: post #43
pfdietz wrote: post #5

I had a test suite with thousands of tests. One way of running it was to take all the passing tests, and then run them repeatedly in random order. This found new bugs involving unintended persistent state.

While I don't doubt the veracity of your report, I don't think this is an efficient testing strategy. Ideally, you don't want a very large number of tests, no matter how big the system is. Tests are, effectively, an interface to the program that assesses the system quality / readiness for use. Any interface with thousands of individual pieces is difficult to use. Random combinations of tests also don't spark joy beca…

> While I don't doubt the veracity of your report, I don't think this is an efficient testing strategy.

The marginal cost of running the tests like that was very small, and bugs were found. The cost per found bug seemed very reasonable. Every test failure in this strategy indicates a bug.

(This was the standard compliance test suite for Common Lisp, btw.)

Re: Composable Tests

#55

Post by npodbielski »

Jtsummers wrote: post #47
npodbielski wrote: post #46

What I do not like about that kind of example is its abstractiveness. Yes sure you can argue about testing `doSomething()` and it all falls apart when there is an actual business scenario to test.

It's not hard, with some modest experience (say 1-2 years of professional work or serious amateur interest), to extrapolate from his deliberately high-level discussion (it was written for a blog/newsletter, not for a book) to something more interesting and "real world". What's hard is coming up with an example that's nearly complete and fits into something the size of a newsletter or comment. He did an alright job of…

If it is not hard, with an experience of this guy, he could came up with a better example? It is your own words so I am sure you will agree? He did alright job so he could exercise gray matter a bit and came up with some API for saving customer data and then test it his way? I mean... This would be more interesting and "real world" so I am sure it would be worth to write longer blog post about it!

Re: Composable Tests

#56

Post by crabbone »

pfdietz wrote: post #54
crabbone wrote: post #43

Earlier quoted context omitted.

While I don't doubt the veracity of your report, I don't think this is an efficient testing strategy. Ideally, you don't want a very large number of tests, no matter how big the system is. Tests are, effectively, an interface to the program that assesses the system quality / readiness for use. Any interface with thousands of individual pieces is difficult to use. Random combinations of tests also don't spark joy beca…

> While I don't doubt the veracity of your report, I don't think this is an efficient testing strategy. The marginal cost of running the tests like that was very small, and bugs were found. The cost per found bug seemed very reasonable. Every test failure in this strategy indicates a bug. (This was the standard compliance test suite for Common Lisp, btw.)

Now imagine someone who needs to test something on eg. P6e-GB200 series VM in EC2 (this is one of the newest Nvidia accelerator models: GB200 multiplied by something like 60 iirc).

I don't know what the price is and I'm not even sure you can just create this kind of a VM with any regular account, but I'm sure you can imagine a price of running such a test.

The price doesn't have to be the limiting factor. It may be time (if a test takes a very long time to set up), or it could be labor (imagine needing to flash the ROM of a PCB for each test or similar).

This is what I mean when I say that your experience is likely not transferable to many real-life testing applications. But, even if you don't pay that much for an individual test, still, why waste time and resources, if you could do better?

Re: Composable Tests

#57

Post by crabbone »

pkhuong wrote: post #49
crabbone wrote: post #43

Earlier quoted context omitted.

While I don't doubt the veracity of your report, I don't think this is an efficient testing strategy. Ideally, you don't want a very large number of tests, no matter how big the system is. Tests are, effectively, an interface to the program that assesses the system quality / readiness for use. Any interface with thousands of individual pieces is difficult to use. Random combinations of tests also don't spark joy beca…

> Any interface with thousands of individual pieces is difficult to use. Standard libraries considered harmful: 100 interfaces with 10 pieces each are obviously easier to use.

If they are organized into meaningful groups, then, of course: the organization makes it easier to use. Compare: have thousands of name-phone number pairings in any order vs same pairings, but now they are grouped by the first letter of the name? Don't you think that the later is easier to use?

Re: Composable Tests

#58

Post by drdexebtjl »

RHSeeger wrote: post #53
drdexebtjl wrote: post #38

Earlier quoted context omitted.

The hard parts to design about this, imo, are: - How to reconcile this with tests that execute many times with varying input data. You’d need some way to express requirements with specific inputs or shared inputs. - Passing state between test dependencies. - When, if ever, it’s fine to share step results between tests. If tests B and C require A, can you run A just once? Not always, but you should be able to when it’…

> Passing state between test dependencies Actually, I wasn't even thinking about passing state. I was thinking about shared setup steps. I'm perfectly happy for the same steps to run for each test - as long as each test doesn't need to list the steps (when they're setup steps not directly related to the thing being tested)

For example, suppose you want to write a test for the shipOrder(orderId) function, and you want it to depend on the test for the placeOrder(shoppingCart) -> OrderId function. Even if you are fine calling placeOrder twice, once for the placeOrder test and once for the placeOrderAndShipOrder test, you still need the placeOrder test to provide an order ID to the second test, and not just a confirmation that it completed successfully.

Re: Composable Tests

#59

Post by cadamsdotcom »

Why wouldn't you use parameterized tests or compose your setup out of fixtures that run the shared setup code?

Running a test, then running it a second time, then asserting something else.. is just weird.

Re: Composable Tests

#60

Post by locknitpicker »

solarengineer wrote: post #42
locknitpicker wrote: post #32

Earlier quoted context omitted.

> The isolation comes from the test implementation not the framework. There isn’t any framework out there that can guarantee/give you isolation. I'm not sure what you mean by that. All the frameworks I ever used were designed with test isolation as the primary design goal. Even when you set shared test fixtures and setup/teardown code, all they provide is a way to share code across tests, which are by themselves inde…

In JUnit, tests run sequentially in a single thread by default [1]. Parallel execution must be explicitly enabled. When enabled, JUnit uses a fork-join thread pool, so tests may run concurrently on different worker threads. Because these threads are reused, a ThreadLocal value left behind by one test could be visible to a later test that happens to run on the same thread. Setup and teardown methods can be used to cre…

> Parallel execution must be explicitly enabled. When enabled, JUnit uses a fork-join thread pool, so tests may run concurrently on different worker threads. Because these threads are reused, a ThreadLocal value left behind by one test could be visible to a later test that happens to run on the same thread.

I don't understand your comment. JUnit docs describe parallel execution as an experimental feature that still has gotchas that you need to be mindful to avoid. So you have a test framework designed with isolation in mind for a specific execution mode, but when you go out of your way to try another execution mode that is described as experimental then you can stumble upon very corner cases where isolation is not ensured. What point did you wanted to convey?

Post reply on HN