I think my dream test framework would have the following features:
1) Test suites are organized as trees, not as lists:
I found one of the most common reasons to have many assertions in a test was that you want to share some complicated setup/teardown logic - or that one testable action depends on another testable action having happened before. (i.e., adding an item - asserting it's there, then removing it, asserting it's gone).
The disadvantage is that you have to lower the granularity of your tests - if you want to debug a specific action, you still have to rerun the whole test.
I think a better way to solve this would be to organize tests as a tree, maybe something like this:
- A single unit test consists of a setup phase, a teardown phase, 0 or more assertions and 0 or more child tests. Each child test is organzed the same, i.e. can have child tests on its own, etc.
- When running a test, first the setup phase and assertions are executed, then each child test recursively, then the teardown phase. Success/failure is tracked for each test separately, but child tests are ran in the same process/context as the parent test.
- Each test can be started individually, including child tests. When a child test (or grandchild test, etc) is ran individually, the test runner will first run the setup phases of all ancestors, then run the test, then run the teardown phases of the ancestors.
- Bonus: In the setup phase, a test can dynamically generate child tests (e.g. as lambdas/closures). Each test must have a unique ID with which it can tracked across different test runs or started individually. This could be useful for parameterized tests or if you want to test a loop invariant across multiple iterations.
This would allow you to write your test script like one big multi-assert test, but still get fine-grained reports and control as if you'd have put each assert in a separate script.
2) Provide "metrics" and "change detection" as an alternative to assertions:
I think one of the most involved parts of writing tests can often be to verify the results - think which particular state you want to assert, how you can access that state in your script, etc.
A way to make this easier would be to provide a second kind of "output" for the test script: The test script simply outputs a list of key/value pairs without any notion whether or not the value is "correct" or "incorrect". The test runner stores the list and compares the values with the list from a previous test run - e.g. the previous commit. Every value that was changed between the runs is shown to the user and can be marked as "correct" or "incorrect".
This way, you could sort of interactively "learn" which values are correct and which aren't instead of having to figure out all of it beforehand.
The runner could also implement more complex conditions instead of "changed"/"did not change", such as "value may only change in one direction" e.g. for quality measures or "value must stay the same within a certain confidence interval" for flaky tests.
This could also let you track more difficult to manage metrics in a test, such as runtime or memory consumption of particular method calls.