Is there anything like this in Python or C#? I have worked with OCaml extensively in coursework, but there’s no chance I’ll be using it in prod any time soon and I’d love toying with this approach in my working languages.
“Expect tests” make test-writing feel like a REPL session
81–90 of 93 posts
Re: “Expect tests” make test-writing feel like a REPL session
#82Earlier quoted context omitted.
In many contexts, there's just no such thing as a safe behavior change which should be allowed without a specific decision from you to allow it. As a database systems guy, I've seen countless examples of customer breakages caused by a developer's decision that some behavior or another is so trivial it doesn't need to be tested. When you're working on developing a random utility function (real example!), it's easy to…
In your story though the hapless dev just changed the test. And the reviewers approved it. This suggests that there are so many changes to tests that it's just become background noise.
It had, and that's precisely because of the lack of anything like the expect() tests described in the OP. It's laborious to reliably scan through a big test diff and identify when it's describing a user-facing change, and people are inevitably going to autopilot through it. If you have a golden file (the standard name in my area for an equivalent mechanism to expect() tests), the reviewer's work is a lot simpler: any non-append-only diff is a breaking change and must be either fixed or communicated broadly before deploying it.
Re: “Expect tests” make test-writing feel like a REPL session
#83Earlier quoted context omitted.
Yes, I have difficulty understanding the point of a test-writing system that relies on your explicit assumption that whatever the code already does is correct. What are you testing? Why?
A regression test is checking causality: Changes in new code, updating dependencies, updating the OS the software is running on, updating shared libraries, porting the code to a new platform, etc. aren't supposed to change the test results. "I may not know what cos(x) means, but whatever it is shouldn't depend on what OS version I'm running"
> "I may not know what cos(x) means, but whatever it is shouldn't depend on what OS version I'm running"
Cosine is a terrible example to use for that idea. It's pretty likely to change, for certain x, in similar circumstances to your examples of "when test results should never change".
Re: “Expect tests” make test-writing feel like a REPL session
#84> I think you’re supposed to write some nonsense, like assert fibonacci(15) == 8, then when the test says “WRONG! Expected 8, got 610”, you’re supposed to copy and paste the 610 from your terminal buffer into your editor. > This is insane! The sane approach is presumably to either expand the call tree and verify all the unique subsolutions. Or to do every step with a calculator if you can’t expand the call tree. > Th…
Well, the non-insane thing is to do property-based testing. Instead of testing only a handful of examples.
I prefer "code it twice and hope you get it right once" testing.
Complex systems use that system everywhere. Why aren't we doing it for our code?
Re: “Expect tests” make test-writing feel like a REPL session
#85Earlier quoted context omitted.
At Google the nickname for these kinds of tests was 'change detector tests'.
If you are saying this approach would tend to produce a lot of change-detector tests, then that is an issue, but I think scotty79 is making a different point: this approach would seem to make it easy to overlook any regressions that the latest change has created.
Yes, and that's exactly the issue with change detector tests.
Re: “Expect tests” make test-writing feel like a REPL session
#86Earlier quoted context omitted.
Well, the non-insane thing is to do property-based testing. Instead of testing only a handful of examples.
I prefer "code it twice and hope you get it right once" testing. Complex systems use that system everywhere. Why aren't we doing it for our code?
Comparing the output of your system against an oracle is one property you can test.
But you don't always have an oracle. So other properties still make sense.
As a simple example: if you code up a quantum mechanics simulator, that's hard, and I wouldn't be able to code up an oracle for you straight away. But I can tell you that you probably want to check that things like momentum and energy better be conserved.
Re: “Expect tests” make test-writing feel like a REPL session
#87Earlier quoted context omitted.
Well, the non-insane thing is to do property-based testing. Instead of testing only a handful of examples.
They also do that, the post refers to their Quickcheck library. But how do you property test the Fibonacci function ? There isn't much to say about it...
In addition to what
travisjungroth said, you can also check against a reference implementation.
Eg if you coded up an O(n) version of the Fibonacci calculation, you can check against the naive recursive one (or if you are feeling confident, you can check against the O(log n) solution via repeated squaring of matrices.)
Re: “Expect tests” make test-writing feel like a REPL session
#88Earlier quoted context omitted.
A regression test is checking causality: Changes in new code, updating dependencies, updating the OS the software is running on, updating shared libraries, porting the code to a new platform, etc. aren't supposed to change the test results. "I may not know what cos(x) means, but whatever it is shouldn't depend on what OS version I'm running"
> "I may not know what cos(x) means, but whatever it is shouldn't depend on what OS version I'm running" Cosine is a terrible example to use for that idea. It's pretty likely to change, for certain x, in similar circumstances to your examples of "when test results should never change".
If it's likely to change, then you especially want the regression test so you can decide how to handle the divergence during your port. Maybe one library preserves the signal on NaNs and the other doesn't. Or maybe the CPU's default rounding mode is different when called in this context, and you're off by 1 ulp.
In either case, if the behavior is to change, it should change as an informed decision and not because nobody noticed.
Re: “Expect tests” make test-writing feel like a REPL session
#89Earlier quoted context omitted.
> Any programmer dumb enough to just blindly accept that their program is correct is also a dumb enough programmer not to have begun writing a test in the first place. Then what's the point of this methodology? It requires you to write tests and also blindly accept that your program is correct. Maybe they should just rename it to "plausibility tests" or similar because that's what they're really testing. And while th…
> It requires you to write tests and also blindly accept that your program is correct. No. You can say no. Just don’t accept it. You’re a human and it asks. Even if you do accept it you can modify it because you have eyes and a keyboard and it’s written right there where you wrote your test. See https://github.com/rust-analyzer/expect-test for a demo gif of the rust version.
> No. You can say no. Just don’t accept it.
Yes you can except...
> You’re a human
Precisely. You're a human. Humans are lazy and bad at manually checking things are correct, especially if there's an "eh it's probably fine" option.
This is extremely well studied: https://en.wikipedia.org/wiki/Vigilance_(psychology)
As I said before, it's probably better than nothing in that it will help you detect obviously implausible results. But it really needs to be labelled as such otherwise people will assume that these are properly curated "golden" tests.
Re: “Expect tests” make test-writing feel like a REPL session
#90Earlier quoted context omitted.
...and my favourite term, "characterization test": https://en.wikipedia.org/wiki/Characterization_test "Regression test" means something else, at least at the companies I've worked at: It means a test that was written after a defect was found in production, to ensure that the same defect doesn't happen again (that the fix doesn't "regress"). It can be a manual test or an automated test. https://en.wikipedia.org/wiki/…
That’s fine and I have no objective argument against it. But I don’t see much reason to need two different names for tests that do the same thing merely based on how they were introduced. Sometimes I add a regression test because I fixed a bug, and sometimes I add a regression test because I just implemented a feature that I don’t want my future self to ruin: six months from now they will co-exist in the same suite a…
"Regression testing" can also refer to a process: When the QA team says they're doing regression testing, it means they're testing that existing functionality hasn't regressed (as opposed to testing a new feature).
I'm not particularly wedded to any of these terms, I'm just pointing out that "regression testing" has an established meaning, and it isn't snapshot testing (outside of certain industries, at least). I do find it amusing that one implementation of snapshot testing (https://pypi.org/project/pytest-regtest/) links to https://en.wikipedia.org/wiki/Regression_testing but that article doesn't describe snapshot testing at all! Maybe the article changed? Oh well, language changes too. ¯\_(ツ)_/¯