Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Expect tests” make test-writing feel like a REPL session

blog.janestreet.com

Post reply on HN
Print view View original

61–70 of 93 posts

Re: “Expect tests” make test-writing feel like a REPL session

#61

Post by potatoyogurt »

ElliotH wrote: post #3

I wonder if this has the same downsides as golden and screenshot type tests, where you end up over-asserting resulting in tests that break for unrelated changes? Obviously that’s a risk for hand written tests too but it’s easier (today… who knows what copilot like systems will offer soon!) for a human to reason about what’s relevant.

Yes, that is definitely a downside for these tests. The worst is when the text of some exception is printed and it includes line numbers. It does still require some discipline to think about what you're printing and avoid output that will be very noisy. This problem is mitigated quite a bit by the ease of accepting changes when these tests fail for obviously nonsense reasons though (just hit a couple buttons in an emacs buffer).

Re: “Expect tests” make test-writing feel like a REPL session

#62

Post by thedufer »

tantalor wrote: post #45
CGamesPlay wrote: post #2

Snapshot testing is great, and I wish more test frameworks included first-class support for them. This means that they can auto update with a flag, and can be stored either in the source inline or in an external file (both modes have different use cases). Note that doc tests can also be a form of this, e.g. in Python's. "Expect tests" seems like a bad name, since that covers all tests.

> update with a flag Yes this is right level of automation, not whatever this article is going on about with the editor integration. Yuck.

The open source use pattern for expect tests in OCaml (via dune) is exactly as you describe (see https://dune.readthedocs.io/en/stable/tests.html) - you run the tests with `--auto-promote` to tell it to update. The editor integration is a very simple keybinding on top of more generic tooling.

Re: “Expect tests” make test-writing feel like a REPL session

#64

Post by drothlis »

avgcorrection wrote: post #53
avgcorrection wrote: post #26

Earlier quoted context omitted.

Yeah, weird to see all these variations - Aha, an expect test! - Oh, you mean a snapshot test! - This here is akin to UI testing framework X where the test framework can compare an expected screenshot of the UI to a screenshot of the actual UI! The last one basically requires automation if you want anyone to make use of it. The regression testing automation described in the OP is a nice-to-have, not a so-good-that-it…

… And apparently also “change detector test” https://news.ycombinator.com/item?id=34379175

...and my favourite term, "characterization test": https://en.wikipedia.org/wiki/Characterization_test

"Regression test" means something else, at least at the companies I've worked at: It means a test that was written after a defect was found in production, to ensure that the same defect doesn't happen again (that the fix doesn't "regress"). It can be a manual test or an automated test. https://en.wikipedia.org/wiki/Regression_testing

Re: “Expect tests” make test-writing feel like a REPL session

#65

Post by theptip »

I tend to think that tests should be carefully crafted for readability just like normal code. The “content of a REPL” is unlikely to be well-thought out enough to preserve meaningful invariants while remaining supple in the direction of likely changes. Perhaps in the hands of very good engineers this tool is net positive, but I shudder at giving junior engineers a tool that encourages less structure in tests.

A good set of fixture/helper functions should let you write really short and expressive tests (or tabular parametrized tests, if you prefer) which seems to me to resolve most of the pain points the author is complaining about.

One big advantage I do see with this approach is it seems to be a very compact rendering of a table of outputs; in Python+pytest+PyCharm if I run a 10-example parametrized test, I have to click through to see each failure individually. Perhaps there is a UX learning here that just rendering the raw errors into the code beside the test matrix could help visualize results faster.

As an aside, I have recently been enjoying the “write an ascii representation as your test assert” mode of testing, it can give a different way of intuiting what is going on.

Re: “Expect tests” make test-writing feel like a REPL session

#66

Post by User23 »

mabbo wrote: post #16

> But think: everything in those describe blocks had to be written by hand. It also had to be thought about by the developer. Someone had to say "I want the code to do this under these conditions". If your tests can be autogenerated then they aren't verifying expected behaviour , they're just locking in your implementation such that it can't change later. They are saying "hey look everyone, I got my coverage metric t…

If you’re autogenerating your tests from a specification and not an implementation then it can potentially be useful.

Re: “Expect tests” make test-writing feel like a REPL session

#67

Post by adrianmonk »

I think this would suffer from the same problem as partial self-driving cars: it's human nature for vigilance to falter if it doesn't feel like you're the sole/primary one in control.

Of course, you can say "I won't let myself do that", but working against human nature is not a formula for success. If my back hurts, I can tell myself I'm just going to go lie down on the bed for 10 minutes but not take a nap, but then 30 minutes later I wake up feeling groggy.

Re: “Expect tests” make test-writing feel like a REPL session

#68

Post by wildcow »

arcturus17 wrote: post #63

Is there anything like this in Python or C#? I have worked with OCaml extensively in coursework, but there’s no chance I’ll be using it in prod any time soon and I’d love toying with this approach in my working languages.

In C# https://theramis.github.io/Snapper/#/pages/quickstart. I actually think there are other as well. But this is what i am trying to scratch to see if I can use it somehow.

Re: “Expect tests” make test-writing feel like a REPL session

#69

Post by avgcorrection »

drothlis wrote: post #64
avgcorrection wrote: post #53

Earlier quoted context omitted.

… And apparently also “change detector test” https://news.ycombinator.com/item?id=34379175

...and my favourite term, "characterization test": https://en.wikipedia.org/wiki/Characterization_test "Regression test" means something else, at least at the companies I've worked at: It means a test that was written after a defect was found in production, to ensure that the same defect doesn't happen again (that the fix doesn't "regress"). It can be a manual test or an automated test. https://en.wikipedia.org/wiki/…

That’s fine and I have no objective argument against it. But I don’t see much reason to need two different names for tests that do the same thing merely based on how they were introduced. Sometimes I add a regression test because I fixed a bug, and sometimes I add a regression test because I just implemented a feature that I don’t want my future self to ruin: six months from now they will co-exist in the same suite and serve the same function.

One reason to call bug fix tests for “regression tests” (and only those kinds of tests) is that someone might regress the code base through a merge conflict (maybe they effectively undo a commit?). So that’s one argument I suppose.

Re: “Expect tests” make test-writing feel like a REPL session

#70

Post by CJefferson »

I work with a language where all test are expect tests ( GAP ). The biggest problem is you can basically never change how built in types are printed, as you'll break all tests in every program. For example, someone wanted to improve how plurals are printed, but that would break every test.
Post reply on HN