“Expect tests” make test-writing feel like a REPL session
blog.janestreet.com
“Expect tests” make test-writing feel like a REPL session
1–10 of 93 posts
Re: “Expect tests” make test-writing feel like a REPL session
#2Snapshot testing is great, and I wish more test frameworks included first-class support for them. This means that they can auto update with a flag, and can be stored either in the source inline or in an external file (both modes have different use cases). Note that doc tests can also be a form of this, e.g. in Python's.
"Expect tests" seems like a bad name, since that covers all tests.
Re: “Expect tests” make test-writing feel like a REPL session
#3I wonder if this has the same downsides as golden and screenshot type tests, where you end up over-asserting resulting in tests that break for unrelated changes?
Obviously that’s a risk for hand written tests too but it’s easier (today… who knows what copilot like systems will offer soon!) for a human to reason about what’s relevant.
Re: “Expect tests” make test-writing feel like a REPL session
#4A similar approach with pytest and pdb https://simonwillison.net/2020/Feb/11/cheating-at-unit-tests...
This does get me writing tests sooner.
Re: “Expect tests” make test-writing feel like a REPL session
#5A similar approach with pytest and pdb https://simonwillison.net/2020/Feb/11/cheating-at-unit-tests... This does get me writing tests sooner.
I do this with pytest-regtest's --regtest-reset command.
Re: “Expect tests” make test-writing feel like a REPL session
#6> I think you’re supposed to write some nonsense, like assert fibonacci(15) == 8, then when the test says “WRONG! Expected 8, got 610”, you’re supposed to copy and paste the 610 from your terminal buffer into your editor.
> This is insane!
The sane approach is presumably to either expand the call tree and verify all the unique subsolutions. Or to do every step with a calculator if you can’t expand the call tree.
> The %expect block starts out blank precisely because you don’t know what to expect. You let the computer figure it out for you. In our setup, you don’t just get a build failure telling you that you want 610 instead of a blank string. You get a diff showing you the exact change you’d need to make to your file to make this test pass; and with a keybinding you can “accept” that diff. The Emacs buffer you’re in will literally be overwritten in place with the new contents [1]:
Oh okay. The non-insane approach is to do the first thing but Emacs copies the result on your behalf.
Re: “Expect tests” make test-writing feel like a REPL session
#7Here's an older post from 2015 (also from Jane Street) explaining the same process https://blog.janestreet.com/testing-with-expectations/, but at the infancy of the method. It looks like they heavily polished it!
I like the approach, and I was indeed copy-pasting the result from my console...
Re: “Expect tests” make test-writing feel like a REPL session
#8Doesn't this approach make you update results of failing tests wholesale and possibly miss where a new result of some test is actually wrong?
Re: “Expect tests” make test-writing feel like a REPL session
#9Snapshot testing is great, and I wish more test frameworks included first-class support for them. This means that they can auto update with a flag, and can be stored either in the source inline or in an external file (both modes have different use cases). Note that doc tests can also be a form of this, e.g. in Python's. "Expect tests" seems like a bad name, since that covers all tests.
i find that snapshot testing gets overused in javascript - and mistakes can creep in easily, and if the snapshot is big, and in a separate file, code review can miss it.
I much prefer property based testing over expectation based testing. You have to explicitly think about what properties hold true about the thing you're writing.
For example, fib(N+1) = fib(N) + fib(N), so this property can be tested for all N; primitive generators can easily generate the data, and good composition framework can easily generate complex data from primitive data.
Of course, you have to have a property you can specify easily. Otherwise, it'd be exactly the same as expectation based testing.
Re: “Expect tests” make test-writing feel like a REPL session
#10> I think you’re supposed to write some nonsense, like assert fibonacci(15) == 8, then when the test says “WRONG! Expected 8, got 610”, you’re supposed to copy and paste the 610 from your terminal buffer into your editor. > This is insane! The sane approach is presumably to either expand the call tree and verify all the unique subsolutions. Or to do every step with a calculator if you can’t expand the call tree. > Th…
Well, the non-insane thing is to do property-based testing. Instead of testing only a handful of examples.