“Expect tests” make test-writing feel like a REPL session • Hacker News Board

Re: “Expect tests” make test-writing feel like a REPL session

#41

Post by imajoredinecon » Sat, Jan 14, 2023, 2:32 PM UTC

eru wrote: post #12

scotty79 wrote: post #8

Doesn't this approach make you update results of failing tests wholesale and possibly miss where a new result of some test is actually wrong? https://docs.rs/expect-test/latest/expect_test/

At Google the nickname for these kinds of tests was 'change detector tests'.

Yeah, the OP's counterargument is that you can filter down what goes into the test output. But at that point it seems not too different qualitatively from the traditional bottom-up approach where you just write assertions yourself, except that the framework does the job of populating the assertions' expected values.

Re: “Expect tests” make test-writing feel like a REPL session

#42

Post by tantalor » Sat, Jan 14, 2023, 2:40 PM UTC

mabbo wrote: post #16

> But think: everything in those describe blocks had to be written by hand. It also had to be thought about by the developer. Someone had to say "I want the code to do this under these conditions". If your tests can be autogenerated then they aren't verifying expected behaviour , they're just locking in your implementation such that it can't change later. They are saying "hey look everyone, I got my coverage metric t…

> locking in your implementation such that it can't change later

That's the whole point of tests. All tests do that.

This protects against later code changes that change behavior (output or side effects) unintentionally.

When you intend to change behavior then you need to change the tests tests too.

Re: “Expect tests” make test-writing feel like a REPL session

#43

Post by khuey » Sat, Jan 14, 2023, 2:42 PM UTC

mabbo wrote: post #16

> But think: everything in those describe blocks had to be written by hand. It also had to be thought about by the developer. Someone had to say "I want the code to do this under these conditions". If your tests can be autogenerated then they aren't verifying expected behaviour , they're just locking in your implementation such that it can't change later. They are saying "hey look everyone, I got my coverage metric t…

In many contexts there's value in ensuring the behavior doesn't change without being noticed. You're just moving the developer thinking about the expected behavior from when the test is written to when the test fails.

Re: “Expect tests” make test-writing feel like a REPL session

#44

Post by schwartzworld » Sat, Jan 14, 2023, 2:42 PM UTC

tantalor wrote: post #42

mabbo wrote: post #16

> But think: everything in those describe blocks had to be written by hand. It also had to be thought about by the developer. Someone had to say "I want the code to do this under these conditions". If your tests can be autogenerated then they aren't verifying expected behaviour , they're just locking in your implementation such that it can't change later. They are saying "hey look everyone, I got my coverage metric t…

> locking in your implementation such that it can't change later That's the whole point of tests. All tests do that. This protects against later code changes that change behavior (output or side effects) unintentionally . When you intend to change behavior then you need to change the tests tests too.

Implementation !== Behavior. You want to test the behavior, not the implementation. I'd expect tests to change when behavior changes, but reimplementing the same behavior, the tests should pass when you're done.

Re: “Expect tests” make test-writing feel like a REPL session

#45

Post by tantalor » Sat, Jan 14, 2023, 2:42 PM UTC

CGamesPlay wrote: post #2

Snapshot testing is great, and I wish more test frameworks included first-class support for them. This means that they can auto update with a flag, and can be stored either in the source inline or in an external file (both modes have different use cases). Note that doc tests can also be a form of this, e.g. in Python's. "Expect tests" seems like a bad name, since that covers all tests.

> update with a flag

Yes this is right level of automation, not whatever this article is going on about with the editor integration. Yuck.

Re: “Expect tests” make test-writing feel like a REPL session

#46

Post by gavinray » Sat, Jan 14, 2023, 2:45 PM UTC

avgcorrection wrote: post #6

> I think you’re supposed to write some nonsense, like assert fibonacci(15) == 8, then when the test says “WRONG! Expected 8, got 610”, you’re supposed to copy and paste the 610 from your terminal buffer into your editor. > This is insane! The sane approach is presumably to either expand the call tree and verify all the unique subsolutions. Or to do every step with a calculator if you can’t expand the call tree. > Th…

Lol I came here to post this but you beat me.

Re: “Expect tests” make test-writing feel like a REPL session

#47

Post by mabbo » Sat, Jan 14, 2023, 3:10 PM UTC

tantalor wrote: post #42

mabbo wrote: post #16

> But think: everything in those describe blocks had to be written by hand. It also had to be thought about by the developer. Someone had to say "I want the code to do this under these conditions". If your tests can be autogenerated then they aren't verifying expected behaviour , they're just locking in your implementation such that it can't change later. They are saying "hey look everyone, I got my coverage metric t…

> locking in your implementation such that it can't change later That's the whole point of tests. All tests do that. This protects against later code changes that change behavior (output or side effects) unintentionally . When you intend to change behavior then you need to change the tests tests too.

I disagree.

Tests should define what the expectations are. If a change does not impact those expectations, then it should be allowed and not break any tests.

Locking your code such that all future changes require updating old tests tells me that your tests are just your code written a second time, with no thought about what the code's requirements are.

Re: “Expect tests” make test-writing feel like a REPL session

#48

Post by superb-owl » Sat, Jan 14, 2023, 3:21 PM UTC

codetrotter wrote: post #21

mabbo wrote: post #16

> But think: everything in those describe blocks had to be written by hand. It also had to be thought about by the developer. Someone had to say "I want the code to do this under these conditions". If your tests can be autogenerated then they aren't verifying expected behaviour , they're just locking in your implementation such that it can't change later. They are saying "hey look everyone, I got my coverage metric t…

One of the projects at a place where I have worked was set up so that when you ran the tests it automatically and silently updated the values that were expected. Completely bonkers because the first time I was contributing to the project I prepared the tests first and then started the implementation, and then while I was working on it I ran the tests which at this point should fail because I hadn’t finished writing t…

So long as the diffs get reviewed and checked in, this is a great form of testing called "regression testing". It doesn't replace unit testing, but it can be super valuable.

Re: “Expect tests” make test-writing feel like a REPL session

#49

Post by cormacrelf » Sat, Jan 14, 2023, 3:26 PM UTC

IshKebab wrote: post #37

angio wrote: post #24

Earlier quoted context omitted.

The title mentions writing tests as if they are repl sessions because you're supposed to iterate until you have the correct result.

How do you know if you have the right result though? You might know if you have a plausible result. Like if it output -1 then you know something is wrong I guess. There's a much higher chance of detecting bugs that give plausible output if you aren't given the opportunity to say "eh looks plausible I won't bother double checking it".

Any programmer dumb enough to just blindly accept that their program is correct is also a dumb enough programmer not to have begun writing a test in the first place. If this gets the friction of writing a test at all so close to zero that these programmers start writing tests (albeit sometimes blindly accepting the output), then it's better than just trying their program on some inputs and calling it a day. It writes down the current output of the program. That's a big step up already. Now people evaluating the code can read some of its outputs without downloading anything.

I personally already use a similar cycle to expect-test when I write tests. A great place to start when writing test assertions is the debug output, just like this thing uses. Then you convert the output into assertions after you have thought through which parts are right or wrong. Just like you can do with expect-test, but without the automation. If you don't know whether the output is right or not, just add an assert(false, "hmm, not sure about this") aka todo!() and voilà, your test fails and future you can be prompted to check over it again.

Sometimes the output is obviously wrong, but you still don't know what the right output is. (At this point you know you're doing useful work!) The remedy is the same. Just make the test fail somehow.

Re: “Expect tests” make test-writing feel like a REPL session

#50

Post by teeray » Sat, Jan 14, 2023, 3:35 PM UTC

codetrotter wrote: post #21

mabbo wrote: post #16

> But think: everything in those describe blocks had to be written by hand. It also had to be thought about by the developer. Someone had to say "I want the code to do this under these conditions". If your tests can be autogenerated then they aren't verifying expected behaviour , they're just locking in your implementation such that it can't change later. They are saying "hey look everyone, I got my coverage metric t…

One of the projects at a place where I have worked was set up so that when you ran the tests it automatically and silently updated the values that were expected. Completely bonkers because the first time I was contributing to the project I prepared the tests first and then started the implementation, and then while I was working on it I ran the tests which at this point should fail because I hadn’t finished writing t…

I can somewhat understand, because this is kind of the goal of property based testing—the actual values themselves matter so little to the test that you’re willing to subject those inputs to randomness

That said, this doesn’t sound like a very good way to pull that off because the developer has no control over that randomness (where it’s needed greatly).