Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Expect tests” make test-writing feel like a REPL session

blog.janestreet.com

Post reply on HN
Print view View original

11–20 of 93 posts

Re: “Expect tests” make test-writing feel like a REPL session

#11

Post by thaumasiotes »

avgcorrection wrote: post #6

> I think you’re supposed to write some nonsense, like assert fibonacci(15) == 8, then when the test says “WRONG! Expected 8, got 610”, you’re supposed to copy and paste the 610 from your terminal buffer into your editor. > This is insane! The sane approach is presumably to either expand the call tree and verify all the unique subsolutions. Or to do every step with a calculator if you can’t expand the call tree. > Th…

Yes, I have difficulty understanding the point of a test-writing system that relies on your explicit assumption that whatever the code already does is correct.

What are you testing? Why?

Re: “Expect tests” make test-writing feel like a REPL session

#12

Post by eru »

scotty79 wrote: post #8

Doesn't this approach make you update results of failing tests wholesale and possibly miss where a new result of some test is actually wrong? https://docs.rs/expect-test/latest/expect_test/

At Google the nickname for these kinds of tests was 'change detector tests'.

Re: “Expect tests” make test-writing feel like a REPL session

#13

Post by IanCal »

chii wrote: post #9
CGamesPlay wrote: post #2

Snapshot testing is great, and I wish more test frameworks included first-class support for them. This means that they can auto update with a flag, and can be stored either in the source inline or in an external file (both modes have different use cases). Note that doc tests can also be a form of this, e.g. in Python's. "Expect tests" seems like a bad name, since that covers all tests.

i find that snapshot testing gets overused in javascript - and mistakes can creep in easily, and if the snapshot is big, and in a separate file, code review can miss it. I much prefer property based testing over expectation based testing. You have to explicitly think about what properties hold true about the thing you're writing. For example, fib(N+1) = fib(N) + fib(N), so this property can be tested for all N; primi…

Every single time I've introduced property based testing, even as a simple example, I've discovered a bug in either the code or the spec.

I've found a bug in a Haskell program about fib generation - your test would work (if fixed for the subtractions) but incorrectly as there was an overflow in the addition. A basic property of "fib(n+1) > fib(n)" for n>1 finds this.

I like this type of testing as it asks you to more generally consider what guarantees your code is making about its operation.

Edit - your example is a good one and necessary, I just wanted to add a bit extra as I really like property based testing

Re: “Expect tests” make test-writing feel like a REPL session

#14

Post by sesm »

thaumasiotes wrote: post #11
avgcorrection wrote: post #6

> I think you’re supposed to write some nonsense, like assert fibonacci(15) == 8, then when the test says “WRONG! Expected 8, got 610”, you’re supposed to copy and paste the 610 from your terminal buffer into your editor. > This is insane! The sane approach is presumably to either expand the call tree and verify all the unique subsolutions. Or to do every step with a calculator if you can’t expand the call tree. > Th…

Yes, I have difficulty understanding the point of a test-writing system that relies on your explicit assumption that whatever the code already does is correct. What are you testing? Why?

This looks similar to snapshot testing in UI, where you save an output of UI components and test system notifies you when the output changes. This can be useful to detect changes in components that you didn’t intend to change.

Re: “Expect tests” make test-writing feel like a REPL session

#15

Post by sesm »

chii wrote: post #9
CGamesPlay wrote: post #2

Snapshot testing is great, and I wish more test frameworks included first-class support for them. This means that they can auto update with a flag, and can be stored either in the source inline or in an external file (both modes have different use cases). Note that doc tests can also be a form of this, e.g. in Python's. "Expect tests" seems like a bad name, since that covers all tests.

i find that snapshot testing gets overused in javascript - and mistakes can creep in easily, and if the snapshot is big, and in a separate file, code review can miss it. I much prefer property based testing over expectation based testing. You have to explicitly think about what properties hold true about the thing you're writing. For example, fib(N+1) = fib(N) + fib(N), so this property can be tested for all N; primi…

Snapshot testing works well for component systems, especially with storybook. There is a service called Chromatic that lets you diff component changes visually using storybook output.

Re: “Expect tests” make test-writing feel like a REPL session

#16

Post by mabbo »

> But think: everything in those describe blocks had to be written by hand.

It also had to be thought about by the developer. Someone had to say "I want the code to do this under these conditions".

If your tests can be autogenerated then they aren't verifying expected behaviour, they're just locking in your implementation such that it can't change later. They are saying "hey look everyone, I got my coverage metric to 100% (despite any bugs I may have)."

Re: “Expect tests” make test-writing feel like a REPL session

#17

Post by c-cube »

eru wrote: post #10
avgcorrection wrote: post #6

> I think you’re supposed to write some nonsense, like assert fibonacci(15) == 8, then when the test says “WRONG! Expected 8, got 610”, you’re supposed to copy and paste the 610 from your terminal buffer into your editor. > This is insane! The sane approach is presumably to either expand the call tree and verify all the unique subsolutions. Or to do every step with a calculator if you can’t expand the call tree. > Th…

Well, the non-insane thing is to do property-based testing. Instead of testing only a handful of examples.

They also do that, the post refers to their Quickcheck library. But how do you property test the Fibonacci function ? There isn't much to say about it...

Re: “Expect tests” make test-writing feel like a REPL session

#18

Post by Temporary_31337 »

sesm wrote: post #14
thaumasiotes wrote: post #11

Earlier quoted context omitted.

Yes, I have difficulty understanding the point of a test-writing system that relies on your explicit assumption that whatever the code already does is correct. What are you testing? Why?

This looks similar to snapshot testing in UI, where you save an output of UI components and test system notifies you when the output changes. This can be useful to detect changes in components that you didn’t intend to change.

Do you simply mean regression testing?

Re: “Expect tests” make test-writing feel like a REPL session

#20

Post by avgcorrection »

c-cube wrote: post #17
eru wrote: post #10

Earlier quoted context omitted.

Well, the non-insane thing is to do property-based testing. Instead of testing only a handful of examples.

They also do that, the post refers to their Quickcheck library. But how do you property test the Fibonacci function ? There isn't much to say about it...

You use the naive implementation as a test oracle, limit `n` to something small (through the property tester), and use the test oracle on your efficient implementation.

Unit testing elegant functions has no value.

(fib is often used as an example. But you asked how to test it.)

Post reply on HN