Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Expect tests” make test-writing feel like a REPL session

blog.janestreet.com

Post reply on HN
Print view View original

21–30 of 93 posts

Re: “Expect tests” make test-writing feel like a REPL session

#21

Post by codetrotter »

mabbo wrote: post #16

> But think: everything in those describe blocks had to be written by hand. It also had to be thought about by the developer. Someone had to say "I want the code to do this under these conditions". If your tests can be autogenerated then they aren't verifying expected behaviour , they're just locking in your implementation such that it can't change later. They are saying "hey look everyone, I got my coverage metric t…

One of the projects at a place where I have worked was set up so that when you ran the tests it automatically and silently updated the values that were expected. Completely bonkers because the first time I was contributing to the project I prepared the tests first and then started the implementation, and then while I was working on it I ran the tests which at this point should fail because I hadn’t finished writing the code but instead all tests passed. Because helpfully the test setup overwrote the expected values that I had prepared in my new tests, with the bad data. Yeah great, very helpful >:(

Oh yeah and the whole test setup was also way too tied to the implementation rather than verifying behaviour. Complete trash the whole thing.

Re: “Expect tests” make test-writing feel like a REPL session

#22

Post by IshKebab »

mabbo wrote: post #16

> But think: everything in those describe blocks had to be written by hand. It also had to be thought about by the developer. Someone had to say "I want the code to do this under these conditions". If your tests can be autogenerated then they aren't verifying expected behaviour , they're just locking in your implementation such that it can't change later. They are saying "hey look everyone, I got my coverage metric t…

Yeah in their Fibonacci example if it printed out 510 instead of 610 you'd still have a bug and think you had tested it. Especially confusing for future people who will assume it works because there are passing tests!

Re: “Expect tests” make test-writing feel like a REPL session

#23

Post by angio »

c-cube wrote: post #17
eru wrote: post #10

Earlier quoted context omitted.

Well, the non-insane thing is to do property-based testing. Instead of testing only a handful of examples.

They also do that, the post refers to their Quickcheck library. But how do you property test the Fibonacci function ? There isn't much to say about it...

I believe the way to test it is to have a property like `n in integer | fib(n+2) == fib(n+1) + fib(n)`. This is close to the naive (but obviously correct?) implementation of fib and can be used to test the optimized version of the function.

You can also test that the sequence is increasing like `fib(n+2) > fib(n+1) > fib(n)`.

Re: “Expect tests” make test-writing feel like a REPL session

#24

Post by angio »

IshKebab wrote: post #22
mabbo wrote: post #16

> But think: everything in those describe blocks had to be written by hand. It also had to be thought about by the developer. Someone had to say "I want the code to do this under these conditions". If your tests can be autogenerated then they aren't verifying expected behaviour , they're just locking in your implementation such that it can't change later. They are saying "hey look everyone, I got my coverage metric t…

Yeah in their Fibonacci example if it printed out 510 instead of 610 you'd still have a bug and think you had tested it. Especially confusing for future people who will assume it works because there are passing tests!

The title mentions writing tests as if they are repl sessions because you're supposed to iterate until you have the correct result.

Re: “Expect tests” make test-writing feel like a REPL session

#25

Post by bottled_poe »

c-cube wrote: post #17
eru wrote: post #10

Earlier quoted context omitted.

Well, the non-insane thing is to do property-based testing. Instead of testing only a handful of examples.

They also do that, the post refers to their Quickcheck library. But how do you property test the Fibonacci function ? There isn't much to say about it...

Actually, Fibonacci results can be fairly precisely tested with the golden ratio ratio.

Re: “Expect tests” make test-writing feel like a REPL session

#26

Post by avgcorrection »

Temporary_31337 wrote: post #18
sesm wrote: post #14

Earlier quoted context omitted.

This looks similar to snapshot testing in UI, where you save an output of UI components and test system notifies you when the output changes. This can be useful to detect changes in components that you didn’t intend to change.

Do you simply mean regression testing?

Yeah, weird to see all these variations

- Aha, an expect test!

- Oh, you mean a snapshot test!

- This here is akin to UI testing framework X where the test framework can compare an expected screenshot of the UI to a screenshot of the actual UI!

The last one basically requires automation if you want anyone to make use of it. The regression testing automation described in the OP is a nice-to-have, not a so-good-that-it-gets-a-new-name.

Re: “Expect tests” make test-writing feel like a REPL session

#27

Post by davidgrenier »

c-cube wrote: post #17
eru wrote: post #10

Earlier quoted context omitted.

Well, the non-insane thing is to do property-based testing. Instead of testing only a handful of examples.

They also do that, the post refers to their Quickcheck library. But how do you property test the Fibonacci function ? There isn't much to say about it...

You could compare against a closed-form solution:

    let fib2 n =
        let sq5 = sqrt(5.0)
        ((1.0 + sq5)/2.0)**(float n)/sq5
        |> round |> int

Re: “Expect tests” make test-writing feel like a REPL session

#28

Post by pydry »

mabbo wrote: post #16

> But think: everything in those describe blocks had to be written by hand. It also had to be thought about by the developer. Someone had to say "I want the code to do this under these conditions". If your tests can be autogenerated then they aren't verifying expected behaviour , they're just locking in your implementation such that it can't change later. They are saying "hey look everyone, I got my coverage metric t…

For Fibonacci (or indeed the result of most mathematical calculations) it makes no sense but I use this kind of thing all the time where the expected output is, for example, a templated string like an error message.

There are plenty of kinds of test outputs where rewriting the test and eyeballing the result is quicker, easier and ultimately better.

Re: “Expect tests” make test-writing feel like a REPL session

#29

Post by avgcorrection »

ElliotH wrote: post #3

I wonder if this has the same downsides as golden and screenshot type tests, where you end up over-asserting resulting in tests that break for unrelated changes? Obviously that’s a risk for hand written tests too but it’s easier (today… who knows what copilot like systems will offer soon!) for a human to reason about what’s relevant.

Q: Why does this test assert the value X?

A: The value X was revealed to me by ChatGPT.

Re: “Expect tests” make test-writing feel like a REPL session

#30

Post by davidgrenier »

avgcorrection wrote: post #20
c-cube wrote: post #17

Earlier quoted context omitted.

They also do that, the post refers to their Quickcheck library. But how do you property test the Fibonacci function ? There isn't much to say about it...

You use the naive implementation as a test oracle, limit `n` to something small (through the property tester), and use the test oracle on your efficient implementation. Unit testing elegant functions has no value. (fib is often used as an example. But you asked how to test it.)

In combinatorics the adjusted fibonacci numbers start with 1 instead and is more commonly used as it aligns with many other results. One might want to document in the code, via a test, which sequence is of interest.

This is just an example of course but elegant functions might need to be tested.

Post reply on HN