Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Expect tests” make test-writing feel like a REPL session

blog.janestreet.com

Post reply on HN
Print view View original

31–40 of 93 posts

Re: “Expect tests” make test-writing feel like a REPL session

#31

Post by travisjungroth »

codetrotter wrote: post #21
mabbo wrote: post #16

> But think: everything in those describe blocks had to be written by hand. It also had to be thought about by the developer. Someone had to say "I want the code to do this under these conditions". If your tests can be autogenerated then they aren't verifying expected behaviour , they're just locking in your implementation such that it can't change later. They are saying "hey look everyone, I got my coverage metric t…

One of the projects at a place where I have worked was set up so that when you ran the tests it automatically and silently updated the values that were expected. Completely bonkers because the first time I was contributing to the project I prepared the tests first and then started the implementation, and then while I was working on it I ran the tests which at this point should fail because I hadn’t finished writing t…

Would it do this just the first time? It’s still bad it was doing this silently, but it’s pretty common to test web APIs in a similar way manually. Make a request, check the response you get back looks right (important step) and then save it as the expected value.

Edit: or after reading the article, like in the article.

Re: “Expect tests” make test-writing feel like a REPL session

#32

Post by mabbo »

codetrotter wrote: post #21
mabbo wrote: post #16

> But think: everything in those describe blocks had to be written by hand. It also had to be thought about by the developer. Someone had to say "I want the code to do this under these conditions". If your tests can be autogenerated then they aren't verifying expected behaviour , they're just locking in your implementation such that it can't change later. They are saying "hey look everyone, I got my coverage metric t…

One of the projects at a place where I have worked was set up so that when you ran the tests it automatically and silently updated the values that were expected. Completely bonkers because the first time I was contributing to the project I prepared the tests first and then started the implementation, and then while I was working on it I ran the tests which at this point should fail because I hadn’t finished writing t…

I keep rereading this hoping I'm misunderstanding.

That is cargo cult level behaviour. They know that software with lots of tests tend to have few bugs, so let's automatically have lots of tests!

I just hope whatever you were building wasn't critical to human lives.

https://en.m.wikipedia.org/wiki/Cargo_cult

Re: “Expect tests” make test-writing feel like a REPL session

#33

Post by codetrotter »

travisjungroth wrote: post #31
codetrotter wrote: post #21

Earlier quoted context omitted.

One of the projects at a place where I have worked was set up so that when you ran the tests it automatically and silently updated the values that were expected. Completely bonkers because the first time I was contributing to the project I prepared the tests first and then started the implementation, and then while I was working on it I ran the tests which at this point should fail because I hadn’t finished writing t…

Would it do this just the first time? It’s still bad it was doing this silently, but it’s pretty common to test web APIs in a similar way manually. Make a request, check the response you get back looks right (important step) and then save it as the expected value. Edit: or after reading the article, like in the article.

It did this every time, not just the first time.

Re: “Expect tests” make test-writing feel like a REPL session

#34

Post by travisjungroth »

c-cube wrote: post #17
eru wrote: post #10

Earlier quoted context omitted.

Well, the non-insane thing is to do property-based testing. Instead of testing only a handful of examples.

They also do that, the post refers to their Quickcheck library. But how do you property test the Fibonacci function ? There isn't much to say about it...

Properties of the Fibonacci function:

It is non-decreasing monotonic. fib(n) It is increasing monotonic after 1. fib(n) Its domain and codomain are non-negative integers.

fib(n) + fib(n+1) == fib(n+2) Notice this is like the recursive solution except going the other way (addition not subtraction) and is missing the base case.

Re: “Expect tests” make test-writing feel like a REPL session

#35

Post by travisjungroth »

codetrotter wrote: post #33
travisjungroth wrote: post #31

Earlier quoted context omitted.

Would it do this just the first time? It’s still bad it was doing this silently, but it’s pretty common to test web APIs in a similar way manually. Make a request, check the response you get back looks right (important step) and then save it as the expected value. Edit: or after reading the article, like in the article.

It did this every time, not just the first time.

Well, you know what they say: Expect the unexpected!

Re: “Expect tests” make test-writing feel like a REPL session

#36

Post by MereInterest »

davidgrenier wrote: post #27
c-cube wrote: post #17

Earlier quoted context omitted.

They also do that, the post refers to their Quickcheck library. But how do you property test the Fibonacci function ? There isn't much to say about it...

You could compare against a closed-form solution: let fib2 n = let sq5 = sqrt(5.0) ((1.0 + sq5)/2.0)**(float n)/sq5 |> round |> int

The problem is that the closed-form solution is vulnerable to floating-point error. If the calculations are done in float32 (including all intermediate steps), then the 32nd fibonacci number is erroneously given as 2178310, instead of the correct value of 2178309. Using float64 does better, but still has an error at the 71st fibonacci number. (I made a quick plot of the error as a function of N at https://i.imgur.com/bbc9OFC.png. As soon as the error crossed ±0.5, the rounding results in the wrong result.)

These are fine for property-based testing, so long as you restrict yourself to the range in which you have a correct value. But at that point, you might as well just hard-code the first 93 fibonacci numbers (the most that will fit in a uint64_t) and be done with it.

Re: “Expect tests” make test-writing feel like a REPL session

#37

Post by IshKebab »

angio wrote: post #24
IshKebab wrote: post #22

Earlier quoted context omitted.

Yeah in their Fibonacci example if it printed out 510 instead of 610 you'd still have a bug and think you had tested it. Especially confusing for future people who will assume it works because there are passing tests!

The title mentions writing tests as if they are repl sessions because you're supposed to iterate until you have the correct result.

How do you know if you have the right result though? You might know if you have a plausible result. Like if it output -1 then you know something is wrong I guess.

There's a much higher chance of detecting bugs that give plausible output if you aren't given the opportunity to say "eh looks plausible I won't bother double checking it".

Re: “Expect tests” make test-writing feel like a REPL session

#38

Post by mannykannot »

eru wrote: post #12
scotty79 wrote: post #8

Doesn't this approach make you update results of failing tests wholesale and possibly miss where a new result of some test is actually wrong? https://docs.rs/expect-test/latest/expect_test/

At Google the nickname for these kinds of tests was 'change detector tests'.

If you are saying this approach would tend to produce a lot of change-detector tests, then that is an issue, but I think scotty79 is making a different point: this approach would seem to make it easy to overlook any regressions that the latest change has created.

Re: “Expect tests” make test-writing feel like a REPL session

#39

Post by mannykannot »

mabbo wrote: post #16

> But think: everything in those describe blocks had to be written by hand. It also had to be thought about by the developer. Someone had to say "I want the code to do this under these conditions". If your tests can be autogenerated then they aren't verifying expected behaviour , they're just locking in your implementation such that it can't change later. They are saying "hey look everyone, I got my coverage metric t…

See the related memes "code never lies", "the code is the contract" and “when I use a word, it means just what I choose it to mean — neither more nor less."

Re: “Expect tests” make test-writing feel like a REPL session

#40

Post by MichaelBurge »

thaumasiotes wrote: post #11
avgcorrection wrote: post #6

> I think you’re supposed to write some nonsense, like assert fibonacci(15) == 8, then when the test says “WRONG! Expected 8, got 610”, you’re supposed to copy and paste the 610 from your terminal buffer into your editor. > This is insane! The sane approach is presumably to either expand the call tree and verify all the unique subsolutions. Or to do every step with a calculator if you can’t expand the call tree. > Th…

Yes, I have difficulty understanding the point of a test-writing system that relies on your explicit assumption that whatever the code already does is correct. What are you testing? Why?

A regression test is checking causality: Changes in new code, updating dependencies, updating the OS the software is running on, updating shared libraries, porting the code to a new platform, etc. aren't supposed to change the test results.

"I may not know what cos(x) means, but whatever it is shouldn't depend on what OS version I'm running"

Post reply on HN