Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Expect tests” make test-writing feel like a REPL session

blog.janestreet.com

Post reply on HN
Print view View original

71–80 of 93 posts

Re: “Expect tests” make test-writing feel like a REPL session

#71

Post by carry_bit »

mabbo wrote: post #16

> But think: everything in those describe blocks had to be written by hand. It also had to be thought about by the developer. Someone had to say "I want the code to do this under these conditions". If your tests can be autogenerated then they aren't verifying expected behaviour , they're just locking in your implementation such that it can't change later. They are saying "hey look everyone, I got my coverage metric t…

Think of it as manual testing where your work is captured so it can be ran later in an automated fashion. There are many problems where verifying the answer is easier than coming up with the answer.

Asserting formatted output can also be really useful. A picture might be worth a thousand words, but when it comes to tests it can save you a thousand asserts. Writing those thousand asserts separately also would be so tedious that in practice you'd probably not write them all, leaving part of your output uncovered by tests.

When I wrote a LALR parser generator for fun, I added some code to print out a nicely formatted parsing table with debugging information. Besides being useful for debugging, it let me write simple yet powerful tests: I would feed the generator a grammar and then assert on the formatted parsing table. That made it easy to verify that I was asserting the right thing, and let me assert everything in one go.

Re: “Expect tests” make test-writing feel like a REPL session

#72

Post by AndreasHae »

chubot wrote: post #55
mabbo wrote: post #32

Earlier quoted context omitted.

I keep rereading this hoping I'm misunderstanding. That is cargo cult level behaviour. They know that software with lots of tests tend to have few bugs, so let's automatically have lots of tests! I just hope whatever you were building wasn't critical to human lives. https://en.m.wikipedia.org/wiki/Cargo_cult

I hate to dwell on this, but I've also seen it in real life and it boggles the mind. Like "give review feedback that this code isn't doing the right thing" -> "change the test to make it pass, not change the code to make it work". And it wasn't really a small case where you could plausibly do that and still understand what you were trying to do. Coincidentally that was a few weeks after I saw a comment here on HN abo…

>Coincidentally that was a few weeks after I saw a comment here on HN about someone who hired someone from Facebook, and the guy would change the tests so he could push to production, rather than fixing the bug that the tests pointed out ...

Can't blame him, he moved fast and broke things /s

Re: “Expect tests” make test-writing feel like a REPL session

#73

Post by xtagon »

travisjungroth wrote: post #34
c-cube wrote: post #17

Earlier quoted context omitted.

They also do that, the post refers to their Quickcheck library. But how do you property test the Fibonacci function ? There isn't much to say about it...

Properties of the Fibonacci function: It is non-decreasing monotonic. fib(n) It is increasing monotonic after 1. fib(n) Its domain and codomain are non-negative integers. fib(n) + fib(n+1) == fib(n+2) Notice this is like the recursive solution except going the other way (addition not subtraction) and is missing the base case.

It also converges closer and closer to the golden ratio. Implementing a property test for that would be interesting.

Re: “Expect tests” make test-writing feel like a REPL session

#74

Post by SpicyLemonZest »

mabbo wrote: post #47
tantalor wrote: post #42

Earlier quoted context omitted.

> locking in your implementation such that it can't change later That's the whole point of tests. All tests do that. This protects against later code changes that change behavior (output or side effects) unintentionally . When you intend to change behavior then you need to change the tests tests too.

I disagree. Tests should define what the expectations are. If a change does not impact those expectations, then it should be allowed and not break any tests. Locking your code such that all future changes require updating old tests tells me that your tests are just your code written a second time, with no thought about what the code's requirements are.

In many contexts, there's just no such thing as a safe behavior change which should be allowed without a specific decision from you to allow it. As a database systems guy, I've seen countless examples of customer breakages caused by a developer's decision that some behavior or another is so trivial it doesn't need to be tested.

When you're working on developing a random utility function (real example!), it's easy to say "come on, it's no big deal to return DECIMAL(14, 4) instead of DECIMAL(12, 3)". It feels like they're basically the same, updating the test is make-work, and the guidelines saying you must document it as a breaking change are pointless annoyances. It's hard, requiring substantial amounts of knowledge and expertise, to recognize that this change will cause a production outage because the schema of a customer's view is no longer write-compatible with their existing data.

Re: “Expect tests” make test-writing feel like a REPL session

#75

Post by travisjungroth »

xtagon wrote: post #73
travisjungroth wrote: post #34

Earlier quoted context omitted.

Properties of the Fibonacci function: It is non-decreasing monotonic. fib(n) It is increasing monotonic after 1. fib(n) Its domain and codomain are non-negative integers. fib(n) + fib(n+1) == fib(n+2) Notice this is like the recursive solution except going the other way (addition not subtraction) and is missing the base case.

It also converges closer and closer to the golden ratio. Implementing a property test for that would be interesting.

    a, b, c = fib(n), fib(n+1), fib(n+2)
    assert abs(c / b - phi)

Re: “Expect tests” make test-writing feel like a REPL session

#76

Post by Viliam1234 »

mabbo wrote: post #32
codetrotter wrote: post #21

Earlier quoted context omitted.

One of the projects at a place where I have worked was set up so that when you ran the tests it automatically and silently updated the values that were expected. Completely bonkers because the first time I was contributing to the project I prepared the tests first and then started the implementation, and then while I was working on it I ran the tests which at this point should fail because I hadn’t finished writing t…

I keep rereading this hoping I'm misunderstanding. That is cargo cult level behaviour. They know that software with lots of tests tend to have few bugs, so let's automatically have lots of tests! I just hope whatever you were building wasn't critical to human lives. https://en.m.wikipedia.org/wiki/Cargo_cult

> That is cargo cult level behaviour.

One person's "cargo cult behavior" is another person's "best practices". :P

My favorite example is automatically generated documentation. The kind that merely repeats the name of the method, the names and types of arguments, and the type of return value. The ironic part is that this is later used as an evidence that all documentation is useless. Uhm, how about documenting the methods where something is not obvious, and leaving the obvious ones (getters, setters) alone? But then the documentation coverage checker would return a number smaller than 100% and someone would freak out...

This is just one of many examples, of course.

Re: “Expect tests” make test-writing feel like a REPL session

#77

Post by irthomasthomas »

chubot wrote: post #55
mabbo wrote: post #32

Earlier quoted context omitted.

I keep rereading this hoping I'm misunderstanding. That is cargo cult level behaviour. They know that software with lots of tests tend to have few bugs, so let's automatically have lots of tests! I just hope whatever you were building wasn't critical to human lives. https://en.m.wikipedia.org/wiki/Cargo_cult

I hate to dwell on this, but I've also seen it in real life and it boggles the mind. Like "give review feedback that this code isn't doing the right thing" -> "change the test to make it pass, not change the code to make it work". And it wasn't really a small case where you could plausibly do that and still understand what you were trying to do. Coincidentally that was a few weeks after I saw a comment here on HN abo…

Perhaps he's a Buddhist? "If the software is going to break, then the software will be broken." Then he adds a little wabi-sabi for good measure. https://en.wikipedia.org/wiki/Wabi-sabi

I remember once, using some in-house software, which for god knows why could not log it's errors back to the IT department. Instead, they relied on users to call up IT, or email them with the error. To make it more fun for users, each error message contained a humorous haiku.

  Chaos reigns within.
  Reflect, repent, and reboot.
  Order shall return.
Edit: Just found this from 2001 https://www.gnu.org/fun/jokes/error-haiku.en.html And my experience with haiku error messages at work was 01 or 02.

Re: “Expect tests” make test-writing feel like a REPL session

#78

Post by EamonnMR »

SpicyLemonZest wrote: post #74
mabbo wrote: post #47

Earlier quoted context omitted.

I disagree. Tests should define what the expectations are. If a change does not impact those expectations, then it should be allowed and not break any tests. Locking your code such that all future changes require updating old tests tells me that your tests are just your code written a second time, with no thought about what the code's requirements are.

In many contexts, there's just no such thing as a safe behavior change which should be allowed without a specific decision from you to allow it. As a database systems guy, I've seen countless examples of customer breakages caused by a developer's decision that some behavior or another is so trivial it doesn't need to be tested. When you're working on developing a random utility function (real example!), it's easy to…

In your story though the hapless dev just changed the test. And the reviewers approved it.

This suggests that there are so many changes to tests that it's just become background noise.

Re: “Expect tests” make test-writing feel like a REPL session

#79

Post by BoppreH »

Some years ago I wrote a Python function, "replace_me"[1], that edits the caller's source code. You can use it for code generation, inserting comments, generating fixed random seeds, etc.

And one more use case I found was exactly what TFA describes, but even easier:

   import replace_me
   replace_me.test(1+1)
Once executed, it evaluates the argument and becomes an assertion:

   import replace_me
   replace_me.test(1+1, 2)
I never actually used it for anything important, but it comes back to my mind once in a while.

[1]: https://github.com/boppreh/replace_me

Re: “Expect tests” make test-writing feel like a REPL session

#80

Post by polio »

pydry wrote: post #28
mabbo wrote: post #16

> But think: everything in those describe blocks had to be written by hand. It also had to be thought about by the developer. Someone had to say "I want the code to do this under these conditions". If your tests can be autogenerated then they aren't verifying expected behaviour , they're just locking in your implementation such that it can't change later. They are saying "hey look everyone, I got my coverage metric t…

For Fibonacci (or indeed the result of most mathematical calculations) it makes no sense but I use this kind of thing all the time where the expected output is, for example, a templated string like an error message. There are plenty of kinds of test outputs where rewriting the test and eyeballing the result is quicker, easier and ultimately better.

It makes sense in scenarios where it's easier to verify a provided solution than it is to create one.
Post reply on HN