Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Expect tests” make test-writing feel like a REPL session

blog.janestreet.com

Post reply on HN
Print view View original

51–60 of 93 posts

Re: “Expect tests” make test-writing feel like a REPL session

#51

Post by avgcorrection »

superb-owl wrote: post #48
codetrotter wrote: post #21

Earlier quoted context omitted.

One of the projects at a place where I have worked was set up so that when you ran the tests it automatically and silently updated the values that were expected. Completely bonkers because the first time I was contributing to the project I prepared the tests first and then started the implementation, and then while I was working on it I ran the tests which at this point should fail because I hadn’t finished writing t…

So long as the diffs get reviewed and checked in, this is a great form of testing called "regression testing". It doesn't replace unit testing, but it can be super valuable.

What’s described in the OP (Jane Street) is regression testing.

What the commenter just described is tautology testing: whatever result of the computation I get is what I expected.

Re: “Expect tests” make test-writing feel like a REPL session

#52

Post by bmitc »

I don’t really understand this. How is this different from just writing the code and just assuming that you got it correct, and then locking in a potentially wrong implementation?

> What does fibonacci(15) equal? If you already know, terrific—but what are you meant to do if you don’t?

> I think you’re supposed to write some nonsense, like assert fibonacci(15) == 8, then when the test says “WRONG! Expected 8, got 610”, you’re supposed to copy and paste the 610 from your terminal buffer into your editor.

Who does that? How do you know 610 is correct? That’s just assuming your implementation is right from the get go. For such a function, I’d independently calculate it, using some method I trust (maybe Wolfram Alpha). I’d do this for a handful of examples, trying to cover base and extreme cases. And then I’d do property testing if I really wanted good coverage. Further, this expect test library seems to just smoothen the experience of copying what the function returns into a test.

This whole “expect test” business seems to rely on the developer looking at what the function returns for a given input, evaluating if it’s correct or not and then locking that in as “this is what this function is supposed to do”. That seems backwards and no different from how one implements functions in the first place, so I don’t know what is actually being tested.

The entire point of testing is saying “this is what this function should do” and not “this is what the function did and thus that’s what it should always do”.

Re: “Expect tests” make test-writing feel like a REPL session

#53

Post by avgcorrection »

avgcorrection wrote: post #26
Temporary_31337 wrote: post #18

Earlier quoted context omitted.

Do you simply mean regression testing?

Yeah, weird to see all these variations - Aha, an expect test! - Oh, you mean a snapshot test! - This here is akin to UI testing framework X where the test framework can compare an expected screenshot of the UI to a screenshot of the actual UI! The last one basically requires automation if you want anyone to make use of it. The regression testing automation described in the OP is a nice-to-have, not a so-good-that-it…

… And apparently also “change detector test” https://news.ycombinator.com/item?id=34379175

Re: “Expect tests” make test-writing feel like a REPL session

#54

Post by WastingMyTime89 »

mabbo wrote: post #16

> But think: everything in those describe blocks had to be written by hand. It also had to be thought about by the developer. Someone had to say "I want the code to do this under these conditions". If your tests can be autogenerated then they aren't verifying expected behaviour , they're just locking in your implementation such that it can't change later. They are saying "hey look everyone, I got my coverage metric t…

You are missing the point entirely. It’s actually discussed at length in the article btw if you had bothered reading it.

Regression tests are extremely useful because you don’t want working code to get broken but they are tedious to write. What the author is describing is pretty much how everyone does it if you want anything moderately complex in the test, you just run and then copy-paste. Having something do it for you in a frictionless way is a huge win.

Plus the way the framework works you can still test expected behaviours before writing the code if that’s what you actually want.

Re: “Expect tests” make test-writing feel like a REPL session

#55

Post by chubot »

mabbo wrote: post #32
codetrotter wrote: post #21

Earlier quoted context omitted.

One of the projects at a place where I have worked was set up so that when you ran the tests it automatically and silently updated the values that were expected. Completely bonkers because the first time I was contributing to the project I prepared the tests first and then started the implementation, and then while I was working on it I ran the tests which at this point should fail because I hadn’t finished writing t…

I keep rereading this hoping I'm misunderstanding. That is cargo cult level behaviour. They know that software with lots of tests tend to have few bugs, so let's automatically have lots of tests! I just hope whatever you were building wasn't critical to human lives. https://en.m.wikipedia.org/wiki/Cargo_cult

I hate to dwell on this, but I've also seen it in real life and it boggles the mind.

Like "give review feedback that this code isn't doing the right thing" -> "change the test to make it pass, not change the code to make it work". And it wasn't really a small case where you could plausibly do that and still understand what you were trying to do.

Coincidentally that was a few weeks after I saw a comment here on HN about someone who hired someone from Facebook, and the guy would change the tests so he could push to production, rather than fixing the bug that the tests pointed out ...

So yes it happens.

Re: “Expect tests” make test-writing feel like a REPL session

#56

Post by angio »

IshKebab wrote: post #37
angio wrote: post #24

Earlier quoted context omitted.

The title mentions writing tests as if they are repl sessions because you're supposed to iterate until you have the correct result.

How do you know if you have the right result though? You might know if you have a plausible result. Like if it output -1 then you know something is wrong I guess. There's a much higher chance of detecting bugs that give plausible output if you aren't given the opportunity to say "eh looks plausible I won't bother double checking it".

It's a repl, so you build the final output incrementally. Testing becomes part of the development workflow like you would do in languages that rely on the repl like lisps.

For example, you start with the inputs and you apply the first layer of transformations, then check what it does makes sense. Then maybe you refactor it out in its own function and add the generated test for it. Then you move on the next step and so on until you have the final result.

Re: “Expect tests” make test-writing feel like a REPL session

#57

Post by IshKebab »

cormacrelf wrote: post #49
IshKebab wrote: post #37

Earlier quoted context omitted.

How do you know if you have the right result though? You might know if you have a plausible result. Like if it output -1 then you know something is wrong I guess. There's a much higher chance of detecting bugs that give plausible output if you aren't given the opportunity to say "eh looks plausible I won't bother double checking it".

Any programmer dumb enough to just blindly accept that their program is correct is also a dumb enough programmer not to have begun writing a test in the first place. If this gets the friction of writing a test at all so close to zero that these programmers start writing tests (albeit sometimes blindly accepting the output), then it's better than just trying their program on some inputs and calling it a day. It writes…

> Any programmer dumb enough to just blindly accept that their program is correct is also a dumb enough programmer not to have begun writing a test in the first place.

Then what's the point of this methodology? It requires you to write tests and also blindly accept that your program is correct.

Maybe they should just rename it to "plausibility tests" or similar because that's what they're really testing. And while that does have some value, I think most of the value is negated by the fact that it sounds like they are properly vetted tests which they are not.

So a more appropriate name would help a lot. I still think it's a bad idea though.

Re: “Expect tests” make test-writing feel like a REPL session

#58

Post by angio »

bmitc wrote: post #52

I don’t really understand this. How is this different from just writing the code and just assuming that you got it correct, and then locking in a potentially wrong implementation? > What does fibonacci(15) equal? If you already know, terrific—but what are you meant to do if you don’t? > I think you’re supposed to write some nonsense, like assert fibonacci(15) == 8, then when the test says “WRONG! Expected 8, got 610”…

You're supposed to use it as a repl, so you start with a test for `fib(1) = 1`, then `fib(2)` and so on. Once you're confident of your implementation, you use quickcheck to test general properties of the system.

Similarly if you find a bug in the live system, you add a test for that and the initial output will be wrong. Then you fix your code until it prints the correct value and commit that so any regression will be caught.

Re: “Expect tests” make test-writing feel like a REPL session

#60

Post by cormacrelf »

IshKebab wrote: post #57
cormacrelf wrote: post #49

Earlier quoted context omitted.

Any programmer dumb enough to just blindly accept that their program is correct is also a dumb enough programmer not to have begun writing a test in the first place. If this gets the friction of writing a test at all so close to zero that these programmers start writing tests (albeit sometimes blindly accepting the output), then it's better than just trying their program on some inputs and calling it a day. It writes…

> Any programmer dumb enough to just blindly accept that their program is correct is also a dumb enough programmer not to have begun writing a test in the first place. Then what's the point of this methodology? It requires you to write tests and also blindly accept that your program is correct. Maybe they should just rename it to "plausibility tests" or similar because that's what they're really testing. And while th…

> It requires you to write tests and also blindly accept that your program is correct.

No. You can say no. Just don’t accept it. You’re a human and it asks. Even if you do accept it you can modify it because you have eyes and a keyboard and it’s written right there where you wrote your test.

See https://github.com/rust-analyzer/expect-test for a demo gif of the rust version.

Post reply on HN