Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pynguin – Generate Python unit tests automatically

github.com

Post reply on HN
Print view View original

71–80 of 80 posts

Re: Pynguin – Generate Python unit tests automatically

#71

Post by motoboi »

one2three4 wrote: post #69

>> Pynguin can cause serious harm to your computer, for example, wipe your entire hard disk! We recommend running Pynguin in an isolated environment, for example, a Docker container, to minimise the risk of damaging your system. That sounds ominous. Why do they execute the generated tests though? I mean I'd expect this to go only as far as generating the tests. Also that means that at least someone needs to review th…

They provide arguments for your code, runs it and record the output.

Based on the input types, it creates a hyperdimensional surface which is then searched for possible output values.

So, it runs your code, and runs it a lot. If your code deletes /, it will delete it a lot.

Re: Pynguin – Generate Python unit tests automatically

#72

Post by motoboi »

wpietri wrote: post #70
mmmkkaaayy wrote: post #48

There wasn't much info on the GitHub repo about how it actually works. Here's the paper from the authors: https://arxiv.org/abs/2007.14049 From what I gather from the paper, they frame the problem of test generation as a search problem. An evolutionary algorithm randomly mutates a randomly generated test suite. The evolutionary algorithm optimizes for greater branch coverage. Excerpt from the abstract: "Our experimen…

I have trouble even understanding the point of this. To me "automatic test generation" seems as bad an idea as "automatic code generation". The point of tests for me is to express intent for what the software should do. The code then expresses the details of how we do it. I could see a use for evolutionary algorithms to probe code for hidden bugs. But even there it seems limited. I definitely see a use for things lik…

Automatic test generation in this case is not testing your hypothesis of how the code behaves, but rather records how the code actually does.

This can be very useful if you are dealing with legacy code bases or just don’t want to write tests yourself.

I agree that if your are going to write the tests, using hypothesis is a very wise decision.

Re: Pynguin – Generate Python unit tests automatically

#73

Post by CivBase »

It would be nice if the README included use cases for a tool like this.

In my experience with unit testing, the process of writing the unit tests is where the most value is derived from. It forces developers to more careful examine use cases and potential inputs for the unit under test (the latter being especially important for Python). Therefore I would never consider something like this for testing new code.

The rest of the value of unit tests mostly comes from validating the scope of a change's impact. If a unit tests which once passed fails after I make a change, it could mean my change affected some other part of the code in a way I did not expect. But in my experience it usually just means I overlooked something in the test suite - not that I broke something in the software itself. Auto-generated unit tests might be useful for validating the scope of a change's impact on old code, but I'd expect most failures to boil down to issues with the tests not getting updated correctly.

Where I imagine auto-generated unit tests being particularly useful is for ensuring functionality is not affected when working on bug fixes, security patches, optimizations, cruft busting, or simple refactoring.

Re: Pynguin – Generate Python unit tests automatically

#74

Post by wpietri »

motoboi wrote: post #72
wpietri wrote: post #70

Earlier quoted context omitted.

I have trouble even understanding the point of this. To me "automatic test generation" seems as bad an idea as "automatic code generation". The point of tests for me is to express intent for what the software should do. The code then expresses the details of how we do it. I could see a use for evolutionary algorithms to probe code for hidden bugs. But even there it seems limited. I definitely see a use for things lik…

Automatic test generation in this case is not testing your hypothesis of how the code behaves, but rather records how the code actually does. This can be very useful if you are dealing with legacy code bases or just don’t want to write tests yourself. I agree that if your are going to write the tests, using hypothesis is a very wise decision.

Even there, the code already records how the code behaves. The question is always which of those behaviors are intentional, meaningful. I am unable to fathom the utility of recording an arbitrary slice of behaviors, because those are very likely to be orthogonal or even contrary to the actual intent of the system. For example, every bug is part of how the code actually behaves.

I would much rather inherit a code base with zero tests than ones automatically generated by people who "just don't want to write tests".

Re: Pynguin – Generate Python unit tests automatically

#75

Post by usagitoneko97 »

Just skimmed through the paper: https://arxiv.org/pdf/2007.14049.pdf, it seems like they are using some sort of heuristically mutating a test suite until it's fully branch covered, based on calculating `branch distance` for each predicate. Why isn't a SMT solver like Z3-solver being used here to solve for the predicate (generating inputs to evaluate to true/false)? Since it's getting so powerful, and python container/dict/string(regex) operation can also be modeled conveniently.

And I'm also wondering, whether there is a return based automatic test generation, that start from the return value, and resolve all variables used and gather all possible return values with its constraint, and feed those to z3 to generate inputs to cover. It seems like it will help with branch explosion by eliminating unused branch, and only focus on branch that is being used.

Edit: it looks like CrossHair[0] is a similar tool that uses Z3 to find counter examples for predicates.

[0] https://github.com/pschanely/CrossHair

Re: Pynguin – Generate Python unit tests automatically

#76

Post by helges »

sverhagen wrote: post #8

> mature tools exist—for statically typed languages, such as Java Is that really a thing Java developers use? I am one, I remember doing a PoC with Jtest (now Parasoft) in 2008, and it was utterly useless, never come across any such since, I'd be genuinely interested to learn more (I'll go duckduckgo now).

They probably refer to EvoSuite, which is/was mainly developed by one of the authors of the paper: https://www.evosuite.org/

Re: Pynguin – Generate Python unit tests automatically

#77

Post by motoboi »

wpietri wrote: post #74
motoboi wrote: post #72

Earlier quoted context omitted.

Automatic test generation in this case is not testing your hypothesis of how the code behaves, but rather records how the code actually does. This can be very useful if you are dealing with legacy code bases or just don’t want to write tests yourself. I agree that if your are going to write the tests, using hypothesis is a very wise decision.

Even there, the code already records how the code behaves. The question is always which of those behaviors are intentional, meaningful. I am unable to fathom the utility of recording an arbitrary slice of behaviors, because those are very likely to be orthogonal or even contrary to the actual intent of the system. For example, every bug is part of how the code actually behaves. I would much rather inherit a code base…

A test fixes the behavior. When later the need to change the code arises, you have a meaningful way of comparing results.

Any test is better than none, I’m my opinion.

Re: Pynguin – Generate Python unit tests automatically

#78

Post by wpietri »

motoboi wrote: post #77
wpietri wrote: post #74

Earlier quoted context omitted.

Even there, the code already records how the code behaves. The question is always which of those behaviors are intentional, meaningful. I am unable to fathom the utility of recording an arbitrary slice of behaviors, because those are very likely to be orthogonal or even contrary to the actual intent of the system. For example, every bug is part of how the code actually behaves. I would much rather inherit a code base…

A test fixes the behavior. When later the need to change the code arises, you have a meaningful way of comparing results. Any test is better than none, I’m my opinion.

That's definitely not true. Tests that lock in a bug or an accidental epiphenomenon of code are demonstrably worse, because they actively discourage improvement.

Re: Pynguin – Generate Python unit tests automatically

#79

Post by ptx »

kortilla wrote: post #56
ptx wrote: post #46

Earlier quoted context omitted.

This warning quoted from the README doesn't seem to suggest anything unexpected or unusual. Obviously running the tests would execute the module under test. (Although I suppose if someone wanted to generate tests without ever executing them the warning would be relevant.)

This warning is very important. It’s absolutely not expected that generating tests means executing module code, side effects and all.

You wouldn't expect it to load the module to inspect it? Loading a module executes top-level code.

Re: Pynguin – Generate Python unit tests automatically

#80

Post by kortilla »

ptx wrote: post #79
kortilla wrote: post #56

Earlier quoted context omitted.

This warning is very important. It’s absolutely not expected that generating tests means executing module code, side effects and all.

You wouldn't expect it to load the module to inspect it? Loading a module executes top-level code.

You’re confusing loading the module vs executing every method inside. I would not expect loading a module that has dangerous methods to do the destructive thing by default.
Post reply on HN