> Please consider reading the online documentation to start your Pynguin adventure.
Nice wink to Penguin Adventure, brings back very old memories.
Pynguin – Generate Python unit tests automatically
31–40 of 80 posts
Re: Pynguin – Generate Python unit tests automatically
#32Earlier quoted context omitted.
It does, but their documentation's Quickstart has an example: https://pynguin.readthedocs.io/en/latest/user/quickstart.htm...
This looks like they combined the exploration strategy of property based testing with unit testing. This is not even testing, it tries to find some examples that result in different code paths. There is no concept of "expected result, given a certain input" and it does not try to find extreme cases or equivalence classes. Using it will result in a heap of buggy code that has a useless "but tested" sticker.
I think you are thinking of the use case wrong. In my work I come across a lot of untested legacy code. An autogenerated tests would be like guardrails when working with these code bases. I am not going to work with them blindly, it doesn't even do variable names yet, but its boilerplate stuff I would have written anyways. If this code gives me extensive coverage, adding test cases for specific things becomes trivial.
Re: Pynguin – Generate Python unit tests automatically
#33Sometimes automated testing is the only way to go. Consider a case when you have a code that takes about 100 input parameters and makes a decision on whether something should or should not happen. The logic that determines the decision is about 1,000 LOC that doesn't divide easily into smaller chunks. The code consists of a lot of different conditions and calculations based on the input parameters and some constants.…
I'm intrigued by this example. What's an example of a domain or problem set that produces a function with 100 input parameters and a boolean result that can't be broken down cleanly?
This might not be quite the same, but logic systems -- we might have a logic expression on a hundred variables, and need to check if it is true.
While one would unit test with tiny expressions, often bugs only show up in large integration tests, as systems use many optimisations to cache and speed up behaviour, which can interact poorly
Re: Pynguin – Generate Python unit tests automatically
#34If I write faulty code, the generated test will make sure I keep the code faulty right?
Re: Pynguin – Generate Python unit tests automatically
#35Earlier quoted context omitted.
Slightly off-topic but I thought it might be interesting: you can easily modify tests to adapt to your changing architecture as well. Personally I use TDD to have an easy entry point to run through my code, so the first instance of a test for an api might only check status code It makes the process very iterative, as you can continuously run through your code just by saving
> you can easily modify tests to adapt to your changing architecture as well. Can you? Surely it depends; most systems end up with more LOC in the tests than the tested code, and the tests multiply the cost of refactoring. I've had success with the "happy path first" kind of TDD, where you write a use case and implement to that and only then come back and fill in other tests.
I might've been too unclear in my comment. I was indeed speaking specifically about the issue the parent comment had with writing detailed tests before the application logic is implemented, just to be forced to change them because the initial idea for the api had to be changed.
Changing existing apis comes with a very different set of challenges
Re: Pynguin – Generate Python unit tests automatically
#36I can see this being useful for refactoring legacy codebases. You create assertions for existing code, whether correct or wrong as a result of existing bugs, and then make sure nothing changes after your refactor.
For new code, not so much. The generated tests wouldn't prevent bugs by themselves, but they may uncover bugs if you see assertions that don't make sense. But you have to spend time reviewing the tests.
The tests also don't describe what they are testing in human terms so you have to refactor them before commiting.
They also would end up testing more than necessary. Sometimes the code behavior might be intentionally ambiguous, not expected to matter in the real world and behavior that will likely change in the future that you should not depend on today.
Re: Pynguin – Generate Python unit tests automatically
#37Earlier quoted context omitted.
The decision is actually 5 different variables. Types are: boolean, a decimal percent and integer being a monetary value. The domain is fintech: making a decision on how much money to give and some other aspects about the loan.
The whole thing does not sound very competent. Like noone could possibly guarantee that the result is not off by a factor of 2 or 3 or perhaps even 10 or more in some cases. What would happen in that case? It also sounds like a domain where mistakes could be very expensive or lead to lawsuits or similar such things.
Things are not that bad. While the whole thing is kind of a monolith, there are at least broad sections that split code into logical sections. With some practice, it is possible to easily narrow down changes and any issues to 10-20 LOC. Later, breaking code by them could be a starting place for refactoring. In the end, the decision is manually reviewed. There's also a log that tells how and why the code had arrived at the numbers. So, if you know that client cannot be X but the log says X then you know that there's a bug for example.
This code was around before I joined. It started small, but more and more requirements were added until it became like that. Not much effort went into structuring it. Now it's my responsibility, and I'm trying to make it better for me and anyone who will have to deal with it in the future.
Re: Pynguin – Generate Python unit tests automatically
#38Earlier quoted context omitted.
Slightly off-topic but I thought it might be interesting: you can easily modify tests to adapt to your changing architecture as well. Personally I use TDD to have an easy entry point to run through my code, so the first instance of a test for an api might only check status code It makes the process very iterative, as you can continuously run through your code just by saving
Let's say you test an api, since yoy are taling about status code. You have to test all kind of cases with wrong inputs, various user states and so on. You can of course write the test to simulate all that, but's way harder than just using your user UI which lets you discover it and put a breakpoint in the endpoint to inspect. I may not remember where to get the header data in this framework, or maybe the client send…
your whole argument is a straw-man.
i don't think that anyone would disagree that you can get caught up in writing too many tests too early, consequently loosing sight out of your objective and ending up with a bad implementation.
My point was that just writing super easy and happy-path tests at the start is a valid strategy for TDD as well, and does not cause any of your imaginary issues.
Re: Pynguin – Generate Python unit tests automatically
#39Sometimes automated testing is the only way to go. Consider a case when you have a code that takes about 100 input parameters and makes a decision on whether something should or should not happen. The logic that determines the decision is about 1,000 LOC that doesn't divide easily into smaller chunks. The code consists of a lot of different conditions and calculations based on the input parameters and some constants.…
In that situation, why do you WANT tests?
If, for example, someone intends to refactor the code into something more maintainable and wants tests to prevent any accidental changes in functionality, then this is useful.
If the goal is to catch bugs or to facilitate future development or to meet some test coverage standard then I argue you might be better off not having tests rather than having auto-generated tests that have not been carefully reviewed.
Re: Pynguin – Generate Python unit tests automatically
#40Just in case you are looking for an alternative approach: if you write contracts in your code, you might also consider crosshair [1] or icontract-hypothesis [2]. If your function/method does not need any pre-conditions then the the type annotations can be directly used. (I'm one of the authors of icontract-hypothesis.) [1] https://github.com/pschanely/CrossHair [2] https://github.com/mristin/icontract-hypothesis
Coding with contracts has always been interesting to me but I haven't had the option/time to try it seriously in a project. I assume you've had experience with it, how much productivity/code maintanability you gain compared to not using it (or using type annotations)?