Hypothesis 1.0: A property-based testing library for Python

The search strategy is described in the docs [1]. I wonder if they could capitalize on the algorithm NIST uses in their ACTS software [2]? It's great for maximizing time spent testing parameters in N different dimensions.

[1] http://hypothesis.readthedocs.org/en/latest/details.html#sea... [2] http://csrc.nist.gov/groups/SNS/acts/

Re: Hypothesis 1.0: A property-based testing library for Python

#5

Post by DRMacIver » Mon, Mar 30, 2015, 6:34 PM UTC

wyldfire wrote: post #4

The search strategy is described in the docs [1]. I wonder if they could capitalize on the algorithm NIST uses in their ACTS software [2]? It's great for maximizing time spent testing parameters in N different dimensions. [1] http://hypothesis.readthedocs.org/en/latest/details.html#sea... [2] http://csrc.nist.gov/groups/SNS/acts/

This looks super interesting and I'll check it out when I'm not doing release management stuff, thanks!

If you haven't seen it, http://hypothesis.readthedocs.org/en/latest/internals.html#p... describes the current algorithm for generation. It's smarter than is typical for quickcheck, but could definitely be improved.

Re: Hypothesis 1.0: A property-based testing library for Python

#6

Post by lgierth » Mon, Mar 30, 2015, 7:06 PM UTC

Is this similar to Mutation Testing, where the code being tested is mutated, in order to identify unspecified behavior?

Mutant is a Ruby library for this: https://github.com/mbj/mutant

Re: Hypothesis 1.0: A property-based testing library for Python

#7

Post by DRMacIver » Mon, Mar 30, 2015, 7:22 PM UTC

lgierth wrote: post #6

Is this similar to Mutation Testing, where the code being tested is mutated, in order to identify unspecified behavior? Mutant is a Ruby library for this: https://github.com/mbj/mutant

No, it's much closer to classic fuzz testing, where the test is held constant and the data fed to it is varied. I've been meaning to see if I can figure out a way to use mutation testing to feed into Hypothesis, but it's on the long list of things I'll try "at some point"

Re: Hypothesis 1.0: A property-based testing library for Python

#8

Post by bowyakka » Mon, Mar 30, 2015, 7:26 PM UTC

This library is awesome, I am a massive fan of randomized and quick-check style testing. This type of testing has for me found so many bugs its surreal.

I love the fact that hypothesis is a proper quickcheck and not just the fuzzing part, and that the author has a crazy stack-based-vm-language-testing thing project that might be useful.

I have been using this library with py.test for 6 months now and its been a godsend.

If I ever meet the author I would buy him beers.

Re: Hypothesis 1.0: A property-based testing library for Python

#9

Post by DRMacIver » Mon, Mar 30, 2015, 7:32 PM UTC

bowyakka wrote: post #8

This library is awesome, I am a massive fan of randomized and quick-check style testing. This type of testing has for me found so many bugs its surreal. I love the fact that hypothesis is a proper quickcheck and not just the fuzzing part, and that the author has a crazy stack-based-vm-language-testing thing project that might be useful. I have been using this library with py.test for 6 months now and its been a godse…

Thanks so much for the kind words!

The best part of this message is that if you were someone I knew and thus could trust to only say nice things about me you would offer to buy me gin instead of beer. ;-)

Re: Hypothesis 1.0: A property-based testing library for Python

#10

Post by leondutoit » Mon, Mar 30, 2015, 7:35 PM UTC

Thanks for a great library, can't wait to use it.