Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Solving Algorithmic Problems in Python with Pytest (2019)

adamj.eu

Post reply on HN
Print view View original

11–20 of 45 posts

Re: Solving Algorithmic Problems in Python with Pytest (2019)

#11

Post by barefeg »

Yajirobe wrote: post #6
0x008 wrote: post #4

Test driven development is just such a good habit to get into. You trade just a little bit of ramp-up speed at the beginning of an implementation for a massive reduction in cognitive overhead.

I'd love to see how test driven development looks like for ML systems. You write the test for your non-existent model, then write the model and try to train the model to 'pass' the test? Do you also train it on the test case or on other data only?

But isn’t that how models are trained anyway? The training data provides the “test cases”. How would you write a test that doesn’t fit that paradigm?

Re: Solving Algorithmic Problems in Python with Pytest (2019)

#12

Post by barefeg »

ploika wrote: post #10
Yajirobe wrote: post #6

Earlier quoted context omitted.

I'd love to see how test driven development looks like for ML systems. You write the test for your non-existent model, then write the model and try to train the model to 'pass' the test? Do you also train it on the test case or on other data only?

It's doable once you're out of pure experimentation and into the development phase at which test driven development can help. Test that this ETL function expects a DataFrame with a given schema and returns one with a different (but also known) schema, even with all these edge cases in the filters and group-bys. Test that the "train_classifier" method/function rejects negative penalisation parameters, returns an objec…

These type tests are not really model tests but of the infrastructure surrounding the model. MLOps tools are tested like normal software development

Re: Solving Algorithmic Problems in Python with Pytest (2019)

#13

Post by mlthoughts2018 »

abdullahkhalids wrote: post #9

How would you write tests for the following problem with random output? Write a function biasedcoin(n,p) that takes the number of coin flips, n, and the probability of heads, p. It flips a biased coin n times, and returns the ratio of number of heads/number of tails. p is guaranteed to have two significant numbers eg. p = 0.60 or p = 0.74. You should use the random.randrange function to generate random numbers. Examp…

Simple: use mocker.patch to patch out random.randrange and have it return a fixed sequence of results that would imply a known result.

Your test should not depend on the internal workings of randrange or anything relying on unfixed random state. Your test is only checking if, given correct results from randrange (or any other external world source of random draws) that the rest of your function correctly produces the two-digit bias number.

Otherwise you are just testing randrange itself.

If you are asking how to test a pseudorandom number generator, you have a few choices. You can either fix the random seed and test the algorithm’s precise implementation on a large number of known results. Or you can define tests statistically with margins of error, for example testing the entropy in a series of uniformly generated bits or the resulting distribution in a series of random draws from a fixed list, and decide what level of precision is tolerable before considering a test failed.

Re: Solving Algorithmic Problems in Python with Pytest (2019)

#14

Post by henrikeh »

abdullahkhalids wrote: post #9

How would you write tests for the following problem with random output? Write a function biasedcoin(n,p) that takes the number of coin flips, n, and the probability of heads, p. It flips a biased coin n times, and returns the ratio of number of heads/number of tails. p is guaranteed to have two significant numbers eg. p = 0.60 or p = 0.74. You should use the random.randrange function to generate random numbers. Examp…

You could approach this a number of ways.

First, if you can ensure the seed of the generator, then that could be used on a case by case basis. But this tricky, since it introduces a subtle dependency on the implementation of biasedcoin.

Another option is to split the function/process into a stochastic and deterministic part. Test the deterministic part thoroughly. biasedcoin is too trivial for this, but it works nicely in more complicated setups.

For biasedcoin, I would probably go with a statistical approach. Given a certain bias, the expected value and distribution is known. I would simply test that several trials of the function lie within some bound.

Re: Solving Algorithmic Problems in Python with Pytest (2019)

#15

Post by mlthoughts2018 »

rational_indian wrote: post #8

Why do you need the minimum == 0 test in the following chunk of code ... if i > 0 and (minimum == 0 or i

The stated problem requires returning the smallest integer greater than 0 in the list. 0 will only be the running minimum if only non-positive numbers have been seen so far, and since i is greater than zero in the condition, if minimum == 0 it means i is the first positive integer we’ve seen, so even though i > minimum (i > 0) we need to swap i as the minimum at that point.

Re: Solving Algorithmic Problems in Python with Pytest (2019)

#17

Post by matsemann »

abdullahkhalids wrote: post #9

How would you write tests for the following problem with random output? Write a function biasedcoin(n,p) that takes the number of coin flips, n, and the probability of heads, p. It flips a biased coin n times, and returns the ratio of number of heads/number of tails. p is guaranteed to have two significant numbers eg. p = 0.60 or p = 0.74. You should use the random.randrange function to generate random numbers. Examp…

If it were a non-trivial example I would split it in two: One function to generate the data, and one function to count and do the statistics. Then you can pass the second one known datasets and verify it gives you correct statistics, and at least be confident that that part is correct.

Of course that still leaves the generation and randomness part. You could make a separate function for numberToTailOrHead(...) that you can test. Or maybe you should have the ability to send in which source of randomness it uses. And then you can provide a fake random implementation that tests the edge cases (0, 0.5, 0.999).

Re: Solving Algorithmic Problems in Python with Pytest (2019)

#18

Post by abdullahkhalids »

mlthoughts2018 wrote: post #13
abdullahkhalids wrote: post #9

How would you write tests for the following problem with random output? Write a function biasedcoin(n,p) that takes the number of coin flips, n, and the probability of heads, p. It flips a biased coin n times, and returns the ratio of number of heads/number of tails. p is guaranteed to have two significant numbers eg. p = 0.60 or p = 0.74. You should use the random.randrange function to generate random numbers. Examp…

Simple: use mocker.patch to patch out random.randrange and have it return a fixed sequence of results that would imply a known result. Your test should not depend on the internal workings of randrange or anything relying on unfixed random state. Your test is only checking if, given correct results from randrange (or any other external world source of random draws) that the rest of your function correctly produces the…

Thanks for your answer. With mocking, what happens if someone writes their if condition as: (if randrange(1000) (1000-1000p).

I am trying to write tests to autograde some HW assignments, and so I really have to think adversarially and imagine my students writing the weirdest yet correctly working functions.

I really haven't come up with a better answer than checking the statistics of the returned value by running the function a whole bunch of times, like you suggest.

Re: Solving Algorithmic Problems in Python with Pytest (2019)

#19

Post by Yajirobe »

ploika wrote: post #10
Yajirobe wrote: post #6

Earlier quoted context omitted.

I'd love to see how test driven development looks like for ML systems. You write the test for your non-existent model, then write the model and try to train the model to 'pass' the test? Do you also train it on the test case or on other data only?

It's doable once you're out of pure experimentation and into the development phase at which test driven development can help. Test that this ETL function expects a DataFrame with a given schema and returns one with a different (but also known) schema, even with all these edge cases in the filters and group-bys. Test that the "train_classifier" method/function rejects negative penalisation parameters, returns an objec…

And how do you test that your model has sufficient accuracy? How do you make sure that your model does not deteriorate over time?

Re: Solving Algorithmic Problems in Python with Pytest (2019)

#20

Post by mlthoughts2018 »

abdullahkhalids wrote: post #18
mlthoughts2018 wrote: post #13

Earlier quoted context omitted.

Simple: use mocker.patch to patch out random.randrange and have it return a fixed sequence of results that would imply a known result. Your test should not depend on the internal workings of randrange or anything relying on unfixed random state. Your test is only checking if, given correct results from randrange (or any other external world source of random draws) that the rest of your function correctly produces the…

Thanks for your answer. With mocking, what happens if someone writes their if condition as: (if randrange(1000) (1000-1000p). I am trying to write tests to autograde some HW assignments, and so I really have to think adversarially and imagine my students writing the weirdest yet correctly working functions. I really haven't come up with a better answer than checking the statistics of the returned value by running the…

“How do I test all possible equivalent definitions of this function” is probably not practically answerable for this. What if they wrote this?

    def biasedcoin(n, p):
        for answer in answer_key[n][p]:
            yield answer
Or this:

    def biasedcoin(n, p):
        while True:
            mine_bitcoin_on_teachers_computer()
Post reply on HN