Solving Algorithmic Problems in Python with Pytest (2019) • Hacker News Board

Re: Solving Algorithmic Problems in Python with Pytest (2019)

#21

Post by nanis » Sat, Dec 19, 2020, 2:23 PM UTC

mlthoughts2018 wrote: post #13

abdullahkhalids wrote: post #9

How would you write tests for the following problem with random output? Write a function biasedcoin(n,p) that takes the number of coin flips, n, and the probability of heads, p. It flips a biased coin n times, and returns the ratio of number of heads/number of tails. p is guaranteed to have two significant numbers eg. p = 0.60 or p = 0.74. You should use the random.randrange function to generate random numbers. Examp…

Simple: use mocker.patch to patch out random.randrange and have it return a fixed sequence of results that would imply a known result. Your test should not depend on the internal workings of randrange or anything relying on unfixed random state. Your test is only checking if, given correct results from randrange (or any other external world source of random draws) that the rest of your function correctly produces the…

> Simple: use mocker.patch to patch out random.randrange and have it return a fixed sequence of results that would imply a known result.

Thank you for this. Another way to put this is "your tests for code that uses a 3rd party API should not amount to an uptime test for the 3rd party API."

Some time ago, I observed that tests in the Boost.Accumulators library were similarly confounding tests for the code one wrote vs tests of statistical hypotheses:

https://www.nu42.com/2016/12/cpp-boost-median-test.html

Re: Solving Algorithmic Problems in Python with Pytest (2019)

#22

Post by Fiveplus » Sat, Dec 19, 2020, 2:45 PM UTC

morelandjs wrote: post #16

Side remark, but I really wish interviewers gave me the option of a terminal with vim since that’s pretty much standard on any Unix machine I might ssh into these days. All too often someone puts me in some weird IDE and I feel like a cat with boots on.

Someone on our school's group chat shared they had a technical interview where she was asked to write code on a shared google doc. I really hope that was an outlier.

Re: Solving Algorithmic Problems in Python with Pytest (2019)

#23

Post by mrfox321 » Sat, Dec 19, 2020, 2:51 PM UTC

Fiveplus wrote: post #22

morelandjs wrote: post #16

Side remark, but I really wish interviewers gave me the option of a terminal with vim since that’s pretty much standard on any Unix machine I might ssh into these days. All too often someone puts me in some weird IDE and I feel like a cat with boots on.

Someone on our school's group chat shared they had a technical interview where she was asked to write code on a shared google doc. I really hope that was an outlier.

Google did this during the pandemic

Are other companies copying them to the Tee?

Re: Solving Algorithmic Problems in Python with Pytest (2019)

#24

Post by tcollier » Sat, Dec 19, 2020, 2:56 PM UTC

henrikeh wrote: post #14

abdullahkhalids wrote: post #9

How would you write tests for the following problem with random output? Write a function biasedcoin(n,p) that takes the number of coin flips, n, and the probability of heads, p. It flips a biased coin n times, and returns the ratio of number of heads/number of tails. p is guaranteed to have two significant numbers eg. p = 0.60 or p = 0.74. You should use the random.randrange function to generate random numbers. Examp…

You could approach this a number of ways. First, if you can ensure the seed of the generator, then that could be used on a case by case basis. But this tricky, since it introduces a subtle dependency on the implementation of biasedcoin. Another option is to split the function/process into a stochastic and deterministic part. Test the deterministic part thoroughly. biasedcoin is too trivial for this, but it works nice…

The statistical approach definitely gives a developer more confidence in the correctness of a solution and guards against regressions. Though any test that checks the outcome of a random process against statistical measures is expected to fail occasionally.

For example, if the test flips the coin 1,000 times, there is better than a 99% chance that the outcome would result in 450 to 550 heads. So if you write the test using `450 Having said that, I still find statistical tests to be very helpful when building out code that uses randomness. However, these tests typically do not make it into the CI/CD pipeline.

Re: Solving Algorithmic Problems in Python with Pytest (2019)

#25

Post by jononor » Sat, Dec 19, 2020, 3:00 PM UTC

Yajirobe wrote: post #19

ploika wrote: post #10

Earlier quoted context omitted.

It's doable once you're out of pure experimentation and into the development phase at which test driven development can help. Test that this ETL function expects a DataFrame with a given schema and returns one with a different (but also known) schema, even with all these edge cases in the filters and group-bys. Test that the "train_classifier" method/function rejects negative penalisation parameters, returns an objec…

And how do you test that your model has sufficient accuracy? How do you make sure that your model does not deteriorate over time?

assert test_performance > threshold ? This would only be a sanity check for big deviations though. For more subtle changes one may need to do a proper statistical test. I believe the jury is still out of how to do that properly for deep ML models - challenge is the lack of independence in CV folds and generally the compute time it takes to evaluate.

However, I would probably not do performance check inside a unit-testing framework. Instead treat this as quality indicators like performance benchmarks, code coverage etc. It may be a "gate", that needs to pass to allow a new model into production.

To evaluate performance over time, one would preferably want labeled datasets for test gathered at different points in time. Which requires a (reliable) continuous labeling process. One can also gather customer feedback about performance, track those as metrics. These things are probably more in the "monitoring" part of a system, rather than unit-testing time though.

Re: Solving Algorithmic Problems in Python with Pytest (2019)

#26

Post by madsbuch » Sat, Dec 19, 2020, 3:19 PM UTC

abdullahkhalids wrote: post #18

mlthoughts2018 wrote: post #13

Earlier quoted context omitted.

Simple: use mocker.patch to patch out random.randrange and have it return a fixed sequence of results that would imply a known result. Your test should not depend on the internal workings of randrange or anything relying on unfixed random state. Your test is only checking if, given correct results from randrange (or any other external world source of random draws) that the rest of your function correctly produces the…

Thanks for your answer. With mocking, what happens if someone writes their if condition as: (if randrange(1000) (1000-1000p). I am trying to write tests to autograde some HW assignments, and so I really have to think adversarially and imagine my students writing the weirdest yet correctly working functions. I really haven't come up with a better answer than checking the statistics of the returned value by running the…

Sure, this is simple! You provide them with a definition written in Coq and make them implement a proof that their implementation satisfies that definition. Use probabilistic couplings to make sure the output of the biased coin function adheres to specified distribution.

You can easily implement a grader that checks the the proof.

Re: Solving Algorithmic Problems in Python with Pytest (2019)

#27

Post by profunctor » Sat, Dec 19, 2020, 3:27 PM UTC

Yajirobe wrote: post #19

ploika wrote: post #10

Earlier quoted context omitted.

It's doable once you're out of pure experimentation and into the development phase at which test driven development can help. Test that this ETL function expects a DataFrame with a given schema and returns one with a different (but also known) schema, even with all these edge cases in the filters and group-bys. Test that the "train_classifier" method/function rejects negative penalisation parameters, returns an objec…

And how do you test that your model has sufficient accuracy? How do you make sure that your model does not deteriorate over time?

You can test on a tester of use some of the examples from the checklist paper. In that paper they might add "I hate you" to some random data and assert that sentiment doesn't improve.

Re: Solving Algorithmic Problems in Python with Pytest (2019)

#28

Post by inertiatic » Sat, Dec 19, 2020, 3:32 PM UTC

mlthoughts2018 wrote: post #13

abdullahkhalids wrote: post #9

How would you write tests for the following problem with random output? Write a function biasedcoin(n,p) that takes the number of coin flips, n, and the probability of heads, p. It flips a biased coin n times, and returns the ratio of number of heads/number of tails. p is guaranteed to have two significant numbers eg. p = 0.60 or p = 0.74. You should use the random.randrange function to generate random numbers. Examp…

Simple: use mocker.patch to patch out random.randrange and have it return a fixed sequence of results that would imply a known result. Your test should not depend on the internal workings of randrange or anything relying on unfixed random state. Your test is only checking if, given correct results from randrange (or any other external world source of random draws) that the rest of your function correctly produces the…

Mocks should be used when the call you depend on is hard to get to behave in the ways you need for your test.

"Randrange" isn't hard to get to behave in the way you want. In fact, the way you want it to behave here is all it does!

I'd argue if what you want is to do engineering, mocking this isn't what you'd want to do. You want to ensure your code works. And your test can tell you that it works, even if "randrange" works differently while maintaining the same signature after you update the module it came from.

If you're doing software engineering for aviation for example, I doubt "well, my tests using mocks passed, it's just that the dependency broke it's contract" is a good enough excuse for a catastrophe.

Re: Solving Algorithmic Problems in Python with Pytest (2019)

#29

Post by konjin » Sat, Dec 19, 2020, 3:34 PM UTC

Fiveplus wrote: post #22

morelandjs wrote: post #16

Side remark, but I really wish interviewers gave me the option of a terminal with vim since that’s pretty much standard on any Unix machine I might ssh into these days. All too often someone puts me in some weird IDE and I feel like a cat with boots on.

Someone on our school's group chat shared they had a technical interview where she was asked to write code on a shared google doc. I really hope that was an outlier.

I had someone do that to me a few weeks back.

I told them flat out no, gave them my ip, made an account on my local machine and started a shared tmux session over ssh. Since that's my usual ide I didn't lose any capability.

Re: Solving Algorithmic Problems in Python with Pytest (2019)

#30

Post by smallnamespace » Sat, Dec 19, 2020, 3:41 PM UTC

tcollier wrote: post #24

henrikeh wrote: post #14

Earlier quoted context omitted.

You could approach this a number of ways. First, if you can ensure the seed of the generator, then that could be used on a case by case basis. But this tricky, since it introduces a subtle dependency on the implementation of biasedcoin. Another option is to split the function/process into a stochastic and deterministic part. Test the deterministic part thoroughly. biasedcoin is too trivial for this, but it works nice…

The statistical approach definitely gives a developer more confidence in the correctness of a solution and guards against regressions. Though any test that checks the outcome of a random process against statistical measures is expected to fail occasionally. For example, if the test flips the coin 1,000 times, there is better than a 99% chance that the outcome would result in 450 to 550 heads. So if you write the test…

In practice there is no such tradeoff, the tails of a Gaussian fall off extremely rapidly so if you simply 10x the number of iterations you’ll have failure become astronomically unlikely.