Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

99% code coverage (2017)

rachelcarmena.github.io

Post reply on HN
Print view View original

91–99 of 99 posts

Re: 99% code coverage (2017)

#91

Post by twodave »

rypskar wrote: post #27
twodave wrote: post #13

Earlier quoted context omitted.

The basic rule I go by is whether or not the test is self-contained. For example if you are making a trip to a database or hitting a 3rd party dll then it’s an integration test. I thought this was pretty widely accepted.

I would say that for most CRUD applications it is much easier to have the tests run against an sqlite database instead of mocking it, so would let unit tests update the database and write tests to check if the data in the database match what I would expect from the input. If you change how you store the data, it isn't a side effect of changes you are doing but the main effect so the test could be rewritten anyway. So…

I wouldn’t mock a data layer unless I was unit testing (for example) response codes on a route depending on the data layer. In those cases you aren’t interested in the data as much as how it is handled. I like to be able to point an integration test at a “clean” database in order to test a data layer. In this way you can have separate tests to basically ensure your API responses and database queries are all valid given your database is in a valid state and the API request is valid. With this kind of setup I rarely actually have to run an application anymore to feel it is in good working condition, and that saves me a LOT of time.

Re: 99% code coverage (2017)

#92

Post by blueyed »

adrianN wrote: post #6
Vinnl wrote: post #5

Earlier quoted context omitted.

I haven't worked with it extensively, but from what I've seen, it's far beyond what most teams should consider. It's basically a more thorough method of measuring test coverage, pointing out cases that have not been covered by your tests yet. However, the number of tests that it would have you write to reach 100% mutation coverage is not justified by the number of bugs you'll catch, unless the impact of any bug is ve…

I think mutation testing really shines in code bases that are already heavily tested, because it let's you discover test cases that you don't actually need. Tests are a burden since you have to adapt them when you change the behavior. With mutation testing you can prune your test code by identifying tests that test very similar behavior.

Would that be done based on which (different) tests killed a particular mutant?

Often an integration test would catch the multiple mutations also being caught by (different) unit tests.

I assume that you mean that if a certain (broader) test kills the same mutants as X unit tests, those X tests are not really necessary?

I've looked into https://github.com/boxed/mutmut and https://github.com/sixty-north/cosmic-ray for Python project, and there it is only important that a mutant gets killed, but not how often and by which tests (therefore you can use `pytest -x` to continue with the next mutation after the first test failed due to it).

Re: 99% code coverage (2017)

#95

Post by dnautics »

Macha wrote: post #86
dnautics wrote: post #85

Earlier quoted context omitted.

can't you run these feature tests in parallel? I'm runnning a system where it full on downloads and mounts containers, and executes them (in a loopback, they get hosted by transient webservers on the test host aka my laptop, but possibly also travis), about 50 of them in parallel with tons of database calls, and it usually takes around 10-20s to complete.

It only takes you 20s to start up 50 containers? I doubt I could start 50 alpine containers in that time on my company issue 2015 MBP (Edit: A quick test reveals my personal desktop can in fact accomplish this, but it's much more powerful than my work MBP and has no virtualization overhead since it's running native docker instead of docker for mac). I don't know about jeffasinger's company, but in optimal circumstanc…

Singularity containers, my dev laptop is Linux, way less overhead than docker on mac. I'm simulating request load on a scheduler for compute jobs. In principle those jobs are not generally local, but it helps to stresstest the scheduler.

Re: 99% code coverage (2017)

#96

Post by closeparen »

JoeAltmaier wrote: post #38
austincheney wrote: post #37

Earlier quoted context omitted.

It drives me nuts during interviews talking about test automation because people are very particular about the type of testing whether its unit testing, integration testing, acceptance testing, or whatever. In my mind you only need 1 kind of testing: feature tests. Does the application provide the expected output for a given input and/or configuration. The application does all that it claims to do in a very precise w…

In a complex system a feature test can detect a problem but not diagnose it. For that a unit test is desirable. The hole you get into is, lots of units are changed and a feature quits working and everybody says 'its not me!' and it doesn't get fixed.

Your development process should not allow someone to merge a change that fails the test suite, nor one that is too large to reason about.

Re: 99% code coverage (2017)

#97

Post by jbergens »

jondubois wrote: post #14
rypskar wrote: post #10

Earlier quoted context omitted.

What is an unit? Is it the wheel or the car? When you start to see that the car is the unit the difference between an unit test and an integration test gets blurry.

Unit tests should test individual subcomponents of your system in isolation. If your project is to build a car from scratch, then the unit cannot be the car. If your project is to build an autonomous fleet of self-driving cars, then from your project's perspective, the car could be a unit; the project to build the autonomous car would have different units from the project which manages the fleet of cars.

This means that the unit tests will miss a lot of possible errors. Nothing wrong with that, just use integration testing or functional tests to cover the other part.

Re: 99% code coverage (2017)

#98

Post by quacker »

austincheney wrote: post #37
cryptica wrote: post #9

The industry is obsessed with getting 100% unit test code coverage even though it doesn't mean anything to the project. The purpose of unit tests is to lock down the project's source code once it's essentially completed; to avoid regressions when making minor changes. If you start writing unit tests too early in the project, you're effecively locking down units of code which haven't yet proved themseves to be useful…

It drives me nuts during interviews talking about test automation because people are very particular about the type of testing whether its unit testing, integration testing, acceptance testing, or whatever. In my mind you only need 1 kind of testing: feature tests. Does the application provide the expected output for a given input and/or configuration. The application does all that it claims to do in a very precise w…

I tend to agree, but in practice this doesn't work well if the feature is slow.

I was a tester on a project where the primary functionality was a workflow that took 40 minutes minimum to complete, often much longer (it involved migrating servers from one cloud provider to another), and the dev team had this same philosophy: no writing unit tests, just acceptance/feature tests run against live environments, which seemed okay - it was the start of the project, a new codebase, and it probably worked out well on a previous project.

But it was kind of a disaster on this project. With only feature testing: a dev makes a change, updated code is deployed for test, automated acceptance tests kick off and an hour later the tests fail, then they look through the app logs and finally discover a NoneType error or a KeyError or something (Python's dynamic typing did not help either). Then they fix the code and repeat the whole process. Hours later you've got one change tested and merged. Then it goes out to a pre-prod environment for in-depth feature testing with more permutations of inputs, and we'd find trivially broken code there too.

That's a worst case scenario, but it happened all the time. It just took way too long to run all the feature tests needed to get good coverage of code paths. We'd literally waste full days trying to get something out to production because of the slow dev/test cycle. Eventually, the devs caved and added unit tests because we ran into so much broken code.

I never appreciated unit tests much until after that project, especially for dynamically typed languages. It's the quickest and easiest way to weed out trivial issues in the code - human eyes will not catch everything in peer review and feature tests with a sufficient number of input permutations can take way too long to run.

Re: 99% code coverage (2017)

#99

Post by luord »

I can't imagine, as a developer, writing tests without assertions. Any developer who does that is actively making the situation worse for the team and the application.

Conversely, I've worked in teams whose managers encouraged the developers to forgo tests for the sake of delivering faster. Never again.

Post reply on HN