Who is going to test that the mutators are doing the right kind of mutations? lol I’ve worked on code that has 1000’s of tests and is still a bug ridden hell.
99% code coverage (2017)
21–30 of 99 posts
Re: 99% code coverage (2017)
#22The author suggests to add mutations to the code, like replacing ' I think this might apply when you write your own algorithms and want to test them. But if you are, like most of us, working on Business Logic, then you are probably writing the wrong tests. We usually want to know whether a workflow or a customer story works as intended and as such we should write more Integration tests. The idea itself has merit thou…
'There are some cases where it's not a big deal. 'Some other examples might involve accelerometer data, pressure sensors, GPS boundaries, debounce algorithms, animations, etc. But I think it's just a good habit to write test cases that test those boundary values.
Re: 99% code coverage (2017)
#23I hadn't realized this mutation testing existed as automated tooling. I'll be looking more into it. Traditionally, I've gone with "sabotaging" the code when writing unit tests; altering the code to verify a test goes from red to green or vise versa. Never trust a test that has never failed.
> Traditionally, I've gone with "sabotaging" the code when writing unit tests; altering the code to verify a test goes from red to green or vise versa. Never trust a test that has never failed.
That's completely backwards compared to how you should be doing things.
You either write the test:
1. to verify the existence of a bug by reproducing it, or
2. to formalize the spec for yet-to-be implemented code/feature.
And then you make the test green. Retroactively writing tests for working code, only to sabotage the code... Seems like an odd way of doing things.
Re: 99% code coverage (2017)
#24The author suggests to add mutations to the code, like replacing ' I think this might apply when you write your own algorithms and want to test them. But if you are, like most of us, working on Business Logic, then you are probably writing the wrong tests. We usually want to know whether a workflow or a customer story works as intended and as such we should write more Integration tests. The idea itself has merit thou…
You think customers aren't affected if you mistakenly replace a < with a <= in your code?
while (car.wheels Re: 99% code coverage (2017)
#25The industry is obsessed with getting 100% unit test code coverage even though it doesn't mean anything to the project. The purpose of unit tests is to lock down the project's source code once it's essentially completed; to avoid regressions when making minor changes. If you start writing unit tests too early in the project, you're effecively locking down units of code which haven't yet proved themseves to be useful…
What is an unit? Is it the wheel or the car? When you start to see that the car is the unit the difference between an unit test and an integration test gets blurry.
It does not matter. Are your tests preventing defects? Is it possible to run new or affected tests quickly? Are your tests reasonably easy to maintain?
When I am testing an application that is basically a layer between my app and a database, it does not make sense to mock the database. What then am I actually testing.
When I am testing a package that is pure logic, it makes sense to testing it in isolation.
When I am building a user interface that is in flux, the best tests are manual.
My definition of a unit test: The smallest piece of functionality that can be tested in isolation and provide value.
This does not test a damned thing: assert.equal(dbQuery, "SELECT * FROM users")
Re: 99% code coverage (2017)
#26I hadn't realized this mutation testing existed as automated tooling. I'll be looking more into it. Traditionally, I've gone with "sabotaging" the code when writing unit tests; altering the code to verify a test goes from red to green or vise versa. Never trust a test that has never failed.
> Traditionally, I've gone with "sabotaging" the code when writing unit tests; altering the code to verify a test goes from red to green or vise versa. Never trust a test that has never failed. That's completely backwards compared to how you should be doing things. You either write the test: 1. to verify the existence of a bug by reproducing it, or 2. to formalize the spec for yet-to-be implemented code/feature. And…
You don't always write a test first. A bug? Sure, write a test that shows the bug, fix it, make it green. A very well defined something? Sure, TDD. Many times, you will write code (or come across pre-existing code) and tests come last.
Re: 99% code coverage (2017)
#27Earlier quoted context omitted.
What is an unit? Is it the wheel or the car? When you start to see that the car is the unit the difference between an unit test and an integration test gets blurry.
The basic rule I go by is whether or not the test is self-contained. For example if you are making a trip to a database or hitting a 3rd party dll then it’s an integration test. I thought this was pretty widely accepted.
I would say that for most CRUD applications it is much easier to have the tests run against an sqlite database instead of mocking it, so would let unit tests update the database and write tests to check if the data in the database match what I would expect from the input. If you change how you store the data, it isn't a side effect of changes you are doing but the main effect so the test could be rewritten anyway. Some call that an integration test
Re: 99% code coverage (2017)
#28I hadn't realized this mutation testing existed as automated tooling. I'll be looking more into it. Traditionally, I've gone with "sabotaging" the code when writing unit tests; altering the code to verify a test goes from red to green or vise versa. Never trust a test that has never failed.
> Traditionally, I've gone with "sabotaging" the code when writing unit tests; altering the code to verify a test goes from red to green or vise versa. Never trust a test that has never failed. That's completely backwards compared to how you should be doing things. You either write the test: 1. to verify the existence of a bug by reproducing it, or 2. to formalize the spec for yet-to-be implemented code/feature. And…
Some people feel more comfortable writing code without tests first, gradually shaping final design of the interface. Especially if the tests involve a lot of stubbing and mocking of things like Redis and Sphinx Search, or messing with crypto tokens, parsing HTML, freezing time, setting up global config attributes like support emails, interaction between 2 different databases, etc. Tests are code as well and oftentimes might feel heavier than production code. You can of course say that there is a way things should be no matter what but that might lead to negative emotions, toxicity and complaining instead of getting stuff done.
Re: 99% code coverage (2017)
#29Earlier quoted context omitted.
What is an unit? Is it the wheel or the car? When you start to see that the car is the unit the difference between an unit test and an integration test gets blurry.
Unit tests should test individual subcomponents of your system in isolation. If your project is to build a car from scratch, then the unit cannot be the car. If your project is to build an autonomous fleet of self-driving cars, then from your project's perspective, the car could be a unit; the project to build the autonomous car would have different units from the project which manages the fleet of cars.
Exactly, context matter. Most cases when some is writing for or against unit testing they miss the part about finding what they define as an unit. If the class need other classes to work, the whole set can be seen as one unit.
Who didn't want to test a private method before they see that a unit is more than a function?
Re: 99% code coverage (2017)
#30I had a similar experience, you always get what you measure. IMO the best a approach is to first integrate the test coverage in the code reviews, cause there is no hard rule [0] and second write property-based tests [1]. [0] http://www.se-radio.net/2018/05/se-radio-episode-324-marc-ho... [1] Sample framework for JavaScript https://github.com/jsverify/jsverify
I prefer just setting coverage requirements to 100%, and give developers the freedom to mark code as ignored for coverage (e.g. /* istanbul ignore next */ for many Javascript applications). That way, the annotation is something that can be brought up during code reviews in case the reviewer does not agree with that not being covered, and it doesn't depend on the reviewer having to remember to run or look at a separate coverage report.
(I wrote more about this here: https://vincenttunru.com/100-percent-coverage/)