Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

99% code coverage (2017)

rachelcarmena.github.io

Post reply on HN
Print view View original

81–90 of 99 posts

Re: 99% code coverage (2017)

#81

Post by dtech »

SkyPuncher wrote: post #59
cryptica wrote: post #9

The industry is obsessed with getting 100% unit test code coverage even though it doesn't mean anything to the project. The purpose of unit tests is to lock down the project's source code once it's essentially completed; to avoid regressions when making minor changes. If you start writing unit tests too early in the project, you're effecively locking down units of code which haven't yet proved themseves to be useful…

I agree. The project I'm working on hovers around 70% coverage. As far as I can tell, we'd likely loose productivity pushing for any better coverage. We get some bugs/issues that could be identified by better coverage, but most of our "bugs" stem from real world issues or differing views on functionality. Getting that right takes a ton of effort that simply isn't worth it right now.

Yep, testing definitely has a hugely decreasing marginal value.

I personally like the approach of writing a small amount of "obvious" tests to detect large functional breakage, and add a test for a bug once it's been identified aka a regression test.

Imo it's maximum test value per effort. Of course it depends on how impactful a bug could be.

Re: 99% code coverage (2017)

#82

Post by brlewis »

cryptica wrote: post #9

The industry is obsessed with getting 100% unit test code coverage even though it doesn't mean anything to the project. The purpose of unit tests is to lock down the project's source code once it's essentially completed; to avoid regressions when making minor changes. If you start writing unit tests too early in the project, you're effecively locking down units of code which haven't yet proved themseves to be useful…

It's true that obsessing over 100% code coverage is a bad thing.

It's false that the purpose of unit tests is to lock down the project's source code. Unit tests have multiple purposes, but that isn't one of them. The main purpose of unit tests is to allow developers to quickly reach confidence that code works, be it a code change or new code.

There is no such thing as writing unit tests too early.

Re: 99% code coverage (2017)

#83

Post by Macha »

austincheney wrote: post #76
Macha wrote: post #69

Earlier quoted context omitted.

We have 1000s of features. Real users don't use every feature every time they perform an action. A test case at the feature level will likely require several requests if its starting from clean state.

> A test case at the feature level will likely require several requests if its starting from clean state. You say request so I guess you are talking about a network service request. When I first got into A/B testing many years ago we found we could use it to simulate traffic through the entirety of the Travelocity web site from homepage to checkout and back. They never used the A/B test system for test automation, bu…

jeffasinger explained this point better than me, but one thing to point out is that I'm not arguing for unit tests to the exclusion of feature tests.

Rather, I'm arguing against feature tests to the exclusion of unit tests. Both are required in a decently tested systems.

Feature tests provide the assurance that the application works as expected and prevent broken changes going to production, as you've pointed out.

However, they're not a practical replacement for the instant feedback that unit tests provide. Our feature tests currently take 1.5->2.5 hours to run depending on CI system load.

That's... not great. On a given workday I get to try 3-4 builds? I don't know about how others work, but personally I like to get some feedback every like 10-25 lines of code. Unit tests provide that. They take 2 minutes to run. If I had to wait 1.5 hours every 25 lines of code, I would sure be much less productive.

Lets say I somehow convinced management to pay for 20x as many compute resources for CI so that I could get those results in 5-10 minutes rather than 1.5->2.5 hours. And 5-10 is the best case, as that's pretty close to the overhead for our CI system to just clone the code and get the secrets to deploy to a test environment and actually deploy to said test environment.

That's still my flow broken every time I need a test run, because I'm not going to sit and stare at build for 10 minutes, I'm going to answer that email from my product manager or review a team member's PR. If that build comes back with a failure, then I'm context switching back to actually fix it.

The point is that this is a funnel. The unit tests are for instant feedback and to catch obvious mistakes, like "Hey, that value might not be defined, you still need to calculate value B, hope you didn't remove/break the fallback calculation". This ensures that less mistakes incur the full 2.5 hour penalty of a CI build that ultimately fails a test. The feature tests are there for safety. The unit tests are there for productivity. The feature tests are "Does this feature work so we can deploy it". In theory you have at most 1-2 runs of these per feature as unit tests _should_ catch issues before they get there. They're simply too expensive in terms of time to be your first layer of defense, rather they're your final layer.

Re: 99% code coverage (2017)

#84

Post by progval »

nathan_f77 wrote: post #22
Radle wrote: post #2

The author suggests to add mutations to the code, like replacing ' I think this might apply when you write your own algorithms and want to test them. But if you are, like most of us, working on Business Logic, then you are probably writing the wrong tests. We usually want to know whether a workflow or a customer story works as intended and as such we should write more Integration tests. The idea itself has merit thou…

' There are some cases where it's not a big deal. ' Some other examples might involve accelerometer data, pressure sensors, GPS boundaries, debounce algorithms, animations, etc. But I think it's just a good habit to write test cases that test those boundary values.

Basically, it depends on whether we're dealing with floats or integers.

Re: 99% code coverage (2017)

#85

Post by dnautics »

jeffasinger wrote: post #79
austincheney wrote: post #62

Earlier quoted context omitted.

> but it's likely the test itself will take a significant amount of wall time to execute. Why? What about your application changed so that it is slower when testing compared to real world use? If anything it should be dramatically faster because people don't provide microsecond accurate automated responses. If the application naturally executes very quickly I would imagine it would take far long to set up the test sc…

Sure, let's go back to my payroll example. I need to make 8 HTTP requests to properly setup my test from a clean slate, including setting up the deductions, previous earnings, company location and employee home address. If each of those requests takes 50ms, and then I need to make a request to actually execute my test, and verify everything went well, that could easily be 500ms. And that's just what needs to happen t…

can't you run these feature tests in parallel? I'm runnning a system where it full on downloads and mounts containers, and executes them (in a loopback, they get hosted by transient webservers on the test host aka my laptop, but possibly also travis), about 50 of them in parallel with tons of database calls, and it usually takes around 10-20s to complete.

Re: 99% code coverage (2017)

#86

Post by Macha »

dnautics wrote: post #85
jeffasinger wrote: post #79

Earlier quoted context omitted.

Sure, let's go back to my payroll example. I need to make 8 HTTP requests to properly setup my test from a clean slate, including setting up the deductions, previous earnings, company location and employee home address. If each of those requests takes 50ms, and then I need to make a request to actually execute my test, and verify everything went well, that could easily be 500ms. And that's just what needs to happen t…

can't you run these feature tests in parallel? I'm runnning a system where it full on downloads and mounts containers, and executes them (in a loopback, they get hosted by transient webservers on the test host aka my laptop, but possibly also travis), about 50 of them in parallel with tons of database calls, and it usually takes around 10-20s to complete.

It only takes you 20s to start up 50 containers? I doubt I could start 50 alpine containers in that time on my company issue 2015 MBP (Edit: A quick test reveals my personal desktop can in fact accomplish this, but it's much more powerful than my work MBP and has no virtualization overhead since it's running native docker instead of docker for mac).

I don't know about jeffasinger's company, but in optimal circumstances, a test instance of our app running against a in memory h2 DB takes 3 minutes to start. The app internally already performs heavy lifting in parallel, so it's also not clear that running multiple instances of the app will make it that much faster...

Re: 99% code coverage (2017)

#87

Post by snidane »

I always achieve 100% relevant code coverage. I just set up a test with one input which tests my main(), which then obviously calls all the relevant parts of the code.

Or am I missing something in the definition of code coverage?

I don't see anyone using exhaustive-input tests, which is usually impossible anyway, or splitting up each branch of conditionals into separate functions so as to make them unit testable.

I only see people splitting up the code into arbitrary function blocks, having at least one unit test for such function and then declaring all lines of code of that function as test-covered.

Re: 99% code coverage (2017)

#88

Post by stcredzero »

cryptica wrote: post #9

The industry is obsessed with getting 100% unit test code coverage even though it doesn't mean anything to the project. The purpose of unit tests is to lock down the project's source code once it's essentially completed; to avoid regressions when making minor changes. If you start writing unit tests too early in the project, you're effecively locking down units of code which haven't yet proved themseves to be useful…

The purpose of unit tests is to lock down the project's source code once it's essentially completed; to avoid regressions when making minor changes.

This is only true for development environments which have particular weaknesses. If your tools for code changes and refactoring easily cover changing your unit tests while you are refactoring, then you can use unit tests for rapid development. The use of Unit Tests in present day "agile" methods actually originates in an environment where this was the norm. (Extreme Programming, as developed at the Chrysler C3 project in VisualWorks Smalltalk, using the phenomenal Refactoring Browser.)

Your statement above would be 180 degrees turned around in an environment like that.

If you start writing unit tests too early in the project, you're effecively locking down units of code which haven't yet proved themseves to be useful to your project.

Reading this just makes me sigh. Not all environments have the same cost/benefit factors in all parts of the edit/test/debug cycle. Not all environments and languages have the same costs for doing similar development operations. As a matter of fact, these are generally very, very different, even across projects using the same language. Change those cost/benefit factors, and the pronouncements you are making become more valid or less valid, depending.

Not understanding that those cost/benefit factors can change and change exactly what your best practices are -- this is one of the most pervasive big misunderstandings that fosters poor management of programming projects.

(EDIT: I think a lot of the annoyance many people have with "agile" methods, stems from practices which work well in one environment migrated to another environment where they don't work well at all!)

Re: 99% code coverage (2017)

#89

Post by rachelmcarmena »

Radle wrote: post #2

The author suggests to add mutations to the code, like replacing ' I think this might apply when you write your own algorithms and want to test them. But if you are, like most of us, working on Business Logic, then you are probably writing the wrong tests. We usually want to know whether a workflow or a customer story works as intended and as such we should write more Integration tests. The idea itself has merit thou…

Thanks for the feedback. The post only answers "Do we have a good safety net to change this legacy code?". Mutation testing is not a new concept. It's usually used in critical systems: the system is mutated and you measure the time for detecting the mutation and the time for solving the problem.
Post reply on HN