Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

99% code coverage (2017)

rachelcarmena.github.io

Post reply on HN
Print view View original

61–70 of 99 posts

Re: 99% code coverage (2017)

#61

Post by t0mbstone »

jeffasinger wrote: post #52
austincheney wrote: post #37

Earlier quoted context omitted.

It drives me nuts during interviews talking about test automation because people are very particular about the type of testing whether its unit testing, integration testing, acceptance testing, or whatever. In my mind you only need 1 kind of testing: feature tests. Does the application provide the expected output for a given input and/or configuration. The application does all that it claims to do in a very precise w…

I think your argument is missing the "cost" of feature tests, which is that they necessarily must run much slower than tests that are only testing a small unit of code. For example, I worked on payroll software at some point. After finding a bug, we'd want to ensure that could never happen again, and would want to add a test for it going forward. So for example, I may have needed to write a test around someone who wo…

This is why you actually need a mix of feature tests and unit tests.

Have automated feature tests that test most of the "happy path" run-throughs of features that your users do on the front end, and then unit tests that test the minutiae.

My favorite project that I ever worked on was one that was set up this way. We had literally hundreds of feature tests, and thousands of unit tests. When running in a single thread on a local machine, it would take two hours, but when running on Circle CI with parallelization, it would run the entire suite in 6 minutes.

This enabled us to release features to production with a high amount of confidence, any time we wanted.

It was not uncommon for us to release bug fixes to production, while still on the phone with the complaining customer. We earned a lot of customer loyalty points any time we pulled that off. And the best part was that because we had the massive feature test suite running as part of of CI/CD, we were able to do that with the same amount of confidence that we would have compared to having a massive QA team with 50 people testing our complete app before every deployment. It was awesome.

Re: 99% code coverage (2017)

#62

Post by austincheney »

jeffasinger wrote: post #52
austincheney wrote: post #37

Earlier quoted context omitted.

It drives me nuts during interviews talking about test automation because people are very particular about the type of testing whether its unit testing, integration testing, acceptance testing, or whatever. In my mind you only need 1 kind of testing: feature tests. Does the application provide the expected output for a given input and/or configuration. The application does all that it claims to do in a very precise w…

I think your argument is missing the "cost" of feature tests, which is that they necessarily must run much slower than tests that are only testing a small unit of code. For example, I worked on payroll software at some point. After finding a bug, we'd want to ensure that could never happen again, and would want to add a test for it going forward. So for example, I may have needed to write a test around someone who wo…

> but it's likely the test itself will take a significant amount of wall time to execute.

Why? What about your application changed so that it is slower when testing compared to real world use? If anything it should be dramatically faster because people don't provide microsecond accurate automated responses. If the application naturally executes very quickly I would imagine it would take far long to set up the test scenario than to execute against it.

In my own applications if they take more than two seconds to deliver a response (even for 5mb input) then at the very least I have a critical performance defect. There are not many administrative tasks I can complete from start to finish in that frame of time.

Re: 99% code coverage (2017)

#63

Post by mdpopescu »

jesusmg wrote: post #4
peterwaller wrote: post #3

Mutation testing is a neat idea I'd not heard of. Wonder how well it works in practice. Someone's implemented a package for doing it with Go which looks good: https://github.com/zimmski/go-mutesting

Me neither, I liked the concept of mutation testing. (I was using this performing changes manually, without knowing this technique has a nice name). I would appreciate if somebody points out a mutation framework /tools for .net

There's something that used to be called Pex, and now is apparently called IntelliTest [1], but I believe it's only available in the Enterprise edition of Visual Studio.

[1] https://docs.microsoft.com/en-us/visualstudio/test/intellite...

Re: 99% code coverage (2017)

#64

Post by austincheney »

monksy wrote: post #58
austincheney wrote: post #37

Earlier quoted context omitted.

It drives me nuts during interviews talking about test automation because people are very particular about the type of testing whether its unit testing, integration testing, acceptance testing, or whatever. In my mind you only need 1 kind of testing: feature tests. Does the application provide the expected output for a given input and/or configuration. The application does all that it claims to do in a very precise w…

Please tell me all of the products that your code is contained in. I don't want to die in a self-driving car because has superficial test coverage. (Well it passed it's feature tests that you wrote for it) There are very specific reasons for each level of testing and the benefits stack.

I would imagine feature tests for a self-driving car would be everything from testing functionality of the radio to having the car parallel park or back into a parking spot while towing a trailer.

Re: 99% code coverage (2017)

#65

Post by monksy »

ApolloFortyNine wrote: post #60
monksy wrote: post #58

Earlier quoted context omitted.

Please tell me all of the products that your code is contained in. I don't want to die in a self-driving car because has superficial test coverage. (Well it passed it's feature tests that you wrote for it) There are very specific reasons for each level of testing and the benefits stack.

>Well it passed it's feature tests that I wrote for it You (the programmer) write unit tests too, and even well code with well written unit tests will have bugs in it, if simply due to logic errors, or if due to mocks. >Please tell me all of the products that your code is contained in. I don't want to die in a self-driving car because has superficial test coverage. I just don't think this was the best way to go about…

A few things: First

I write unit tests, I tend to write a lot. They get added to as time goes on, bugs are found etc. (The case where he was suggesting feature only) When you have a lot of code coverage, and you find a bug (logic errors) your tests should either change due to change in expectations or they should be added to.

On Mocks

Mocks are really terrible for unit tests. Those tests then become integration tests, because you're not testing the unit of code, you're testing a system under test. The unit test should test the code contained. (Not if you can connect to a faked out driver to the DB) Use interfaces and stubs to avoid that.

The attack

That was because the justification came from a lack of discipline. I don't have the time to teach someone who doesn't want to do the work to test their code, nor do they care to understand why testing on those levels is good. I've been through these arguments with others about it. From my experience they never seem to catch on that the amount of combinations of testing is going to get tiresome very quickly, they're going to miss things in the tests, the coverage reports are more inflated than the VW scandal (yes.. instrumentation says that line was hit.. but was the value tested), and they leave around landmines for anyone else that touches that code later. If you take any stake in what Venkat says: Code quality is about how you feel about others. In their case: people who only write feature tests regularly leave their mess laying around and want other people to deal with it.

DHH is an example of this, in his talk he was upset about how much he had to verify the small details. (That's good, that he's upset, it shows that he should start looking at creating new libraries that isolate him from that. It's not a good reason to say "oh we don't need a test to verify `def add2(a:Int, b:Int)= a+b`)

Re: 99% code coverage (2017)

#66

Post by monksy »

austincheney wrote: post #64
monksy wrote: post #58

Earlier quoted context omitted.

Please tell me all of the products that your code is contained in. I don't want to die in a self-driving car because has superficial test coverage. (Well it passed it's feature tests that you wrote for it) There are very specific reasons for each level of testing and the benefits stack.

I would imagine feature tests for a self-driving car would be everything from testing functionality of the radio to having the car parallel park or back into a parking spot while towing a trailer.

Good luck on ever getting a test on the ABS breaks on that. But hey the radio, and parallel park works. The back into a parking spot is a good example of how bikesheding can happen even in testing. (You shouldn't be towing with a car, they're not a good fit for that)

Re: 99% code coverage (2017)

#67

Post by rkangel »

austincheney wrote: post #62
jeffasinger wrote: post #52

Earlier quoted context omitted.

I think your argument is missing the "cost" of feature tests, which is that they necessarily must run much slower than tests that are only testing a small unit of code. For example, I worked on payroll software at some point. After finding a bug, we'd want to ensure that could never happen again, and would want to add a test for it going forward. So for example, I may have needed to write a test around someone who wo…

> but it's likely the test itself will take a significant amount of wall time to execute. Why? What about your application changed so that it is slower when testing compared to real world use? If anything it should be dramatically faster because people don't provide microsecond accurate automated responses. If the application naturally executes very quickly I would imagine it would take far long to set up the test sc…

I'll give you one real example: because my application is some embedded software that has to be built, downloaded onto the target and then run. That is the only environment the real application runs in. Some tests should be run at this level, but the cost to automate and the benefit means that you usually don't try and get full coverage.

You can build a desktop version of the software with hardware mocked out, which is very useful, but then you're doing something much more akin to an integration test.

Re: 99% code coverage (2017)

#68

Post by rkangel »

monksy wrote: post #65
ApolloFortyNine wrote: post #60

Earlier quoted context omitted.

>Well it passed it's feature tests that I wrote for it You (the programmer) write unit tests too, and even well code with well written unit tests will have bugs in it, if simply due to logic errors, or if due to mocks. >Please tell me all of the products that your code is contained in. I don't want to die in a self-driving car because has superficial test coverage. I just don't think this was the best way to go about…

A few things: First I write unit tests, I tend to write a lot. They get added to as time goes on, bugs are found etc. (The case where he was suggesting feature only) When you have a lot of code coverage, and you find a bug (logic errors) your tests should either change due to change in expectations or they should be added to. On Mocks Mocks are really terrible for unit tests. Those tests then become integration tests…

> I don't have the time to teach someone who

If that's the case then on this forum please ignore them rather than attacking them. We were all inexperienced once.

Re: 99% code coverage (2017)

#69

Post by Macha »

austincheney wrote: post #62
jeffasinger wrote: post #52

Earlier quoted context omitted.

I think your argument is missing the "cost" of feature tests, which is that they necessarily must run much slower than tests that are only testing a small unit of code. For example, I worked on payroll software at some point. After finding a bug, we'd want to ensure that could never happen again, and would want to add a test for it going forward. So for example, I may have needed to write a test around someone who wo…

> but it's likely the test itself will take a significant amount of wall time to execute. Why? What about your application changed so that it is slower when testing compared to real world use? If anything it should be dramatically faster because people don't provide microsecond accurate automated responses. If the application naturally executes very quickly I would imagine it would take far long to set up the test sc…

We have 1000s of features.

Real users don't use every feature every time they perform an action.

A test case at the feature level will likely require several requests if its starting from clean state.

Re: 99% code coverage (2017)

#70

Post by didibus »

cryptica wrote: post #9

The industry is obsessed with getting 100% unit test code coverage even though it doesn't mean anything to the project. The purpose of unit tests is to lock down the project's source code once it's essentially completed; to avoid regressions when making minor changes. If you start writing unit tests too early in the project, you're effecively locking down units of code which haven't yet proved themseves to be useful…

Very well said, I'm in your camp as well.

The major issue is no one ever learns about or talks about the "Unit" in unit tests. But a Unit is supposed to be a component, albeit a very small one, in OOP, that normally translates to a Class.

Your unit is a given Class, assuming its API doesn't change, the unit tests should continue working. If you havn't settled on an API for that unit yet, the tests are going to be coupled to a quick moving target, and that, in my opinion, is a waste of time.

That said, people don't really design their Classes as unit in a bigger set, with self-contained API and SOLID principles. That's the first mistake. And that then leads to tests which assert not a Unit, but some random piece of logic within a Unit, or against things that are not Units of their own, but already tightly coupled and leaky containers. Which ends up both with bad designed software that is inflexible and quick to rot, as well as poor tests which prevent you from making any meaningful design improvements to it, without all the tests breaking.

Post reply on HN