Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

99% code coverage (2017)

rachelcarmena.github.io

Post reply on HN
Print view View original

71–80 of 99 posts

Re: 99% code coverage (2017)

#71

Post by monksy »

rkangel wrote: post #68
monksy wrote: post #65

Earlier quoted context omitted.

A few things: First I write unit tests, I tend to write a lot. They get added to as time goes on, bugs are found etc. (The case where he was suggesting feature only) When you have a lot of code coverage, and you find a bug (logic errors) your tests should either change due to change in expectations or they should be added to. On Mocks Mocks are really terrible for unit tests. Those tests then become integration tests…

> I don't have the time to teach someone who If that's the case then on this forum please ignore them rather than attacking them. We were all inexperienced once.

They're never going to learn if no one ever says anything about a bad practice.

Re: 99% code coverage (2017)

#72

Post by rkangel »

monksy wrote: post #71
rkangel wrote: post #68

Earlier quoted context omitted.

> I don't have the time to teach someone who If that's the case then on this forum please ignore them rather than attacking them. We were all inexperienced once.

They're never going to learn if no one ever says anything about a bad practice.

That's true, but people also don't learn from (and none of us enjoy) being attacked for what they've said. Constructive, positive explanations are of more value ("x has the following benefits because..." rather than "y is bad. full stop").

Re: 99% code coverage (2017)

#73

Post by nordsieck »

austincheney wrote: post #62
jeffasinger wrote: post #52

Earlier quoted context omitted.

I think your argument is missing the "cost" of feature tests, which is that they necessarily must run much slower than tests that are only testing a small unit of code. For example, I worked on payroll software at some point. After finding a bug, we'd want to ensure that could never happen again, and would want to add a test for it going forward. So for example, I may have needed to write a test around someone who wo…

> but it's likely the test itself will take a significant amount of wall time to execute. Why? What about your application changed so that it is slower when testing compared to real world use? If anything it should be dramatically faster because people don't provide microsecond accurate automated responses. If the application naturally executes very quickly I would imagine it would take far long to set up the test sc…

> In my own applications if they take more than two seconds to deliver a response (even for 5mb input) then at the very least I have a critical performance defect.

An entire unit test suite should complete in well under 2 seconds.

Re: 99% code coverage (2017)

#74

Post by daveFNbuck »

austincheney wrote: post #62
jeffasinger wrote: post #52

Earlier quoted context omitted.

I think your argument is missing the "cost" of feature tests, which is that they necessarily must run much slower than tests that are only testing a small unit of code. For example, I worked on payroll software at some point. After finding a bug, we'd want to ensure that could never happen again, and would want to add a test for it going forward. So for example, I may have needed to write a test around someone who wo…

> but it's likely the test itself will take a significant amount of wall time to execute. Why? What about your application changed so that it is slower when testing compared to real world use? If anything it should be dramatically faster because people don't provide microsecond accurate automated responses. If the application naturally executes very quickly I would imagine it would take far long to set up the test sc…

> What about your application changed so that it is slower when testing compared to real world use?

Real world use hits an expensive optimized production cluster. Tests run on my laptop or a cheap testing machine.

Re: 99% code coverage (2017)

#75

Post by GordonS »

t0mbstone wrote: post #61
jeffasinger wrote: post #52

Earlier quoted context omitted.

I think your argument is missing the "cost" of feature tests, which is that they necessarily must run much slower than tests that are only testing a small unit of code. For example, I worked on payroll software at some point. After finding a bug, we'd want to ensure that could never happen again, and would want to add a test for it going forward. So for example, I may have needed to write a test around someone who wo…

This is why you actually need a mix of feature tests and unit tests. Have automated feature tests that test most of the "happy path" run-throughs of features that your users do on the front end, and then unit tests that test the minutiae. My favorite project that I ever worked on was one that was set up this way. We had literally hundreds of feature tests, and thousands of unit tests. When running in a single thread…

This is my preference too - mainly feature tests, and unit and integration tests where they provide enough value to be worthwhile. I generally end up with 70-75% test coverage, but I don't pay too much attention to the actual number.

We just released a fairly large system that took this testing approach - despite a large number of tests hitting a database, file system and a blob storage emulator, they still completed in only 5 minutes or so on our CI machine (or 1 minute locally on very beefy laptops).

Re: 99% code coverage (2017)

#76

Post by austincheney »

Macha wrote: post #69
austincheney wrote: post #62

Earlier quoted context omitted.

> but it's likely the test itself will take a significant amount of wall time to execute. Why? What about your application changed so that it is slower when testing compared to real world use? If anything it should be dramatically faster because people don't provide microsecond accurate automated responses. If the application naturally executes very quickly I would imagine it would take far long to set up the test sc…

We have 1000s of features. Real users don't use every feature every time they perform an action. A test case at the feature level will likely require several requests if its starting from clean state.

> A test case at the feature level will likely require several requests if its starting from clean state.

You say request so I guess you are talking about a network service request. When I first got into A/B testing many years ago we found we could use it to simulate traffic through the entirety of the Travelocity web site from homepage to checkout and back. They never used the A/B test system for test automation, but they could have. Back then the checkout pages were rarely tested with valid feature tests because they were at the end of the path. There were plenty of unit tests isolated to small bits of functionality at those pages. We discovered, thanks to session recording with things like Tea Leaf and Clicktale, just a bit later that those unit tests were worthless. The checkout pages failed all the time and cost the company tremendously in lost revenue. Some of those failures would show up in our A/B test experiments though.

My lesson learned from this is that testing things in isolation isn't reliable and provides false confidence. You tend to get back the answer you wanted such that is amounts to a self-fulling prophecy opposed to a criticism.

Re: 99% code coverage (2017)

#77

Post by austincheney »

nordsieck wrote: post #73
austincheney wrote: post #62

Earlier quoted context omitted.

> but it's likely the test itself will take a significant amount of wall time to execute. Why? What about your application changed so that it is slower when testing compared to real world use? If anything it should be dramatically faster because people don't provide microsecond accurate automated responses. If the application naturally executes very quickly I would imagine it would take far long to set up the test sc…

> In my own applications if they take more than two seconds to deliver a response (even for 5mb input) then at the very least I have a critical performance defect. An entire unit test suite should complete in well under 2 seconds.

That depends on the application, what you are testing, and how many test units there are. As absurd as it sounds I have seen people force test units on logic that does nothing more than validate the presence of testing, which is amazingly fast.

Re: 99% code coverage (2017)

#78

Post by austincheney »

rkangel wrote: post #67
austincheney wrote: post #62

Earlier quoted context omitted.

> but it's likely the test itself will take a significant amount of wall time to execute. Why? What about your application changed so that it is slower when testing compared to real world use? If anything it should be dramatically faster because people don't provide microsecond accurate automated responses. If the application naturally executes very quickly I would imagine it would take far long to set up the test sc…

I'll give you one real example: because my application is some embedded software that has to be built, downloaded onto the target and then run. That is the only environment the real application runs in. Some tests should be run at this level, but the cost to automate and the benefit means that you usually don't try and get full coverage. You can build a desktop version of the software with hardware mocked out, which…

Unless the software is firmware or an OS it rarely directly touches the hardware. It only knows of the hardware available because the firmware or OS provide that information. I have seen a lot of organizations fake this with an array of VMs or when that is not enough they have a bunch of spare hardware to test against, like a whole bunch of different models of Android phones.

Re: 99% code coverage (2017)

#79

Post by jeffasinger »

austincheney wrote: post #62
jeffasinger wrote: post #52

Earlier quoted context omitted.

I think your argument is missing the "cost" of feature tests, which is that they necessarily must run much slower than tests that are only testing a small unit of code. For example, I worked on payroll software at some point. After finding a bug, we'd want to ensure that could never happen again, and would want to add a test for it going forward. So for example, I may have needed to write a test around someone who wo…

> but it's likely the test itself will take a significant amount of wall time to execute. Why? What about your application changed so that it is slower when testing compared to real world use? If anything it should be dramatically faster because people don't provide microsecond accurate automated responses. If the application naturally executes very quickly I would imagine it would take far long to set up the test sc…

Sure, let's go back to my payroll example.

I need to make 8 HTTP requests to properly setup my test from a clean slate, including setting up the deductions, previous earnings, company location and employee home address. If each of those requests takes 50ms, and then I need to make a request to actually execute my test, and verify everything went well, that could easily be 500ms.

And that's just what needs to happen to run one end to end test of the payroll calculation feature. It's absolutely valuable to have a few such tests, but I'm not going to run tests for all the weird one off scenarios (like what happens if someone lives in NJ, and works in Yonkers, and the company has a location in NJ as well?) because they'd take minutes to run. I could run that entire test as a unit test in 2ms for the payroll calculation. That lets me run all of the tests I need very quickly.

Re: 99% code coverage (2017)

#80

Post by johanlejdung »

austincheney wrote: post #37
cryptica wrote: post #9

The industry is obsessed with getting 100% unit test code coverage even though it doesn't mean anything to the project. The purpose of unit tests is to lock down the project's source code once it's essentially completed; to avoid regressions when making minor changes. If you start writing unit tests too early in the project, you're effecively locking down units of code which haven't yet proved themseves to be useful…

It drives me nuts during interviews talking about test automation because people are very particular about the type of testing whether its unit testing, integration testing, acceptance testing, or whatever. In my mind you only need 1 kind of testing: feature tests. Does the application provide the expected output for a given input and/or configuration. The application does all that it claims to do in a very precise w…

Well, while I can somewhat agree with your point; it sounds like you are looking at it from a business standpoint. Someone who will be working with the details of the code is going to appreciate unit tests, not only because it verifies all the tiny pieces of code the person wrote. But also because it makes it very easy to track down where issues might be if a feature is broken.

I dont agree with the "100% test coverage" mentality, but I think there is value with unit and integration tests.

Post reply on HN