The day I started believing in unit tests • Hacker News Board

Re: The day I started believing in unit tests

#151

Post by [deleted] » Tue, Dec 19, 2023, 6:47 PM UTC

ska wrote: post #136

throwaway290 wrote: post #132

Earlier quoted context omitted.

> I don't think it is really unresolved Well, you are currently participating in a thread that discusses this very question so there's that... and such threads are regular on HN. I meant just that. People discuss it. What you want to say is that you have a strong opinion about it, that's OK and still possible with open questions

People discuss lots of things that are pretty well solved; i wouldn't equate "open question" with "lots of discussion". I guess in this context I mean that the question of static vs. dynamic in unit testing turns out to not be that hard, but the questions like "what is a unit test" and "should we unit test at all" are much muddier. Because people are confused or argumentative about the latter, they tend to pull the f…

[deleted]

Re: The day I started believing in unit tests

#152

Post by RaftPeople » Tue, Dec 19, 2023, 6:54 PM UTC

domano wrote: post #18

So at work we would run tons of tests against the real service with a real database, seeding thousands of schemas to allow for parallel testing of tests that change state. This takes 3 minutes, 1 if you use tmpfs. It only takes These actually cover most real world use cases for a query-engine we maintain. Unit tests have their place for pieces of code that run based on a well defined spec, but all in all this integra…

From research I've read, unit tests (whether automated or not) tend to catch around 30% of bugs whereas end to end testing and manual code review (believe it or not) each tend to catch around 80% of bugs.

Re: The day I started believing in unit tests

#153

Post by BurningFrog » Tue, Dec 19, 2023, 7:01 PM UTC

troupo wrote: post #143

BurningFrog wrote: post #139

Earlier quoted context omitted.

> How else are you "unit testing every single function and method"? There are many ways to write unit tests, as well as writing code that is easy to test. I don't know how my ways differs from yours, but I don't have much of the problems you mention. > Otherwise how do you know that your system still works? We do have some integration test, of course. But it's a small part of the total test suite.

> I don't know how my ways differs from yours, but I don't have much of the problems you mention. You haven't answered "How else are you "unit testing every single function and method"?" Given a medium-sized project and at least a passing and a failing test case for each function and method, you end up if not with hundreds, but with dozens of tests largely doing the same thing. > We do have some integration test, of…

> Given a medium-sized project and at least a passing and a failing test case for each function and method, you end up if not with hundreds, but with dozens of tests largely doing the same thing.

I don't understand this comment.

If I have one unit test for each function/method, that's just one test doing the same thing.

Re: The day I started believing in unit tests

#154

Post by 6DM » Tue, Dec 19, 2023, 7:03 PM UTC

MoreQARespect wrote: post #146

6DM wrote: post #106

Earlier quoted context omitted.

I have tried to evangelize unit testing at each company I've worked at and most engineers struggle with two things. The first is getting over the hurdle of trusting that a unit test is good enough, a lot of them only trust an end-to-end test which are usually very brittle. The second reason is, I think, a lot of them don't know how to systematically breakdown test into pieces to validate e.g. I'll do a test for null,…

I've evangelized against unit testing at most companies I work at, except in one specific circumstance. That circumstance is complex logic in stateless code behind a stable API where unit testing is fine. I find this usually represents between 5-30% of most code bases. The idea that unit testing should be the default go to test I find to be horrifying. I find that unit test believers struggle with the following: 1) T…

Well, I don't regard unit tests as the one true way. I don't enforce people on my team do it my way. When I get compliments on my work, I tend to elaborate and spread my approach. That's what I mean by evangelize, not necessarily advocating for a specific criteria to be met.

I find that integration tests are usually are flaky, its my personal experience. In fact, at my company, we just decided to completely turn them off because they fail for many reasons and the usual fix is to adjust the test. If you have had a lot of success with them, great. Just for the record, I am not anti-integration or end-to-end test. I think they have a place and just like unit tests shouldn't be the default, neither should they.

Here are the two most common scenarios where I find integration (usually end-to-end called integration) tests become flaky:

1) DateTime, some part of business logic relies on the current date or time and it wasn't accounted for.

2) Data changes, got deleted, it expired, etc. and the test did not first create everything it needed before running the test.

Regarding your points,

1) "realism" that is what I referred to as trusting that a unit test is good enough. If it didn't go all the way to the database and back did it test your system? In my personal work, I find that pulling the data from a database and supplying it with a mock are the same thing. So it's not only real enough for me, but better because I can simulate all kinds of scenarios that wouldn't be possible in true end-to-end tests.

2) These days the only code that's hard to test is from people that are strictly enforcing OOP. Just like any approach in programming, it will have it's pros and cons. I rarely go down that route, so testing isn't usually difficult for me.

3) It's just been my personal experience. Like I said, I'm not anti-integration tests, but I don't write very many of them.

4) I didn't refer to google, just my personal industry experience.

5) Enforcing ideal is a waste of time in programming. People only care about what they see when it ships. I just ship better quality code when I unit test my business logic. Some engineers benefit from it, some harm themselves in confusion, not much I can do about it.

Most of this is my personal experience, no knock against anyone and I don't force my ideals on anybody. I happily share what and why things work for me. I gradually introduce my own learning over time as I am asked questions and don't seek to enforce anything.

Happy coding!

Re: The day I started believing in unit tests

#155

Post by myaccountonhn » Tue, Dec 19, 2023, 7:04 PM UTC

m3kw9 wrote: post #74

How do you all feel about the need to rewrite a unit test when code gets refactored or business logic changes, isn’t that like a huge pita?

My experience is that the tests should either test functions that are small and do one thing (I.E. sorts, maps with some logic, basically where you want to test edge cases and sanity check). In those cases there is very little reason to change that code. If you are testing something larger, the test should be an integration test, where you test the full business logic flow. That makes the code less PITA to change while still giving you confidence.

If the business logic actually changes, the tests should break IMO because they are there to ensure that the business logic remains consistent. When you test the business logic (without testing the implementation) the code becomes much safer to modify and refactor.

Re: The day I started believing in unit tests

#156

Post by 6DM » Tue, Dec 19, 2023, 7:05 PM UTC

feoren wrote: post #135

6DM wrote: post #106

Earlier quoted context omitted.

I have tried to evangelize unit testing at each company I've worked at and most engineers struggle with two things. The first is getting over the hurdle of trusting that a unit test is good enough, a lot of them only trust an end-to-end test which are usually very brittle. The second reason is, I think, a lot of them don't know how to systematically breakdown test into pieces to validate e.g. I'll do a test for null,…

> I'll do a test for null, then a separate test for something else _assuming_ not null because I've already written a test for that. Honestly, this pedantry around "unit tests must only test one thing" is counter-productive. Just test as many things as you can at once; it's fine. Most tests should not be failing. Yes, it's slightly less annoying to get 2 failed tests instead of 1 fail that you fix and then another fa…

Just to clarify, I am not advocating for tests to only test one thing, rather that after you have tested for one scenario you don't need to rehash it again in another test.

Breaking a test down helps to clarify what you're testing and helps to prevent 80 loc unit tests. When I test for multiple things, I look for the equivalent of nunit's assert.multiple in the language that I'm in.

The approach I advocate for typically simplifies testing multiple scenarios with clear objectives and tends to make it easier when it comes time to refactor/fix/or just delete a no longer needed unit test. The difference I find, is that now you know why, vs having to figure out why.

Re: The day I started believing in unit tests

#157

Post by ebiester » Tue, Dec 19, 2023, 7:21 PM UTC

atticora wrote: post #58

fesc wrote: post #49

Earlier quoted context omitted.

What would "integration tests" (that you don't write) look then in your opinion? I ask because in my team we also a long time made the destinction between unit/integration based on a stupid technicality in the framework we are using. We stopped doing that and now we mostly write integration tests (which in reality we did for a long time). Of course this all arguing over definitions and kind of stupid but I do agree w…

> What would "integration tests" (that you don't write) look then in your opinion? In our local lingo, an integration test is one that also exercises the front-end, while hitting a fully functional back-end. So you could think of our "unit tests" as small back-end integration tests. If you think that way, we don't write very many pure unit tests, mostly just two flavors of integration tests. That works well for our s…

The "impurity" isn't the problem. The problem is that such integration tests take a longer time to run and in aggregate, it takes minutes to run your test suite. This changes how often you run your tests and slows down your feedback loop.

That's why you separate them: not because the integration test isn't valuable, but because it takes longer.

Re: The day I started believing in unit tests

#158

Post by Izkata » Tue, Dec 19, 2023, 7:24 PM UTC

mrweasel wrote: post #84

rileymat2 wrote: post #54

Earlier quoted context omitted.

A sort of non-judgmental question, in your mind are you writing them to cover lines or exercise required behavior with an intent of proving the module is broken? I ask because it seems like requiring line coverage as a metric would have the effect you are describing.

I've seen the same thing with comments. My boss required us to add comments to our code, to make it easier to read. That was all he asked, please add comments. My co-worker added comments like "Increase variable i by 1", while completely ignoring the 8 lines of spaghetti business logic above. Similarly I've seen people add tests that will ensure that code coverage doesn't go down, but it doesn't actually do anything…

I've thought for a long time we present coverage backwards. We shouldn't be highlighting what is covered and getting that metric up, we should highlight what isn't covered and focus on getting that metric down (like how linting is done). Present it like "Hey, here's something that no one has looked at in-depth! It's a great place for bugs to be hiding!"

Re: The day I started believing in unit tests

#159

Post by jkubicek » Tue, Dec 19, 2023, 7:49 PM UTC

HideousKojima wrote: post #129

jkubicek wrote: post #61

Earlier quoted context omitted.

> In actual practice, about 99% of unit tests I see amount to "verifying that our code does what our code does" and are a useless waste of time and effort. If you rephrase this as, "verifying that our code does what it did yesterday" these types of tests are useful. When I'm trying to add tests to previously untested code, this is usually how I start. 1. Method outputs a big blob of JSON 2. Write test to ensure that…

I had to migrate some ancient VB.NET code to .NET 6+ and C#. The code outputs a text file, and I needed to nake sure the new output matched the old output. I could have written some sort of test program that would have been roughly equal in length to what I was rewriting to verify that any change I made didn't affect the output, and to verify that the internal data was the same at each stage. Or... I could just outpu…

I see two advantages in creating tests to check output

    1. You did the work to generate consistent output from the code as a whole, plus output intermediate steps. Writing those into a test lets future folks make use of the same tests.
    2. Having the tests in place prevents people from making changes that accidentally change the output

Don't get me wrong, tests that just compare two large blobs of output aren't fun to work with, but they _can_ be useful, and are an OK intermediate stage while you get proper unit tests written.

Re: The day I started believing in unit tests

#160

Post by jkubicek » Tue, Dec 19, 2023, 7:51 PM UTC

DontchaKnowit wrote: post #102

jkubicek wrote: post #61

Earlier quoted context omitted.

> In actual practice, about 99% of unit tests I see amount to "verifying that our code does what our code does" and are a useless waste of time and effort. If you rephrase this as, "verifying that our code does what it did yesterday" these types of tests are useful. When I'm trying to add tests to previously untested code, this is usually how I start. 1. Method outputs a big blob of JSON 2. Write test to ensure that…

The problem with this for me is that most of the time "verifying that our ccode does what it did yesterday" is not a useful condition : if you make no change to code, its going to do what it did yesterday. If you do make a change to the code, then you are probably intending for it to do something different, so now you have to change the test accordingly. It usually just means you have to make the same change in 2 dif…

> If you do make a change to the code, then you are probably intending for it to do something different, so now you have to change the test accordingly. It usually just means you have to make the same change in 2 different spots for every piece of unit-tested code you want to change.

Sure, but that's how unit-tested code works in general.