Stop mocking your system • Hacker News Board

Re: Stop mocking your system

#141

Post by thyrsus » Mon, Jul 05, 2021, 1:05 AM UTC

My experience with mocking was sad.

I had unit tests; they all passed.

I had end-to-end tests; they all passed.

In production, I had about 1 in 1,000,000 failures. The most likely culprit was broken TCP connections, in turn most likely due to the enormous network load I was generating, which was only tolerable during rare windows. I tried mocking Linux TCP socket behavior, but failed to find a formal descriptions of the possible failure behaviors, and had to guess at a lot of things; I ultimately didn't finish before the project was abandoned.

Re: Stop mocking your system

#142

Post by [deleted] » Mon, Jul 05, 2021, 1:09 AM UTC

astuyvenberg wrote: post #52

cortesoft wrote: post #33

Earlier quoted context omitted.

That means that every person who runs the tests needs credentials for that AWS account. That obviously won’t work for an open source project. Even for a company project, how do you distribute those secrets? It adds friction for developers getting their local dev environment setup. Not only that, but you now need network access to run tests. A network blip or a third party service outage now makes your tests fail. The…

> That obviously won’t work for an open source project On the contrary - I was an employee at Serverless Inc, working on the Serverless Framework for the last two years, we used this pattern extensively (and very successfully) in our open source repos. You can even find an example here which provisions real live AWS infrastructure: https://github.com/serverless/dashboard-plugin/tree/master/i... We used part of our en…

[deleted]

Re: Stop mocking your system

#143

Post by koyote » Mon, Jul 05, 2021, 1:10 AM UTC

lilyball wrote: post #77

Why stop there? Why not delete your test methods too and just test in production? Mocks are just test code, same as your test functions. And they’re necessary fur unit tests. If the thing you’re testing talks to another component without a mock, it’s now an integration test instead of a unit test. Unit tests test the API surface of a component. They’re useful for ensuring a component adhere to its documented API cont…

> Really the lesson that should be taught here instead of “don’t use mocks” is “unit tests aren’t sufficient, write integration tests too”.

I completely agree with your comment.

I also think that people underestimate the kinds of scenarios unit tests can test; scenarios that are not possible or not practical to test by other means:

Unit tests are excellent for testing edge cases and scenarios that do not generally happen in normal production. It makes it easy to test things like 'how does this function behave if X returns some corrupted data?'. I feel like a lot of products don't test the failure cases at all, thinking 'that will never happen'. Mocking external services is a very straightforward way to test these kind of scenarios.

Just last week I was looking at a typical heisenbug that did not happen in 99%+ of production. The bug was masked by a combination of caching and the program flow. I was able to write a very short and specific unit test that was able to deterministically mimic the (bad) state the program was in and reproduce said bug.

There was however no way to properly test this using integration testing because it was relying on specific program state to manifest itself. Even if we had an integration test with the exact same data that caused the bug to appear, it would not be a deterministic test and future code changes might make it disappear even though the underlying bug is still there (or re-appeared).

Re: Stop mocking your system

#144

Post by nmfisher » Mon, Jul 05, 2021, 1:24 AM UTC

fragbait65 wrote: post #136

lugged wrote: post #123

Earlier quoted context omitted.

> The only thing you achieve with unit tests is to prove that you are still bug for bug compatible when changing your code. People love to ignore how fucking awful code is when you aren't forced to write unit tests. Unit tests before you write code help you code, they provide a tight feedback loop, they force or at least encourage modularity to be testable. Adding tests to my code after writing it forces me to challe…

I used to do TDD, I found that it didn't work that well for me. I still think that the code base you work on dictates how you should be testing. It also dictates which testing strategy I use. I test for different things when writing C than I do when I write Python for example. My testing strategy is different if I write a networked C application compared to if I write a Ruby on Rails app. Also the tools I have availa…

I think the point was more that many people can’t write unit tests because their code is so shoddy that it’s basically impossible to do so.

It sounds like you and I share a similar methodology. I don’t write unit tests for all my code. In fact if I had to guess, I’d say I generally test less than 25%.

But I do write all my code such that it’s capable of having unit tests written for it.

In many codebases that’s just not possible. The developers have spent all their time writing and none of their time thinking. It’s one big yarn ball that’s impossible to write unit tests for. For these people, forcing them to write tests up front would have resulted in a much cleaner code base.

Re: Stop mocking your system

#145

Post by aranchelk » Mon, Jul 05, 2021, 1:34 AM UTC

midrus wrote: post #71

aranchelk wrote: post #14

Ghost of Christmas Future: I did some contracting work for a company with a 1.5 hour test suite that ran on every deploy, the section with mocked tests only took a couple of minutes — the rest were end-to-end (no mocks). The worst parts of those non-mocked tests: * They would interfere with each other, and could not be run in parallel. * They were subject to real-world variability and were not entirely deterministic.…

Static typing has nothing to do with testing, or bugs. Static typing is about tooling, autocompletion and being able to follow the code. Using static typing as a way to reduce the number of tests is just terrible advice in my opinion. Also, the fact that tests interfere with each other, etc makes me think of just bad tests, not with a problem with the concept of e2e tests itself.

In effect a sufficiently strong static type system greatly reduces the kinds of (mostly undesirable) programs you can compile and in doing so eliminates entire classes of bugs. If you can’t produce those bugs, then you don’t have to test for them.

In the dynamic languages I’ve used (even the ones that don’t allow coercion) you can’t ensure the arguments passed to a function at runtime will be the correct type, so you’re usually stuck doing introspection or a try catch/except block to stop an undesirable argument type from messing up your application, and you should probably have tests to verify that.

Working in languages with a type system the disallows nulls (nils, nones, etc.) you never ever have to test for null pointer exceptions, that should account for at least a few tests.

Moving to dependent types, you can even start to verify the domain logic of your application.

I think your argument may hold true for languages like Java, but it’s not applicable for the strong static functional stuff: Haskell, PureScript, Idris, etc.

Re: Stop mocking your system

#146

Post by crazygringo » Mon, Jul 05, 2021, 1:35 AM UTC

fragbait65 wrote: post #117

alecbz wrote: post #51

True "unit" tests are: * faster to run * give you less confidence in the correctness of the system (per time spent writing them) * when they fail, give you more information about where the failure is The more integration-y/e2e-y a test is, the more it strays from this: slower to run, more confidence that the system is correct, less info about where the failures are. I think people have learned to undervalue the prope…

I think it's a failure to ever believe that unit tests provide any correctness proof. The only thing you achieve with unit tests is to prove that you are still bug for bug compatible when changing your code. I think experience is needed to know when to unit test. Some code might not need any tests at all while other code might need quite a lot of tests. My personal experience, for the code bases I work on, shows that…

> The only thing you achieve with unit tests is to prove that you are still bug for bug compatible when changing your code.

If your tests are ensuring buggy behavior, then... you're writing your tests wrong?

What are you even trying to say with this? I've never once seen a unit test intended to enforce a bug...?

Re: Stop mocking your system

#147

Post by redconfetti » Mon, Jul 05, 2021, 1:35 AM UTC

lilyball wrote: post #77

Why stop there? Why not delete your test methods too and just test in production? Mocks are just test code, same as your test functions. And they’re necessary fur unit tests. If the thing you’re testing talks to another component without a mock, it’s now an integration test instead of a unit test. Unit tests test the API surface of a component. They’re useful for ensuring a component adhere to its documented API cont…

Honestly I'd rather have all the units covered in isolation, and leave it up to my manual testing to catch any unexpected errors or bugs, then let our QA team do the same with their own Selenium based black-box testing.

If the contract between two units is flawed, you only have to worry about updating the behavior of one or both units. The unit tests make sure they work as designed.

It seems to abstract the problem for me to the level of only detecting missed integration issues. I don't need to worry about how those units do what they do, just that they do what they're designed, because the unit tests say so.

There's nothing I hate more than making a small change, and having to spend hours tracking down why 20+ tests that are written as integration tests rather than unit tests (with mocking).

I'd rather maintain the mocks when making changes, than have to spend all that time trying to figure out what complicated and huge, far reaching, thing is broken.

Re: Stop mocking your system

#148

Post by bhawks » Mon, Jul 05, 2021, 1:52 AM UTC

Aeolun wrote: post #135

bhawks wrote: post #61

Mocks make me sad. They are so often misused by people with the best of intentions. I have seen so many tests that literally only assert the mock expectations and say nothing about the code under test. I have watched upgrades and refactors take 5x longer because someone steps on a landmine of overmocking. It is so common to see people mocking dumb data objects, containers, and pure functions - creating Frankenstein w…

If you correctly use interfaces for your contracts, how can your mocks ever be out of date? Your compilation step would fail.

Contracts cover more than what type systems on most common languages can express.

For instance popping a stack with no elements throws an exception. Or similarly constructing an object with certain propertied set cause methods on the object to behave differently.

Mocked objects at best encode what the behavior of a certain subset of interactions should be - at a certain point in time.

Re: Stop mocking your system

#149

Post by zlynx » Mon, Jul 05, 2021, 1:57 AM UTC

crazygringo wrote: post #146

fragbait65 wrote: post #117

Earlier quoted context omitted.

I think it's a failure to ever believe that unit tests provide any correctness proof. The only thing you achieve with unit tests is to prove that you are still bug for bug compatible when changing your code. I think experience is needed to know when to unit test. Some code might not need any tests at all while other code might need quite a lot of tests. My personal experience, for the code bases I work on, shows that…

> The only thing you achieve with unit tests is to prove that you are still bug for bug compatible when changing your code. If your tests are ensuring buggy behavior, then... you're writing your tests wrong? What are you even trying to say with this? I've never once seen a unit test intended to enforce a bug...?

If you have a web API that returns a 200 OK with a JSON object with a missing entry when an item is not found, instead of a 404 error, then you have a bug in your API.

But you cannot allow it to change because your users are relying on the existing behavior.

Re: Stop mocking your system

#150

Post by ilovwindows » Mon, Jul 05, 2021, 1:58 AM UTC

I agree with the author. Some devs take unit testing too literally. I have come across tests which mock every units that a component being test depends on. It's a terrible practice. First mocks are hard to maintain and second over time this friction generate disparity between real code and mocks. I have seen tests that pass even when the actual implementations have diverged.

The rule of thumb is, if you can test without mocks don't mock.

Most of the tests we write are integration tests not unit tests. Even at function level, a function being tested calls multiple other functions to get the things done.