Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Go Testing with Contract and Scenario Mocks

funnelstory.ai

Post reply on HN
Print view View original

21–29 of 29 posts

Re: Scaling Go Testing with Contract and Scenario Mocks

#21

Post by bikelang »

kardianos wrote: post #19

My DB heavy app, when I run a Go unit test, it spins up a DB instance, populates it, runs the tests, and drops the DB. Never ever any mocks. The best part about unit tests isn't testing the Go code. It is testing the SQL. Yes, it takes longer to run your tests. So be it.

We do the same thing. The core of our service is ingesting data and applying transformations to it. There’s so many permutations and complex interactions in here that the only way to ensure a refactor hasn’t broken one of these interactions is to document those edge cases by piping data through the system and reading it back out. We have thousands of these tests and it’s all tidy controlled via docker compose. It takes about 15 minutes to run the test suite. Sure I wish it was faster - but the real unlock is that we can make big sweeping refactors without breaking behaviors of the system. The organizational speed unlock this kind of safety net is well worth a bit of slowness in CI.

We also have mocks/stubs/spies in our unit tests. Those are great for producing hard-to-trigger edge cases. But contract testing? The contract is the data flow. In the end it’s all about using the right tool for the right test. There is no one-size-fits-all.

Re: Scaling Go Testing with Contract and Scenario Mocks

#22

Post by [deleted] »

aranw wrote: post #12

The problem with mocks is that they test your assumptions, not reality... When you mock a CRM client to return one account, you're assuming it always returns one account, that IDs have a particular format, that there's no pagination, that all fields are populated. Each assumption is a place where production could behave differently whilst your tests stay green Your contract tests use cached JSON fixtures. Salesforce…

[deleted]

Re: Scaling Go Testing with Contract and Scenario Mocks

#23

Post by 9rx »

kardianos wrote: post #19

My DB heavy app, when I run a Go unit test, it spins up a DB instance, populates it, runs the tests, and drops the DB. Never ever any mocks. The best part about unit tests isn't testing the Go code. It is testing the SQL. Yes, it takes longer to run your tests. So be it.

If your SQL is isolated to one place then you only need to test that single package with a real DB and can use mocks everywhere else. Your mocks can be tested with the exact same test suite as the SQL package, so you know it conforms to the same contract.

If you have SQL scattered all over the place... Leave the spaghetti for dinner.

Re: Scaling Go Testing with Contract and Scenario Mocks

#24

Post by Phlebsy »

SPascareli13 wrote: post #3

I really dislike this idea of testing in go: only ever use an interface, never the real implementation + mockgen the mocks based on this interface + use the mocks to assert that a function is called, with exactly this parameters and in this exact order. I find this types of tests incredibly coupled with the implementation, since any chance require you to chance your interfaces + mocks + tests, also very brittle and m…

I'm a fan of writing tests that can be either. Write your tests first such that the real dependencies can be run against. Snapshot the results to feed into integration test mocks for those dependencies so that you can maintain the speed benefit of limited test scope. Re-run against the real dependencies at intervals you feel is right to ensure that your contracts remain satisfied, or just dedicate a test per external endpoint on top of this to validate the response shape hasn't changed.

The fundamental point of tests should be to check that your assumptions about a system's behavior hold true over time. If your tests break that is a good thing. Your tests breaking should mean that your users will have a degraded experience at best if you try to deploy your changes. If your tests break for any other reason then what the hell are they even doing?

Re: Scaling Go Testing with Contract and Scenario Mocks

#25

Post by rhines »

aranw wrote: post #11
SPascareli13 wrote: post #3

I really dislike this idea of testing in go: only ever use an interface, never the real implementation + mockgen the mocks based on this interface + use the mocks to assert that a function is called, with exactly this parameters and in this exact order. I find this types of tests incredibly coupled with the implementation, since any chance require you to chance your interfaces + mocks + tests, also very brittle and m…

> I really dislike this idea of testing in go: only ever use an interface, never the real implementation + mockgen the mocks based on this interface + use the mocks to assert that a function is called, with exactly this parameters and in this exact order. Same I have zero confidence in these tests and the article even states that the tests will fail if a contract for a external service/system changes

I see this kind of testing as more for regression prevention than anything. The tests pass if the code handles all possible return values of the dependencies correctly, so if someone goes and changes your code such that the tests fail they have to either fix the errors they've introduced or go change the tests if the desired code functionality has really changed.

These tests won't detect if a dependency has changed, but that's not what they're meant for. You want infrastructure to monitor that as well.

Re: Scaling Go Testing with Contract and Scenario Mocks

#26

Post by fireflash38 »

I'm of the opinion that mocks should be provided by the thing that you're mocking. That is, if you are wanting to mock out a service, the mock should be owned by the service that is being mocked.

And then it should be part of that service's test suite, to verify it's own mock.

You update your service? Then you must update the mock.

I guess that's more of a fake, but the naming doesn't matter as much as the behavior.

Re: Scaling Go Testing with Contract and Scenario Mocks

#27

Post by 9rx »

et1337 wrote: post #10
esafak wrote: post #4

Earlier quoted context omitted.

Don't you mean testing the interface of the implementation? I see nothing wrong with that, if so.

They mean the dependencies. If you’re testing system A whose sole purpose is to call functions in systems B and C, one approach is to replace B and C with mocks. The test simply checks that A calls the right functions. The pain comes when system B changes. Oftentimes you can’t even make a benign change (like renaming a function) without updating a million tests.

Tests are only concerned with the user interface, not the implementation. If System B changes, that means that you only have to change your implementation around using System B to reflect it. The user interface remains the same, and thus the tests can remain the same, and therefore so can the mocks.

Re: Scaling Go Testing with Contract and Scenario Mocks

#28

Post by et1337 »

9rx wrote: post #27
et1337 wrote: post #10

Earlier quoted context omitted.

They mean the dependencies. If you’re testing system A whose sole purpose is to call functions in systems B and C, one approach is to replace B and C with mocks. The test simply checks that A calls the right functions. The pain comes when system B changes. Oftentimes you can’t even make a benign change (like renaming a function) without updating a million tests.

Tests are only concerned with the user interface, not the implementation. If System B changes, that means that you only have to change your implementation around using System B to reflect it. The user interface remains the same, and thus the tests can remain the same, and therefore so can the mocks.

I think we’re in agreement. Mocks are usually all about reaching inside the implementation and checking things. I prefer highly accurate “fakes” - for example running queries against a real ephemeral Postgres instance in a Docker container instead of mocking out every SQL query and checking that query.Execute was called with the correct arguments.

Re: Scaling Go Testing with Contract and Scenario Mocks

#29

Post by 9rx »

et1337 wrote: post #28
9rx wrote: post #27

Earlier quoted context omitted.

Tests are only concerned with the user interface, not the implementation. If System B changes, that means that you only have to change your implementation around using System B to reflect it. The user interface remains the same, and thus the tests can remain the same, and therefore so can the mocks.

I think we’re in agreement. Mocks are usually all about reaching inside the implementation and checking things. I prefer highly accurate “fakes” - for example running queries against a real ephemeral Postgres instance in a Docker container instead of mocking out every SQL query and checking that query.Execute was called with the correct arguments.

> Mocks are usually all about reaching inside the implementation and checking things.

Unfortunately there is no consistency in the nomenclature used around testing. Testing is, after all, the least understood aspect of computer science. However, the dictionary suggests that a "mock" is something that is not authentic, but does not deceive (i.e. not the real thing, but behaves like the real thing). That is what I consider a "mock", but I'm gathering that is what you call a "fake".

Sticking with your example, a mock data provider to me is something that, for example, uses in-memory data structures instead of SQL. Tested with the same test suite as the SQL implementation. It is not the datastore intended to be used, but behaves the same way (as proven by the shared tests).

> checking that query.Execute was called with the correct arguments.

That sounds ridiculous and I am not sure why anyone would ever do such a thing. I'm not sure that even needs a name.

Post reply on HN