Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Go Testing with Contract and Scenario Mocks

funnelstory.ai

Post reply on HN
Print view View original

11–20 of 29 posts

Re: Scaling Go Testing with Contract and Scenario Mocks

#11

Post by aranw »

SPascareli13 wrote: post #3

I really dislike this idea of testing in go: only ever use an interface, never the real implementation + mockgen the mocks based on this interface + use the mocks to assert that a function is called, with exactly this parameters and in this exact order. I find this types of tests incredibly coupled with the implementation, since any chance require you to chance your interfaces + mocks + tests, also very brittle and m…

> I really dislike this idea of testing in go: only ever use an interface, never the real implementation + mockgen the mocks based on this interface + use the mocks to assert that a function is called, with exactly this parameters and in this exact order.

Same I have zero confidence in these tests and the article even states that the tests will fail if a contract for a external service/system changes

Re: Scaling Go Testing with Contract and Scenario Mocks

#12

Post by aranw »

The problem with mocks is that they test your assumptions, not reality...

When you mock a CRM client to return one account, you're assuming it always returns one account, that IDs have a particular format, that there's no pagination, that all fields are populated. Each assumption is a place where production could behave differently whilst your tests stay green

Your contract tests use cached JSON fixtures. Salesforce changes a field type, your contract test still passes (old fixture), your mocks return the wrong type, production breaks. You've now got three test layers (contract, mock scenarios, E2E) where two can lie to you. All your contract and mock tests won't save you. Production will still go down

I have zero confidence in these types of tests. Integration tests and E2E tests against real infrastructure give me actual confidence. They're slower, but they tell you the truth. Want to test rate limiting? Use real rate limits. Want to test missing data? Delete the data.

Slow tests that tell the truth beat fast tests that lie. That said, fast tests are valuable for developer productivity. The trade-off is whether you want speed or confidence

Re: Scaling Go Testing with Contract and Scenario Mocks

#15

Post by preetamjinka »

tonyhb wrote: post #6
SPascareli13 wrote: post #3

I really dislike this idea of testing in go: only ever use an interface, never the real implementation + mockgen the mocks based on this interface + use the mocks to assert that a function is called, with exactly this parameters and in this exact order. I find this types of tests incredibly coupled with the implementation, since any chance require you to chance your interfaces + mocks + tests, also very brittle and m…

If you're testing the interface, changing the implementation internals won't create any churn (as the mocks and tests don't change). If you are changing the interface, though, that would mean a contract change. And if you're changing the contract, surely you wouldn't be able to even use the old tests? This isn't really a go problem at all. Any contract change means changing tests.

Yes, agreed. What the parent is saying about

> only ever use an interface, never the real implementation + mockgen the mocks based on this interface + use the mocks to assert that a function is called, with exactly this parameters and in this exact order.

is not ideal, and that's what we don't do. We test the real implementation, then that becomes the contract. We assume the contract when we write the mocks.

Re: Scaling Go Testing with Contract and Scenario Mocks

#16

Post by teeray »

mzi wrote: post #9
teeray wrote: post #2

> Mocks are static, but reality evolves. I learned “test your mocks” long ago from Sandi Metz, and that advice has paid off well for me. Have some set of behavioral conformance tests for the kind of thing you expect (e.g. any database worth its salt should be able write and read back the same record). Then stick your mock right under that same battery of tests alongside your implementation(s). If either deviate from…

> any database worth its salt should be able write and read back the same record This excludes a lot of cases, like just a simple postgres where reads are done from a replica.

You're free to come up with a better example. The point is that dependencies have behavioral properties that software depends upon. We can write tests for those behaviors. Mocks that are correct and remain so should implement those same behaviors we expect from the prod implementations of those dependencies.

Re: Scaling Go Testing with Contract and Scenario Mocks

#17

Post by fleahunter »

isuckatcoding wrote: post #13

Wow a full post about contract testing without mentioning pact https://pactflow.io/

Interesting point about mocks being seen as a bad word. I've been in situations where relying solely on integration tests led to some really frustrating moments. It's like every time we thought we had everything covered, some edge case would pop up out of nowhere due to an API behaving differently than we expected. I remember one time we spent hours debugging a production incident, only to realize a mock that hadn’t been updated was the culprit—definitely felt like we'd fallen into that "mock drift" trap.

I've also started to appreciate the idea of contract tests more and more, especially as our system scales. It kind of feels like setting a solid foundation before building on top. I haven’t used Pact or anything similar yet, but it’s been on my mind.

I wonder if there’s a way to combine the benefits of mocks and contracts more seamlessly, maybe some hybrid approach where you can get the speed of mocks but with the assurance of contracts... What do you think?

Re: Scaling Go Testing with Contract and Scenario Mocks

#18

Post by Thaxll »

aranw wrote: post #12

The problem with mocks is that they test your assumptions, not reality... When you mock a CRM client to return one account, you're assuming it always returns one account, that IDs have a particular format, that there's no pagination, that all fields are populated. Each assumption is a place where production could behave differently whilst your tests stay green Your contract tests use cached JSON fixtures. Salesforce…

Testing code is usually testing your code not that third party contract changed.

You make a lot of assumption about contract change which in reality should rarely happen.

Re: Scaling Go Testing with Contract and Scenario Mocks

#19

Post by kardianos »

My DB heavy app, when I run a Go unit test, it spins up a DB instance, populates it, runs the tests, and drops the DB. Never ever any mocks. The best part about unit tests isn't testing the Go code. It is testing the SQL.

Yes, it takes longer to run your tests. So be it.

Re: Scaling Go Testing with Contract and Scenario Mocks

#20

Post by preetamjinka »

kardianos wrote: post #19

My DB heavy app, when I run a Go unit test, it spins up a DB instance, populates it, runs the tests, and drops the DB. Never ever any mocks. The best part about unit tests isn't testing the Go code. It is testing the SQL. Yes, it takes longer to run your tests. So be it.

We do that too. We have hundreds of such tests. That establishes contracts.

We also have mocks. It’s not one way or the other. This post is talking about the mocking side of things.

Post reply on HN