Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop mocking your system

blog.bitgloss.ro

Post reply on HN
Print view View original

51–60 of 178 posts

Re: Stop mocking your system

#51

Post by alecbz »

True "unit" tests are:

* faster to run

* give you less confidence in the correctness of the system (per time spent writing them)

* when they fail, give you more information about where the failure is

The more integration-y/e2e-y a test is, the more it strays from this: slower to run, more confidence that the system is correct, less info about where the failures are.

I think people have learned to undervalue the properties of integration tests and overvalue the properties of unit tests. Is it nice to know exactly what's broken based on the test failure? Sure. Is it _as_ nice as having confidence that the whole system is working? Probably not, in a lot of cases.

(I don't think there's general rules about which kinds of tests are easier to write. Sometimes setting up a real version of a component for test is harder, other times setting up a mock version for the test is harder.)

Re: Stop mocking your system

#52

Post by astuyvenberg »

cortesoft wrote: post #33
astuyvenberg wrote: post #13

Earlier quoted context omitted.

For AWS specifically, I prefer to have an AWS account specifically dedicated to each service + stage. For example, if I have an image service that handles s3 uploads (say Lambda, S3, Cloudfront and API Gateway), then I'd deploy a "test" environment to a dedicated AWS account and run tests against that. Since it's fully serverless, it only costs a few pennies to test (or free). I try not to develop locally at all anym…

That means that every person who runs the tests needs credentials for that AWS account. That obviously won’t work for an open source project. Even for a company project, how do you distribute those secrets? It adds friction for developers getting their local dev environment setup. Not only that, but you now need network access to run tests. A network blip or a third party service outage now makes your tests fail. The…

> That obviously won’t work for an open source project

On the contrary - I was an employee at Serverless Inc, working on the Serverless Framework for the last two years, we used this pattern extensively (and very successfully) in our open source repos.

You can even find an example here which provisions real live AWS infrastructure: https://github.com/serverless/dashboard-plugin/tree/master/i...

We used part of our enterprise SaaS product to provision temporary credentials via STS and an assumable role, and it works great. You could do the same thing with something like HC Vault.

For Lambda, S3, DynamoDB, the perpetual free tier means we've never paid to run our own tests. API Gateway isn't free (after 1 year), but it's still pennies per month. We've had several cases where tests stuck around a long time, but a billing alert and occasionally some CloudFormation stack cleanup takes care of that.

We still have offline unit tests which test business logic, but everything else runs against the cloud - even our development environments ship code straight to lambda.

Re: Stop mocking your system

#53

Post by fiddlerwoaroof »

NBJack wrote: post #37
wffurr wrote: post #20

Earlier quoted context omitted.

Yes, use stubs. But then also have integration tests. The point is to have easier to write lower overhead unit tests, and then have a few full fat integration tests that put everything together. Mocking is a terrible middle ground.

That still doesn't address my concern. I cannot test my S3 implementation without either mocks or a very specific emulator of the protocol. AWS happens to be popular enough that some libraries exist to do the latter, but I assert this is the exception rather than the rule for external integrations. You shouldn't be checking in code without at least some unit testing along for the ride, mocks or otherwise. It is indee…

I agree you need some integration tests, but, in my experience, if you define an interface the defines what you need from third parties, you can make 90% of the code you care about unit testable.

For me, this isn’t just theory: I’ve worked at a place that trained its employees to write code this way, and the benefit was obvious.

Re: Stop mocking your system

#54

Post by fiddlerwoaroof »

gregmac wrote: post #26

> QA says: “it doesn’t work”. Dev says: “it must be working, all the tests pass”. I've never experienced this, and I'm kind of doubtful this is a true reaction. I guess maybe it could happen in a team that is totally dysfunctional, where there's zero trust between QA and developers .. but in that case mocks are not the problem. The realistic reaction is "huh, guess we are missing some test cases".

Yeah, when I’ve run into this sort of bug, I always try to figure out how to make the unit tests reproduce the bug as a failing test case before attempting to fix the bug.

Re: Stop mocking your system

#55

Post by fiddlerwoaroof »

mahidhar wrote: post #22

I've generally had a little bit more success with mocking when I'm hiding that dependency behind my own interface. So for example in Java, instead of trying to mock the AWS provided class, I write my own class (like a facade or repository pattern) which has a very simple interface of a success case and maybe a couple of relevant failure cases. It's calling the AWS library within it. But my mocks are at the level of m…

I think the nice thing about this is that the facades usually don’t change: you might add methods, but, once the code is written, it only changes if you’re doing something major like switching databases. Code that is written and then never changed tends to be less buggy than code that changes, so this sort of pattern tends to reduce bugs at integration points.

Re: Stop mocking your system

#56

Post by curun1r »

alecbz wrote: post #51

True "unit" tests are: * faster to run * give you less confidence in the correctness of the system (per time spent writing them) * when they fail, give you more information about where the failure is The more integration-y/e2e-y a test is, the more it strays from this: slower to run, more confidence that the system is correct, less info about where the failures are. I think people have learned to undervalue the prope…

The other point about unit tests is that they allow you to test far more permutations. At a micro level, you can test more of the possible types of inputs to your function. When you move to integration tests, the permutations multiply to the point that it would require billions of tests to test the same range of inputs.

This is why both are so necessary. Integration testing tests whether the whole thing is coherent when put together. But it’s terrible for testing edge cases and error scenarios. That’s where unit testing comes in. It gives the developer a chance to codify every edge case they’ve considered to automate the process of catching regressions.

Re: Stop mocking your system

#57

Post by jeppester »

Another fanatic blog post about how something is always correct and another thing is always wrong.

Blog posts like this will lure you into thinking that there is a single right approach. Don't fall for it.

Do what makes sense. If it doesn't work try something else the next time. Becoming good is about growing your ability to make the right calls, not blindly following a methology.

Re: Stop mocking your system

#58

Post by ternaryoperator »

Regardless of what this article says, mocks are really useful when your project is partially written and you want to test the pieces you have in hand. Mocking then is absolutely the right approach and might entail mocking many things (which will eventually be replaced by working code.)

Re: Stop mocking your system

#60

Post by nicoburns »

ramesh31 wrote: post #46
pydry wrote: post #30

Earlier quoted context omitted.

>I think this is supposed to mean using dependency injection instead. I'm pretty sure he means "just use integration tests".

>I'm pretty sure he means "just use integration tests". Really he means e2e testing. You can drive yourself mad writing integrations that break constantly or provide false positives. Better to unit test what is truly unit testable, and then rely on e2e to ensure your integrations are fine. Ultimately all that matters is the system running as expected at the user level.

I think that depends on the scope of what you're testing. If you have a database, a set of backend services and 1 or more clients (websites, mobile apps, etc), then I think it makes a lot of sense to test the backend services in isolation from the frontends (which would mean not e2e test), but backend by an actual database (so integration tests, not unit tests)
Post reply on HN