The problem with mocks is that they test your assumptions, not reality...
When you mock a CRM client to return one account, you're assuming it always returns one account, that IDs have a particular format, that there's no pagination, that all fields are populated. Each assumption is a place where production could behave differently whilst your tests stay green
Your contract tests use cached JSON fixtures. Salesforce changes a field type, your contract test still passes (old fixture), your mocks return the wrong type, production breaks. You've now got three test layers (contract, mock scenarios, E2E) where two can lie to you. All your contract and mock tests won't save you. Production will still go down
I have zero confidence in these types of tests. Integration tests and E2E tests against real infrastructure give me actual confidence. They're slower, but they tell you the truth. Want to test rate limiting? Use real rate limits. Want to test missing data? Delete the data.
Slow tests that tell the truth beat fast tests that lie. That said, fast tests are valuable for developer productivity. The trade-off is whether you want speed or confidence
I really dislike this idea of testing in go: only ever use an interface, never the real implementation + mockgen the mocks based on this interface + use the mocks to assert that a function is called, with exactly this parameters and in this exact order.
I find this types of tests incredibly coupled with the implementation, since any chance require you to chance your interfaces + mocks + tests, also very brittle and many times it ends up not even testing the thing that actually matters.
I try to make integration test whenever possible now, even if they are costly I find that the flexibility of being able to change my implementation and not break a thousand tests for no reason much better to work with.
> I really dislike this idea of testing in go: only ever use an interface, never the real implementation + mockgen the mocks based on this interface + use the mocks to assert that a function is called, with exactly this parameters and in this exact order.
Same I have zero confidence in these tests and the article even states that the tests will fail if a contract for a external service/system changes
If you're testing the interface, changing the implementation internals won't create any churn (as the mocks and tests don't change).
If you are changing the interface, though, that would mean a contract change. And if you're changing the contract, surely you wouldn't be able to even use the old tests?
This isn't really a go problem at all. Any contract change means changing tests.
They mean the dependencies. If you’re testing system A whose sole purpose is to call functions in systems B and C, one approach is to replace B and C with mocks. The test simply checks that A calls the right functions.
The pain comes when system B changes. Oftentimes you can’t even make a benign change (like renaming a function) without updating a million tests.
Testing in Go is great, for unit testing. But anything above that should be done manually. "Mocking" never made sense to me and I never used it in my entire life. If I want to test a database/repository, I will spin up a real database with real data and data persistence instead of relying on in-memory storage and try to abstract away things that make no sense to do in the real world, as we're not living in a theoretical world of interfaces and adapters and whatnot.
I learned “test your mocks” long ago from Sandi Metz, and that advice has paid off well for me. Have some set of behavioral conformance tests for the kind of thing you expect (e.g. any database worth its salt should be able write and read back the same record). Then stick your mock right under that same battery of tests alongside your implementation(s). If either deviate from the behavior you depend on, you’ll know about it.
Bang on Sandi! I hadn’t heard that quote but she’s my favorite speaker on testing and OO.
Another way of looking at this advice is that every time there’s a mock there needs to be a test that shows that the real code can be used in the same way that the mock is used.
When you mock a CRM client to return one account, you're assuming it always returns one account, that IDs have a particular format, that there's no pagination, that all fields are populated. Each assumption is a place where production could behave differently whilst your tests stay green
Your contract tests use cached JSON fixtures. Salesforce changes a field type, your contract test still passes (old fixture), your mocks return the wrong type, production breaks. You've now got three test layers (contract, mock scenarios, E2E) where two can lie to you. All your contract and mock tests won't save you. Production will still go down
I have zero confidence in these types of tests. Integration tests and E2E tests against real infrastructure give me actual confidence. They're slower, but they tell you the truth. Want to test rate limiting? Use real rate limits. Want to test missing data? Delete the data.
Slow tests that tell the truth beat fast tests that lie. That said, fast tests are valuable for developer productivity. The trade-off is whether you want speed or confidence
I find this types of tests incredibly coupled with the implementation, since any chance require you to chance your interfaces + mocks + tests, also very brittle and many times it ends up not even testing the thing that actually matters.
I try to make integration test whenever possible now, even if they are costly I find that the flexibility of being able to change my implementation and not break a thousand tests for no reason much better to work with.
Same I have zero confidence in these tests and the article even states that the tests will fail if a contract for a external service/system changes
If you are changing the interface, though, that would mean a contract change. And if you're changing the contract, surely you wouldn't be able to even use the old tests?
This isn't really a go problem at all. Any contract change means changing tests.
The pain comes when system B changes. Oftentimes you can’t even make a benign change (like renaming a function) without updating a million tests.
I learned “test your mocks” long ago from Sandi Metz, and that advice has paid off well for me. Have some set of behavioral conformance tests for the kind of thing you expect (e.g. any database worth its salt should be able write and read back the same record). Then stick your mock right under that same battery of tests alongside your implementation(s). If either deviate from the behavior you depend on, you’ll know about it.
Another way of looking at this advice is that every time there’s a mock there needs to be a test that shows that the real code can be used in the same way that the mock is used.
This excludes a lot of cases, like just a simple postgres where reads are done from a replica.