Poll: Do you test your code?

Unfortunately our sales people are obsessed with agreeing to whatever customers dictate in order to make a sale. The customer wants a full featured, fully customized, fully automated E-commerce solution and they want it for a flat $5000? Sold. Customer says "What is this 'testing' sh*t on the quote? It should just work the first time, or do you only have a Jr developer on staff who needs everything double checked for them? We can go some place more professional" and sales person replies "Oh yeah, that - you're right. Our developer is a wizard and I forgot to take that off."

No matter how many times I explain or quote higher or tell them the feature creep is becoming unreasonable (oh by the way, we have 18 products with complicated interactions, not the 3 we asked for on the quote, but we expect to still pay the same), such that I can't possibly write it all and test it all, they just don't listen and they leave me holding the bag. So, while I'd like to do testing, just getting the thing kind-of working isn't in the budget, never mind getting it working well.

Sorry for the rant and... come to think of it, it may be time for a new job.

If all your algorithms are as trivial as calculating a weighted modulo 11 checksum, then the sort of case I'm thinking of doesn't apply. However, in real code, we sometimes have to model situations and solve problems that are inherently complex. The algorithms and data structures we work with will necessarily reflect that essential complexity, and ultimately so will our code.

Beyond a certain point, I think automated tests that give simple yes/no answers are no longer a particularly effective way to test certain types of complex algorithm. Sometimes there are just too many possible inputs and interactions between different effects to get a sensible level of coverage and draw any useful conclusions from that kind of testing alone. You might still have some automated tests, but they are more like integration tests than unit tests at that point.

Internally, you could try writing almost-unit-tests for the implementation details, but then you get into the usual concerns about backdoor access and tying tests too closely to implementation details that might change frequently. Alternatively, some form of careful algorithm design with systematic formal proof might be called for. Maybe instrumenting the code and checking the actual values at key points will highlight errors that aren't yet manifesting as faults, things that a boolean automated test would miss because they haven't violated some arbitrary threshold but which form an unexpected pattern to a knowledgable human observer. However, in these cases, you really want the code to be as simple as possible, and hooks to permit internal access to run some automated test cases as well could cause an awful lot of clutter.

Everyone says they do Unit Tests, but no one "really" does it.

Thanks for the link.

So: what about GUIS? How does one formally describe pixels displayed on a screen in a way that captures their ability to correctly communicate with the human intended to view them? So that my notice

  Please turn off the foo within bar

doesn't show up truncated to say

  Please turn off the foo

on all reasonably applicable screens? This example is just one of a googleplex of possible failures.

I strongly feel you should try to add one test in each category. That adds a sanity check and lowers to cost to adding more tests when you really need it.

It's pretty painful to think "oh, this really needs a test, but I haven't got a test suite set up and besides, I don't know to write a test of this kind".

Writing tests for edge cases we see in production is the most valuable thing we do. We use Airbrake to find the bugs, and then we add a test for it, if possible (it's not always possible).

That gives us good confidence that other changes aren't fucking things up. It's also a pretty sane strategy for growing a test suite when you inevitably have some portion of your code which has no tests at all.