A Two Month Debugging Story

Oof - seems like you're running on a managed platform that can't be debugged in a reasonable level of detail?

If it were me I'd want to take the problem somewhere I _could_ get root access and debug it properly. So I'd be interested to know what value this (unnamed) CI platform provides to make it worth a wild goose chase.

I always like to read stories of other teams' debugging adventures. One thing that occurred to me as I got toward the end of OPs (ongoing) narrative was that perhaps its time to fall back and consider other ways of setting up the data environment for each test. If the data that each test is dependent on is a small enough subset perhaps it makes sense to create a separate database for each test or class of tests, which can just be dropped and recreated before the test run?

Additional: Maybe it's just me but not being able to debug drives me CRAZY, sorry that I'm not being more helpful :) I've done bits of embedded systems work where you have to chisel out your own debug tools from glue and LEDs and serial ports and I basically hated it, or rather learned a bunch of ways of making it "right first time".

Test technology surely doesn't need that level of secret sauce that you need a hosted service - after two months I'd definitely want another way of doing it.

> To ensure each test has a clean slate, we clear the database between each test.

The proposed solution to manually nuke the database state seems crazy to me. Some alternatives:

1. Run the entire test in a transaction, do flushes and assert as normal. At the end, ROLLBACK instead of COMMIT and now you have a pristine database again. [1]

2. Setup pristine DB state once and then use `CREATE DATABASE ... WITH TEMPLATE ...` to create a temporary database. Not sure what the perf hit is, but it's probably worth trying. [2]

[1] http://alextechrants.blogspot.ca/2013/08/unit-testing-sqlalc...

[2] https://www.postgresql.org/docs/9.4/static/manage-ag-templat... CREATE DATABASE actually works by copying an existing database. By default, it copies the standard system database named template1.

I usually try to minimize the number of tests that rely on this (shared) DB state. So I'll focus more on unit and integration tests. Functional tests still do have value though. See: http://xunitpatterns.com/Testing%20With%20Databases.html

Where possible, I like to have data that can exist independent of the other data on the system. I can make a separate 'tennant' for that test - and just ensure it's wiped before I proceed. Sort of a multi-tennant approach. Works great. I don't bother with a 'teardown', but do any cleanup before the test runs. I also ensure the tests are written to not make assumptions about global state.

Instead of dropping all constraints as the article suggested (that sounds hacky), I use ON DELETE CASCADE constraints. If I miss some, the tests fail. Seems easy enough to maintain.

With the above approach, DB testing is approachable and still pretty quick.