Ask HN: Legacy code – tests first or setup CI?

I've been working on a project for a large IT department to automate all the things. It started out small but more people are starting to use it, and also starting to contribute, adding features; it's starting to get some traction. This is a web based app (Python) that runs scripts to automate various infrastructure tasks, like building and patching servers. It's backed by a database and is very tightly coupled (although it's fairly modular, organized into packages, kind of like MVC). We had to move fast and there were just 2-4 of us, server admins learning to be developers, so we've never had a formal testing or CI/CD process. We do use version control. We have a dev branch, which is everyone's upstream, and once we're happy we merge that to a master branch and that's considered "production".

We know we need to get to CI/CD, unit tests, all the DevOps/Agile/XP stuff. I'm thinking since we don't have any tests at all, before we refactor the code to support unit tests or start integrating with Jenkins (CI), we should start with at least adding some high level tests. Maybe integration or UI tests... just something to make sure we're not breaking things as we clean up the code base.

Some people have suggested we start with getting it into Jenkins and set up an automated build (CI) system first, then start adding tests. I'm thinking start with tests first.

People are using it now, but there are a few bugs. Most of them are integration related - race conditions, DB sessions overlapping, things that come up when it's running under Gunicorn and handling a lot of requests. What's the best roadmap to get a legacy code base up to snuff to be considered a real product?

[edit] I should mention, everyone who develops on this runs it on systems that are almost identical to production. We use a requirements.txt, and the OS is at the same patch level and everything. Giving contributors a way to run tests locally against their changes on their dev boxes, with a testing framework, would be almost as good as running the app in production. Nobody's running this on their laptop or anything like that.

> before we refactor the code to support unit tests or start integrating with Jenkins (CI), we should start with at least adding some high level tests.

Yes, this is a good idea. before trying to make dramatic changes to internals (refactoring etc) it is a good idea to get some kind of automated tests around the system.

you'll get quite a lot of value from a small number of basic smoke tests / end-to-end integration tests.

> Giving contributors a way to run tests locally against their changes on their dev boxes, with a testing framework, would be almost as good as running the app in production

a nice way i've seen this done is to focus on automating a build script for launching the app and any dependencies it needs, running automated tests against it, then tearing everything down.

figure out how to automate into a script that developers can use to run automated tests

figure out how to automate the environmental set up (e.g. installing package dependencies)

once you know how to do those things reasonably reliably for dev machines, set up a jenkins CI build to do the same thing: check out the code, set up dependencies, then run the automated test script

Since you've got a web app, it may already be difficult to get automated reliable set up and tear down of dependencies under control. E.g. you use a DB - can you automate setting that up and populating it with test data before a test suite runs? Your app automates infrastructure tasks -- how can you test that without actually triggering the changes, etc?

Your situation sounds remarkably like the state of our (also Python) codebase 18 months ago.

Start with CI and a couple of trivial tests that don't require refactoring - a smoke test or an end to end test that verifies your home page returns a 200 status code. Expand from there.

I say this because tests are useless if they don't get run. When I finally got CI set up for our code base, my next three weeks were spent updating abandoned tests that had been written to verify a dependency upgrade and then hadn't been run in months.

The real win, though, was integrating CI with Slack. Before that nobody noticed if a build broke, and I would have to point it out to them. Displaying an alert publicly in the middle of our primary channel of communication made it something that couldn't be ignored.

Once you have CI running, developers will start to feel the pleasure of expanding test coverage and using tests instead of clicks to verify their work... and the pain of breaking the test suite.

Before we set up CI, every member of our dev team was somewhere between unenthusiastic and mildly resentful of requests for test coverage. That attitude has changed - most of our engineers now take pride in writing tests that pass, and one formerly resentful developer voluntarily chose to target 100% coverage (something we don't yet require) for the most recent feature he built.

Also, I want to echo another commmenter's suggestion about using a hosted CI service - assuming you're a small team. It'll get you up and running faster and require less maintenance overhead. Since we don't use Docker and have a fairly complex stack that won't run on typical services like CircleCI, we use a service that handles the queue, displays pass/fail, supports notifications, etc. but lets us run the test suite itself on our own boxes.

I agree that they go hand in hand. In practical terms however it can take a long time to write comprehensive test cases for legacy code, even if only writing integration and system tests and ignoring unit tests.

A pragmatic approach would be to write a few tests quickly first. Then implement CI as quickly as possible. After the CI is working end-to-end running the few tests you quickly setup before then you can focus on expanding coverage of testing. Basically, I'd suggest writing only enough tests to test the CI first :-) Then once that's in place you can add tests.

One other thing that is important to get right is reporting (at the end of your CI flow, once each build and test run has completed). This will help you keep on top of regression but most notably demonstrate the real value of what's going on with all the work you're putting in now.

You don't have contact information in your profile but I wanted to add that I've worked in the role of a software engineer focused on automation (CI and testing) at large tech companies and I felt so strongly about how important the reporting aspect of this is that I started a web-based reporting service that is growing in usage. Check my profile for details, I'm the founder, contact me with any questions or feedback.