Ask HN: How Long Is Your CI Process?

Yes, this is for the Kythe (née Grok) team at Google. We build a giant cross reference graph of the codebase. So - most teams don't have to build the whole codebase, but the Kythe team does.

Here is a somewhat outdated talk given by our tech lead: https://www.youtube.com/watch?v=VYI3ji8aSM0. That doesn't really get into any of the CI/CD stuff though, I don't know what if any of that stuff is publicly shareable.

From 2 minutes to 10 minutes. We have mostly Go microservices so building is fast.

The pipeline is: build, unit test, lint in parallel, then package and save the relevant artifacts, then build a Docker image, then run the integration tests, and finally deploy (staging, dev or prod depending on the branch).

We also have end to end tests that run periodically and are a bit longer, but they're not on the path to prod.

CICD systems deliver value to audiences. CI is mostly for the developer team, so you can check your changes don't break other's work, or vice versa. Often there's a CD to an internal system, so QA can take a look to see the new feature works according to the business expectations, and the business can play with it.

None of the above really matters, the important bit is that USERS actually see the work! Everything else is necessary, of course, but doesn't create value in itself.

So, the question is, how does each system create VALUE for its audience, and what's the latency (LAG)? CI is often for 4-10 developers, and takes ~10-20 minutes for smallish web shops. The value the business gets is that devs can check they didn't forget to "git add" a file :)

Devs and the business always complain about the slowness of CICD, but rarely invest the modest effort to make it faster. Here are some ways to improve the development cycle:

Speed up databases. Move from "install database and sample data interactively every time" to having a pre-baked Docker image with the database and seed data. Much faster: you get lower LAG and the same VALUE for the team.

Run fewer tests. Running tests creates business value -- confidence a deployment will give features to users -- but takes time (LAG). However, for 90% of the cases Devs get value by running a subset of the tests. Thus, much faster: less LAG, same VALUE. Run all the tests before a real deploy, or run the full suite nightly. Devs get the value of a full test without having to wait for it.

Simplify. CICD should just run things Devs can run locally. That is, Devs can run fast local test subsets to get rapid feedback (low LAG), and get focused VALUE. When CICD tests fail, it's very easy for Devs to figure out what went wrong, because CICD and local environments are nearly identical.

CICD creates a lot of value for several audiences. Plot out each one, and see what you, the business, want to improve upon!

The system I described is actually a cloud system, and we had both stubs and mocks of all our dependences (which is easy, because they were other cloud systems and we could easily stand up a fake service with the same API when doing integration tests, or switch to use local data when doing unit tests).

We also performed testing against live dependencies but with test accounts to ensure that our stubs/mocks were accurate and up-to-date, and captured realistic interactions (and failures).

I've done the same with hardware systems, again using stubs/mocks of HW dependencies for unit tests and then using actual HW for integration testing.

The time spent investing in stubs/mock quickly pays dividends in both increased development speed and test coverage, especially as you can inject faults and failures (bad data, timeouts, auth failures, corruption, etc).