Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How Long Is Your CI Process?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 81 posts

Re: Ask HN: How Long Is Your CI Process?

#21

Post by bastijn »

This is such an open question it is hard to answer. You have to know what runs in the CI as well as size of the project, languages, number of projects, quality steps executed in build, etc. Anyway, to give it a shot:

* multi-million Loc

* number of projects > 50

* languages C#, C, C++, typescript

* Frameworks: .NET Framework, .NET core, .NET standard, Angular, React

* Quality tools in build: TICS, Coverity, Roslyn, custom tools (>10)

* Tests running in build: nunit, msvstestv2, jest, karma

* number of tests running in build > 5000

* package managers used: Nuget, npm

* number of packages (private and public) > 500

Still a lot I forgot now.

It all runs in approximately 45 mins for stage1 builds, stage2-4 run nightly and weekly and take much longer (>2 hours to >24 hours for long duration stage 4). Increasing stages run longer test suites, up to approx 50k or so for stage 3 and 4, more quality checks, etc.

P.s. We spend countless hours reducing our build times. In addition we have setups to split build pipelines for those who do not need the entire archive build for their dev purposes etc. Yet, CI server aways runs single-core and cold builds.

Re: Ask HN: How Long Is Your CI Process?

#22

Post by megous »

formerly_proven wrote: post #16
jart wrote: post #7

2 minutes and 7 seconds https://github.com/jart/cosmopolitan/runs/2482398460 on travis for a repository that builds 14,479 objects, 67 libraries, and 456 static executables, 284 of which are test executables which are run too. If I want to run all the test binaries on freebsd openbsd netbsd rhel7 rhel5 xnu win7 win10 too, then it takes 15 additional seconds. On a real PC, building and testing everything from scratch…

This is absolutely ridiculous in the best possible way

Well, building Linux kernel takes 20s if you have the right machine https://openbenchmarking.org/test/pts/build-linux-kernel

And that's some many millions lines of C code.

Computing is fast these days.

Re: Ask HN: How Long Is Your CI Process?

#23

Post by nickjj »

For Flask apps, a little over 2 minutes to git push code and then see passing tests in CI using GitHub Actions.

Most of that time is spent building the Docker image.

The CI pipeline does:

- Build Docker images for the project

- Run the project

- Run Shellcheck on any shell scripts

- Run flake8 to lint the code base

- Run black in check mode to ensure proper formatting

- Reset and initialize the DB

- Run test suite

That's a baseline. At this point any increase to the ~2 min is a result of running more tests but it's usually possible to run about 100 assorted tests in ~10 seconds (testing models, views, etc.).

An example of the above is here: https://github.com/nickjj/docker-flask-example

A similar pipeline with comparable tools for Rails takes ~4-5 minutes and Phoenix takes ~4-5 minutes too. You can replace "flask" with "rails" and "phoenix" in the above URL to see those example apps too, complete with GH Action logs and CI scripts. These mainly take longer due to the build process for installing package dependencies, plus Phoenix has a compile phase too.

Re: Ask HN: How Long Is Your CI Process?

#24

Post by carlmr »

Usually it depends on what you need to run. Running some tests on an interpreted language should be done in a few minutes. With a compiled language it takes longer, maybe half an hour. If you have a compiled language that's slow to compile with additional checks, because the language is more footgun than anything (yes, C++) then you might want a standard build, build with various sanitizers, and do some static analysis, and you end up with hours of time spent on building and analysis.

It really depends on what you're trying to achieve and how big the project is.

Re: Ask HN: How Long Is Your CI Process?

#25

Post by openthc »

megous wrote: post #22
formerly_proven wrote: post #16

Earlier quoted context omitted.

This is absolutely ridiculous in the best possible way

Well, building Linux kernel takes 20s if you have the right machine https://openbenchmarking.org/test/pts/build-linux-kernel And that's some many millions lines of C code. Computing is fast these days.

Yea, computing is fast but @jart is also hella good with computers. This CI has two advantages.

edit: this will blow your mind https://justine.lol/ape.html

Re: Ask HN: How Long Is Your CI Process?

#26

Post by other_herbert »

It’s really down to goals... you should have something that triggers based on open pull requests and does a sanity check and ideally deployed to a test environment... that should be “quick” to give feedback in addition to reviews...

Then in the main build that hopefully is deploying to a qa environment that can do more testing, bundle artifacts for whatever dependencies need them, all that kind of stuff...

That’s how ours is set up... we use Jenkins with parallel parts where possible (like build the ui while tests that hit the db are run) it’s a process that takes time to get right and time to optimize...

We’re at about 5 mins for the quick part and 8 or so for the slower part

Both of those will probably get worse as we are planning to include full ui testing on the deployed environment too

Re: Ask HN: How Long Is Your CI Process?

#27

Post by megous »

Yes, they probably could be optimized. Backend API tests are usually fast, or can be made so. Webdriver based tests are annoying to write and usually slow, so I don't test frontend code automatically.

Kinda feels like a waste of time, especially if your code is well componentized and there are not many central points of failure. (and those are pretty easy to see with cursory manual testing)

Re: Ask HN: How Long Is Your CI Process?

#28

Post by rubyn00bie »

Oh yeah; this was the best part of moving to Erlang/OTP... Test suites are absurdly fast (nearly) all the time. Most test suites I use take less than 10 seconds with anywhere from 100-1000 unit tests. The worst I've seen "monoliths" take at most ~90 seconds to run and that is only ever the case if folks are creating insanely too many objects or needlessly testing the internals of `gen_server`.

Re: Ask HN: How Long Is Your CI Process?

#29

Post by dmoy »

Automated tests that my team runs vary from ~seconds to multiple days, depending on what's being tested. Some of the tests involve compiling a multi-billion line repo using over 30+ languages, and doing some analysis on the resulting code graph. So that takes awhile.

30-45 minutes just for a simple test suite, even if it's PHP, Python, and Ruby - that sounds long. But without any details on exactly what's being tested, it's hard to say.

Re: Ask HN: How Long Is Your CI Process?

#30

Post by thinkafterbef »

We had the same kind of problem, where we saw our tests and builds take 20-30 minutes. We also noticed that our own machine could run the tests significantly faster, mainly due to that desktop CPU can easily boost the clock speed for intense work load. Comparably most CIs use Cloud VMs which hardly go beyond 3 GHz. We found this quite strange.

After some talk we decided to build a CI service based on this premise, i.e desktop CPU outform Cloud CPUs for the CI use case. After some months we managed to create BuildJet.

I would say it at minimum cuts the the build time in half and the best part is that it plugs right into Github Actions, just need to change one line in your Github Actions configuration.

If it sounds useful for you, check it out: https://buildjet.com

Post reply on HN