10 Minutes if no one modifies the "Global library". However 90% of builds modifies it so it takes 1h30.
Ask HN: How Long Is Your CI Process?
51–60 of 81 posts
Re: Ask HN: How Long Is Your CI Process?
#52Earlier quoted context omitted.
Multi-billion? Wow. Google? Microsoft?
Yes, this is for the Kythe (née Grok) team at Google. We build a giant cross reference graph of the codebase. So - most teams don't have to build the whole codebase, but the Kythe team does. Here is a somewhat outdated talk given by our tech lead: https://www.youtube.com/watch?v=VYI3ji8aSM0 . That doesn't really get into any of the CI/CD stuff though, I don't know what if any of that stuff is publicly shareable.
This talk is FASCINATING!
Not every organization maintains gigantic polyglot codebases, and it's interesting to see what kinds of challenges arise when that's the operational reality. I would never have realized the need for something like Kythe, because I largely work on codebases that are written in one language.
I feel like a lot of what you all are working on might one day osmose nicely into something CI systems and cloud-based analysis tools use to decouple themselves from individual language semantics - kind of like how MapReduce, Bazel, Kubernetes, et al found brand new use cases outside of Google's organization years after they were invented and instituted.
Re: Ask HN: How Long Is Your CI Process?
#53I don't see how anyone can give you useful information without knowing more about the pipeline and the projects, and as everyone's pipelines/projects are going to work differently (I do web dev work, so pipelines are relatively simple, I can imagine that a game dev team creating Windows/Mac/Linux builds might have multi-hour pipelines though). Anyway as the question is "How Long Is Your CI Process", here we go! I hav…
Computing is cheap these days. If somewhere has a multi-hour build, that is a noxious build system smell. Run away. Or make sure the role involves you getting paid to optimize it. If a build takes 2 hours, and you work 8 hours days, it means you get four tries in one day to get it right? Four!
That is no way to do modern computing. Things were worse back in the day (we compiled uphill, both ways), but we're not in those days any more. Buy bigger & faster servers with more RAM until the problem goes away. It won't necessarily be cheap. But if the company is too stingy, and would prefer to be pennywise and pound foolish (saving "pennies" on a server vs developer time to sit there waiting for "compiles"), you don't want to work there.
Re: Ask HN: How Long Is Your CI Process?
#54Similar scope to yours but Java and Gitlab CI. The CI to dev takes about 15 minutes or so. A shared lib is first built and tested, then several applications are built and tested in parallel. After everything is built, things are deployed serially. About 3 of those 15 minutes are startup times for runners (no clue what we use, but it's super slow), another 2-3 for deployment, about 1:30 for compiling and the rest is E2E tests. The whole thing takes about 4 minutes on a ~2015 era Mac.
Re: Ask HN: How Long Is Your CI Process?
#55It's going to depend on the size of your code base and tests but when I was at AWS by the time we built and ran our test suite it was close to that time as well (30-45 min).
It's really interesting how many companies these days have a primary pricing model of build minutes.
If you are looking for a DIY solution for your CI, check out https://tinystacks.com. We have the fastest way to launch and operate your Docker app on AWS. In one click, we setup infra and an automated pipeline on your AWS. Uses ECS with Fargate. All setup for you with a control center for logs, env vars and scaling. No config nightmare.
Email me safeer at tinystacks.com and I can get you onboarded.
Re: Ask HN: How Long Is Your CI Process?
#56About 10 minutes. 1-2 minutes to refresh a docker container, ~2 minutes to build a mostly C codebase in the docker container, ~5 minutes to build a bunch of python environments and run unit tests in them. 3 or so minutes due to bad design choices in Gitlab-CI. Project has around 100k loc.
I think we could get it down to 3 minutes or so if we changed some things, but 10 minutes vs 3 minutes doesn't really change the workflow for us.
Re: Ask HN: How Long Is Your CI Process?
#57Roughly 30-40 minutes before tests, and another 30 minutes of tests. I work in games, and compiling the game on one platform is ~5-10 minutes even for an incremental compile on "compute optimized" instances on azure and aws (compared to ~30s on my workstation). It takes 20 minutes (per platform) to generate the runtime texture/audio files, and ~10 minutes to upload them to a shared drive. We do 4 platforms right now, my last project was ~10 bigger and did 10 platforms to boot.
Re: Ask HN: How Long Is Your CI Process?
#58About 5-10 minutes from push to deploy. Python/Django monorepo. 1-2 devs for several years kinda project size.
Build and test steps take about equal time. We build from a common docker image which has most of the time consuming work already done.
It can take longer if the Python deps have changed and therefore the ‘poetry install’ step cannot be pulled from the cache.
Also, we deploy multiple individual Django projects, rather than one huge monolithic project. That probably gives some speed up. It means that changes to common code can trigger 5-15 pipelines, but they all take a similar amount of time.
30-45 minutes seems like a really long time to me. Maybe you have a lot of slow tests, but I’d also looking at the build process too. If you’re doing docker images you may find you can extract a lot of the time consuming work to a common base image. You can also get plugins that help docker pull already-built layers from a cache.
If it is the tests then you could always try running tests in parallel. One worker per CPU or some such.
FWIW - I find that these long feedback loops can really kill productivity and morale. 10 mins for a deploy is about my limit.
Re: Ask HN: How Long Is Your CI Process?
#59The D compiler takes about 25 minutes, GCC + D frontend tests takes about an hour.
There is absolutely a huge amount of room for performance in areas like this. With Python especially it's very common so think "Ah yes but numpy", when it comes to performance, and that is true in steady state where you are just number crunching, but there is mindnumbingly large amounts of performance left on the table vs. even a debug build with a compiler. Testing in particular is lots of new code running for a short amount of time, so it's slow when interpreted.
Re: Ask HN: How Long Is Your CI Process?
#60The latency of your CI process doesn't matter[1]. What matters is the development process - local build & test should be fast. Otherwise, with CI/CD, it's a continually-moving release train where changes get pushed, built, tested, and deployed non-stop and automatically without human intervention. Once you remove humans from the process, and you have guard rails (quality) built into the process, it doesn't matter if…
That's only if you can (and do!) test everything locally. Otherwise, you are still beholden to the CI system.