Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A one-line change decreased our build times by 99%

medium.com

Post reply on HN
Print view View original

151–160 of 256 posts

Re: A one-line change decreased our build times by 99%

#151

Post by sercankd »

saagarjha wrote: post #148
alkonaut wrote: post #144

Earlier quoted context omitted.

Sorry no. If an article is paywalled, on Pinterest, or similar, then please let's discuss the source instead, even if it ruins the discussion, so people learn not to post such links.

Paywall complaints are explicitly off-topic: https://news.ycombinator.com/item?id=10178989 . I am not a moderator, but I think I've made it clear that I personally consider comments like the one I responded to be as well. FWIW, in the all years I have been on this site, I have seen this happen regularly and I have yet to see any reduction in such links or these kinds of discussions. Seeing as you've been here longer,…

We heard your complaint but you are being acting entitled now. People are free to register, free to comment, if you don't like it, downvote it. It is the top comment, that means it is being upvoted. get over it.

Re: A one-line change decreased our build times by 99%

#153

Post by whatatita »

yen223 wrote: post #92
YokoZar wrote: post #66

Can someone explain the intended meaning behind calling six different repositories "monorepos"? It sounds to me like you don't have a monorepo at all and instead have six repositories for six project areas.

My interpretation is that each "monorepo" is a big git repository that consists of a collection of individually-deployed services, as opposed to having a single git repository per service. I do not know whether that's what the blog author meant by that though.

I got that impression too. I can imagine the Pintrest monorepo, for example, has the website and server code together.

Their iOS and Android repos may contain the code for multiple apps. Though, I'm not aware of which other apps Pintrest (the company) creates besides the obvious one.

Re: A one-line change decreased our build times by 99%

#154

Post by bald42 »

JOnAgain wrote: post #2

I think it takes some real humility to post this. No doubt someone will follow up with an “of course...” or “if you don’t understand the tech you use...” comment. But thank you for this. It takes a bit of courage to point out you’ve been doing something grotesquely inefficient for years and years.

I don't get why they have to clone their repo frequently in the first place - seems to me as a brute force usage of a version control system prone to high cost in the first place.

Re: A one-line change decreased our build times by 99%

#155

Post by Dylan16807 »

Well, good advice, and good for them, but

> Cloning monorepos that have a lot of code and history is time consuming, and we need to do it frequently throughout the day in our continuous integration pipelines.

No you don't!

If removing per-build clones was the only way to speed things up, I'm absolutely sure you could figure out how with medium difficulty at most.

Re: A one-line change decreased our build times by 99%

#156

Post by mschuster91 »

bald42 wrote: post #154
JOnAgain wrote: post #2

I think it takes some real humility to post this. No doubt someone will follow up with an “of course...” or “if you don’t understand the tech you use...” comment. But thank you for this. It takes a bit of courage to point out you’ve been doing something grotesquely inefficient for years and years.

I don't get why they have to clone their repo frequently in the first place - seems to me as a brute force usage of a version control system prone to high cost in the first place.

Ephemeral CI runners. I have the same problem at work - 4GB repository that is redownloaded on every single pipeline run.

Another reason (which is why we went for ephemeral runners in the first place...) is that if you have stuff that mounts a directory from the repository directory as a volume in a Docker container (e.g. for processing data), you may end up with the Docker container frying permissions in the repo folder (e.g. 0:0 owned files). Now, you can put a cleanup step as part of the CI (=docker run --rm -v $(pwd):/mnt sh -c 'chown -R $runner_uid:$runner_gid)... but unfortunately, Gitlab does not allow a "finally" step that always gets run, so in case the processing fails, the build gets aborted, the server hosting the runner crashes, ... anything happens, the permissions will be fried, and a sysadmin will need to manually intervene.

An ephemeral runner using docker:dind however? It simply gets removed.

Re: A one-line change decreased our build times by 99%

#157

Post by ForHackernews »

bald42 wrote: post #154
JOnAgain wrote: post #2

I think it takes some real humility to post this. No doubt someone will follow up with an “of course...” or “if you don’t understand the tech you use...” comment. But thank you for this. It takes a bit of courage to point out you’ve been doing something grotesquely inefficient for years and years.

I don't get why they have to clone their repo frequently in the first place - seems to me as a brute force usage of a version control system prone to high cost in the first place.

I don't know about a big org like Pinterest, but it's pretty common for "clone the repo" to be the first step of a CI/CD pipeline when using something like CircleCI or GitlabCI.

It's an easy (if inefficient) way to always get the latest changes and if you have disposable build-runners then it all gets thrown away at the end of the pipeline.

Re: A one-line change decreased our build times by 99%

#158

Post by slx26 »

temporallobe wrote: post #29
gpapilion wrote: post #9

I’ve found as an industry we’ve moved to more complex tools, but haven’t built the expertise in them to truly engineer solutions using them. I think lots of organizations could find major optimizations, but it requires really learning about the technology you’re utilizing.

This, this, so much this. When we build more complexity into a system, the less we understand it, similar to how development frameworks create multiple layers of abstraction to the point where the developers have no idea what actual code the framework produces, much less how to fix it.

Yes, we probably need people to stop thinking about tools as if they "solved" problems; what they really do is "transform" them. Now instead of having to deal with the original problem, you only need to deal with part of it and part of the new problem of using the tool that's supposed to help you, plus any leaks you might have because tools rarely solve problems perfectly. It's a trade-off, and you need to be aware of these transformations.

Re: A one-line change decreased our build times by 99%

#159

Post by Dylan16807 »

pushrax wrote: post #83
marta_morena_28 wrote: post #76

Earlier quoted context omitted.

It really doesn't sound complicated to find, unless you just have a handsoff approach to building things and just don't care as long as "something" comes out on the other end. What makes me wonder however is this: 40 min made them look into this? I mean 40 min is crazy long. What builds this long? Chrome, Windows, Linux Kernel on a single core? This should have been raising red flags much earlier. The only explanatio…

It takes >12h to build Windows on the MS build platform. On a single core, Chromium surely takes hours to build. Though I agree that 40min for the repository in question is highly suspect.

I read that comment as only applying the "single core" part to the Linux kernel, not Chromium.

Re: A one-line change decreased our build times by 99%

#160

Post by Dylan16807 »

Thorrez wrote: post #89
CGamesPlay wrote: post #32

Earlier quoted context omitted.

They set out to reduce build times, not to reduce git checkout times. It turns out that 99% of the entire build was spent downloading code.

Where does the article say "99% of the entire build was spent downloading code"?

The title. If they reduced the build time by that much, then at least that much of the build time must have been spent downloading code.

If the title is a lie (which it probably is), then nevermind that number, but it's clear where it came from.

Post reply on HN