Who cares if all you're doing is optimizing the massive spam operation that is Pinterest? You've made garbage faster. Well done.
A one-line change decreased our build times by 99%
81–90 of 256 posts
Re: A one-line change decreased our build times by 99%
#82Earlier quoted context omitted.
I'm not sure this is complacency - this just seems like regular old tech debt. The build takes 40 minutes but everyone has other things to do and there is no time to tend to the debt. Then one day someone has some cycles and discovers a one line change fixes the underlying issue. I'm sure many engineering projects have similar improvements that just get a ticket/issue opened and never revisited due to the mountain of…
IMHO (from having addressed such CI issues personally on teams that otherwise wouldn't bother) it's likely due to other factors, like a lack of interest, being scared of breaking the build, not being terribly comfortable touching build scripts, or the inability to run scripts locally, than a genuine lack of time. The returns you can get can be ridiculously huge across the entire team compared to the hours you might s…
For me the biggest issue is that CI is often siloed to hell and back.
Even when most of the rest of the engineering environment is fine, the build scripts and configuration often aren't under version control themselves, or are manually deployed - meaning any changes require access to carefully guarded server credentials. This may even be by design as a "security measure" - as if I didn't already have the ability to run arbitrary code on the build servers in question through unit tests etc. The gatekeepers in question are often an underfunded IT department that has too much on their plate already, and are underwhelmed by the idea of reviewing a bunch of changes to "legacy" code that they've somehow convinced themselves they'll rewrite "soon" that they don't directly benefit from anyways.
And I find I can rarely run the scripts locally. They're also often hideously locked in to a specific CI solution that I can't locally install without a ton of work on my part to figure out the mess of undocumented dependencies, and rife with edge cases that I can't easily imitate on my dev machines.
My preferred CI setups involve a single configuration file, checked into the same repository it's configuring CI for, that simply forwards to a low-dependencies script that works on dev machines. Getting there from an existing CI setup, however, can be quite the challenge.
Re: A one-line change decreased our build times by 99%
#83Earlier quoted context omitted.
I would expect that internally someone profiled the build (i.e. looked at timestamps) and then either profiled git, or just looked at the logs and did some guessing/research. This didn't seem like it would be complicated to find once you realize the time is spent in git. Also, this probably has been an exponentially increasing problem, and wasn't really a priority to solve until relatively recently. I would bet there…
It really doesn't sound complicated to find, unless you just have a handsoff approach to building things and just don't care as long as "something" comes out on the other end. What makes me wonder however is this: 40 min made them look into this? I mean 40 min is crazy long. What builds this long? Chrome, Windows, Linux Kernel on a single core? This should have been raising red flags much earlier. The only explanatio…
It takes >12h to build Windows on the MS build platform.
On a single core, Chromium surely takes hours to build.
Though I agree that 40min for the repository in question is highly suspect.
Re: A one-line change decreased our build times by 99%
#84I sympathise a lot with this post! Git cloning can be shockingly slow. As a personal anecdote, clones of the Rust repository in CI used to be pretty slow, and on investigating we found out that one key problem was cloning the LLVM submodule (which Rust has a fork of). In the end we put in place a hack to download the tar.gz of our LLVM repo from github and just copy it in place of the submodule, rather than cloning i…
> Contributors want to do their work, not investigate "what does a git clone really do".
Exactly this. Especially if the repo and CI pipeline are complicated, it is incredibly easy to just assume “it’s slow” is a fact of life.
And from the point of view of the dev-productivity team, well, they have tons of possible issues to deal with at any given time. Not just CI but the repos themselves, the build system, maybe IDEs, debuggers, ... Sure the fix ends up being easy but you have to know to go looking for it.
Re: A one-line change decreased our build times by 99%
#8560K pulls per day for 100 commit in a day? What tests are being done that can't leverage earlier pulls?
Re: A one-line change decreased our build times by 99%
#86Parkinson's Law of builds. "work expands so as to fill the time available for its completion", or in this case the available time is the point at which people can't stand the build taking too long. 30-60 minutes is normal because anything > 1 minute required you to context-switch anyway, and > 60 minutes means you are now at risk of taking a day if you have a work queue of a 1-pizza team. So [1..60] range causes a grumble but nothing will be done.
Re: A one-line change decreased our build times by 99%
#87I’ve found as an industry we’ve moved to more complex tools, but haven’t built the expertise in them to truly engineer solutions using them. I think lots of organizations could find major optimizations, but it requires really learning about the technology you’re utilizing.
We have not "as an industry" moved to git. There's a vocal subset of git fans, but it is by no means an industry standard.
Is Git not the most used VCS?
Re: A one-line change decreased our build times by 99%
#88Earlier quoted context omitted.
They set out to reduce build times, not to reduce git checkout times. It turns out that 99% of the entire build was spent downloading code.
This isn't true either, as the article says that builds went from 40 minutes to 30 minutes. The time spent cloning was presumably about 10 minutes and came down very far, presumably by 99%.
> the article says that builds went from 40 minutes to 30 minutes.
Where in the article does it say that? The article says this:
> This simple one line change reduced our clone times by 99% and significantly reduced our build times as a result. Cloning our largest repo, Pinboard went from 40 minutes to 30 seconds.
Both of those sentences say the clone time was reduced by 99%. There are percentage numbers given for how much the build time was reduced, nor any numbers about the total build time.
Re: A one-line change decreased our build times by 99%
#89Better title: A one-line change decreased our "git clone" times by 99%. It's a bit misleading to use "build time" to describe this improvement, as it makes people think about build systems, compilers, header files, or cache. On the other hand, the alternative title is descriptive and helpful to all developers, not only just builders - people who simply need to clone a branch from a large repository can benefit from t…
They set out to reduce build times, not to reduce git checkout times. It turns out that 99% of the entire build was spent downloading code.
Where does the article say "99% of the entire build was spent downloading code"?
Re: A one-line change decreased our build times by 99%
#90I’ve found as an industry we’ve moved to more complex tools, but haven’t built the expertise in them to truly engineer solutions using them. I think lots of organizations could find major optimizations, but it requires really learning about the technology you’re utilizing.
We have not "as an industry" moved to git. There's a vocal subset of git fans, but it is by no means an industry standard.
What industry are you part of?
In many domains git has replaced other version control systems.
I would love to see a new approach to version control. Things like subversion or mercurial have exposed too many drawbacks for them to win back industry.