Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A one-line change decreased our build times by 99%

medium.com

Post reply on HN
Print view View original

61–70 of 256 posts

Re: A one-line change decreased our build times by 99%

#61

Post by AdamJacobMuller »

I expected this to be some micro-optimization of moving a thing from taking 10 seconds to 100ms.

> Cloning our largest repo, Pinboard went from 40 minutes to 30 seconds.

This is both very impressive as well as very disheartening. If a process in my CI was taking 40 minutes I would be investigating sooner than a 40-minute delay.

I don't mean to throw shade on the pintrest engineering team, but, it speaks to an institutional complacency with things like this.

I'm sure everyone was happy when the clone took 1 second.

I doubt anyone noticed when the clone took 1 minute.

Someone probably started to notice when the clone took 5 minutes but didn't look.

Someone probably tried to fix it when the clone was taking 10 minutes and failed.

I wonder what 'institutional complacencies' we have. Problems we assume are unsolvable but are actually very trivial to solve.

Re: A one-line change decreased our build times by 99%

#62

Post by senkora »

pushrax wrote: post #58
MarkSweep wrote: post #52

Earlier quoted context omitted.

I can’t speak for the original post, but I’ve seen other people[1] increase the commit count because part of the build process looks for a specific commit to checkout after cloning. If you have pull requests landing concurrently and you only clone the most recent commit, there is a race condition between when you queue the build with a specific commit id and when you start the clone. All that being said, I don’t know…

Also worth noting that git is pretty efficient at cloning a bunch of subsequent commits, due to delta encoding. edit: looks like git doesn't implement fetching thin packs when populating a shallow clone. It will still avoid fetching unnecessary packs, so the efficiency is still high for most software repositories.

Does git do delta encoding during clones? I know it doesn’t use deltas for most things.

Re: A one-line change decreased our build times by 99%

#63

Post by [deleted] »

chrisweekly wrote: post #54

I find the self-congratulatory tone in the post kind of off-putting, akin to "I saved 99% on my heating bill when I started closing doors and windows in the middle of winter." If your repos weigh in at 20GB in size, with 350k commits, subject to 60k pulls in a single day, having someone with half a devops clue take a look at what your Jenkinsfile is doing with git is not exactly rocket science or a needle in a haysta…

[deleted]

Re: A one-line change decreased our build times by 99%

#64

Post by mehrdadn »

sxp wrote: post #38

> Even though we’re telling Git to do a shallow clone, to not fetch any tags, and to fetch the last 50 commits ... What is the reason for cloning 50 commits? Whenever I clone a repo off GitHub for a quick build and don't care about sending patches back, I always use --depth=1 to avoid any history or stale assets. Is there a reason to get more commits if you don't care about having a local copy of the history? Do auto…

Some tools (like linters) might need to look at the actual changes that occurred for various reasons, such as to avoid doing redundant work on unmodified files. To do that, you need all the merge bases... which can present a kind of a chicken-and-egg problem because, to figure this out with git, you need the commits to be there locally to begin with. I'm sure you can find a way around it if you put enough effort into scripting against the remote git server, but you might need to deal with git internals in the process, and it's kind of a pain compared to just cloning the whole repo.

Re: A one-line change decreased our build times by 99%

#65

Post by m45t3r »

hashkb wrote: post #45
jedberg wrote: post #33

Earlier quoted context omitted.

Why are you so angry about this? You've commented throughout this post about how this is boring and the Pinterest team is incompetent. Why? I found it quite interesting. I've been working in deployments for over 20 years at some pretty big places, and never really though about this before. I now have a new tool in my toolbox, and I'm quite happy about it.

In general I'm frustrated that rigor, standards, etc are out the window in favor of all this warm fuzziness. I guess I might be angry... The culture change in our industry, towards warm fuzzies and away from tech screens, results in calculable waste. Time, money, electricity, customers. We lose good engineers and tell ourselves they were a bad fit. We push crap on users just to sell ads. Then we write engineering pos…

> are we this forgiving of Equifax when they oopsie our data?

The kind of culture you're in favor for, blaming engineer for mistakes and punishing them, is exactly what makes the kinda of Equifax mistake possible. Suddenly, people stop to improve things and just do the minimum possible so they can keep their job, since anything else can cause a mistake that will cost your next performance cycle (or even worse, your job).

Re: A one-line change decreased our build times by 99%

#67

Post by sytringy05 »

phist_mcgee wrote: post #55
hashkb wrote: post #24

Earlier quoted context omitted.

This is Pinterest. Every org I've worked at has been smaller. There's space between shame and ignoring mistakes. The purpose of this post is not to educate. There's nothing in here that anyone can use to improve. It's just marketing.

Well I learned something from this article, and I thought I had a good handle on CI/CD, so either I am incredibly stupid and shouldn't be reading these 'nothing' articles, or maybe there is so much to learn it's impossible to know it all.

If you dont have massive repos, then this is the sort of thing that not often a big problem. Also - if you are using things like gitlab runners, you might be in the same AZ and even large repos clones are fast.

And it is impossible to know it all, I like these articles just for the differing ways people work

Re: A one-line change decreased our build times by 99%

#68

Post by aidanhs »

I sympathise a lot with this post! Git cloning can be shockingly slow.

As a personal anecdote, clones of the Rust repository in CI used to be pretty slow, and on investigating we found out that one key problem was cloning the LLVM submodule (which Rust has a fork of).

In the end we put in place a hack to download the tar.gz of our LLVM repo from github and just copy it in place of the submodule, rather than cloning it. [0]

Also, as a counterpoint to some other comments in this thread - it's really easy to just shrug off CI getting slower. A few minutes here and there adds up. It was only because our CI would hard-fail after 3 hours that the infra team really started digging in (on this and other things) - had we left it, I suspect we might be at around 5 hours by now! Contributors want to do their work, not investigate "what does a git clone really do".

p.s. our first take on this was to have the submodules cloned and stored in the CI cache, then use the rather neat `--reference` flag [1] to grab objects from this local cache when initialising the submodule - incrementally updating the CI cache was way cheaper than recloning each time. Sadly the CI provider wasn't great at handling multi-GB caches, so we went with the approach outlined above.

[0] https://github.com/rust-lang/rust/blob/1.47.0/src/ci/init_re...

[1] https://github.com/rust-lang/rust/commit/0347ff58230af512c95...

Re: A one-line change decreased our build times by 99%

#69

Post by nemothekid »

AdamJacobMuller wrote: post #61

I expected this to be some micro-optimization of moving a thing from taking 10 seconds to 100ms. > Cloning our largest repo, Pinboard went from 40 minutes to 30 seconds. This is both very impressive as well as very disheartening. If a process in my CI was taking 40 minutes I would be investigating sooner than a 40-minute delay. I don't mean to throw shade on the pintrest engineering team, but, it speaks to an institu…

I'm not sure this is complacency - this just seems like regular old tech debt. The build takes 40 minutes but everyone has other things to do and there is no time to tend to the debt. Then one day someone has some cycles and discovers a one line change fixes the underlying issue.

I'm sure many engineering projects have similar improvements that just get a ticket/issue opened and never revisited due to the mountain of other seemingly pressing issues. From IPO to the start of the year Pinterest stock price had been trending downwards - I'm sure there was more external pressure to increase profitability than to fix CI build times. The stock has completely turned around since COVID, so I'm sure that changes things

Post reply on HN