Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A one-line change decreased our build times by 99%

medium.com

Post reply on HN
Print view View original

41–50 of 256 posts

Re: A one-line change decreased our build times by 99%

#41

Post by kevinventullo »

hashkb wrote: post #5
JOnAgain wrote: post #2

I think it takes some real humility to post this. No doubt someone will follow up with an “of course...” or “if you don’t understand the tech you use...” comment. But thank you for this. It takes a bit of courage to point out you’ve been doing something grotesquely inefficient for years and years.

They are a publicly traded company. They have a team dedicated to engineering support. A better article would include a management and hiring postmortem. It's shocking, really. Humility is nice, but competency is also nice.

A team discovers a major efficiency win requiring minimal engineering effort and your response is to... punish them?

Re: A one-line change decreased our build times by 99%

#42

Post by Hnrobert42 »

hashkb wrote: post #5
JOnAgain wrote: post #2

I think it takes some real humility to post this. No doubt someone will follow up with an “of course...” or “if you don’t understand the tech you use...” comment. But thank you for this. It takes a bit of courage to point out you’ve been doing something grotesquely inefficient for years and years.

They are a publicly traded company. They have a team dedicated to engineering support. A better article would include a management and hiring postmortem. It's shocking, really. Humility is nice, but competency is also nice.

Shocking? Incompetent? A hiring postmortem? Really? AYFKM?

Re: A one-line change decreased our build times by 99%

#43

Post by ma2rten »

segfaultbuserr wrote: post #30

Better title: A one-line change decreased our "git clone" times by 99%. It's a bit misleading to use "build time" to describe this improvement, as it makes people think about build systems, compilers, header files, or cache. On the other hand, the alternative title is descriptive and helpful to all developers, not only just builders - people who simply need to clone a branch from a large repository can benefit from t…

Right, from the article:

"This simple one line change reduced our clone times by 99% and significantly reduced our build times as a result."

So the title is just completely wrong.

Re: A one-line change decreased our build times by 99%

#44

Post by [deleted] »

hashkb wrote: post #17
Aeolun wrote: post #13

Earlier quoted context omitted.

I don’t know, I consider myself fairly competent but I’d never even considered that. It’s just not so relevant until your repo is multiple gigabytes big. Still, I’ll see if it works for our pipelines, and we can get our clone from 20s to 1s

Good teams profile everything. This team's only goal is to support other engineers. Build time is a huge issue for every ops team. Missing this for so long is wasted money that's easy to calculate. We can be nice to people while still having high standards. It's a missed opportunity for a deeper postmortem, and it's bland content at best.

[deleted]

Re: A one-line change decreased our build times by 99%

#45

Post by hashkb »

jedberg wrote: post #33
hashkb wrote: post #5

Earlier quoted context omitted.

They are a publicly traded company. They have a team dedicated to engineering support. A better article would include a management and hiring postmortem. It's shocking, really. Humility is nice, but competency is also nice.

Why are you so angry about this? You've commented throughout this post about how this is boring and the Pinterest team is incompetent. Why? I found it quite interesting. I've been working in deployments for over 20 years at some pretty big places, and never really though about this before. I now have a new tool in my toolbox, and I'm quite happy about it.

In general I'm frustrated that rigor, standards, etc are out the window in favor of all this warm fuzziness. I guess I might be angry...

The culture change in our industry, towards warm fuzzies and away from tech screens, results in calculable waste. Time, money, electricity, customers. We lose good engineers and tell ourselves they were a bad fit. We push crap on users just to sell ads. Then we write engineering posts to brag about fixing our own mistakes. It's a terrible shame and I speak up about it to remind everyone there was a time when RTFM would be the only response to this.

Edit: rate limited but one last thing: are we this forgiving of Equifax when they oopsie our data? Seeing this would immediately make me wonder if anything I have shared with Pinterest is safe. That's why they owe us a postmortem and not a thirst trap.

Re: A one-line change decreased our build times by 99%

#46

Post by lytedev »

CGamesPlay wrote: post #32
segfaultbuserr wrote: post #30

Better title: A one-line change decreased our "git clone" times by 99%. It's a bit misleading to use "build time" to describe this improvement, as it makes people think about build systems, compilers, header files, or cache. On the other hand, the alternative title is descriptive and helpful to all developers, not only just builders - people who simply need to clone a branch from a large repository can benefit from t…

They set out to reduce build times, not to reduce git checkout times. It turns out that 99% of the entire build was spent downloading code.

This isn't true either, as the article says that builds went from 40 minutes to 30 minutes. The time spent cloning was presumably about 10 minutes and came down very far, presumably by 99%.

Re: A one-line change decreased our build times by 99%

#47

Post by mikepurvis »

chinhodado wrote: post #26

When I first joined one of my previous jobs, the build process had a checkout stage where it was blowing away the git folder and checked out from scratch the whole repo every time (!). Since the build machine was reserved for that build job I simply made some changes to do git clean -dfx & git reset --hard & git checkout origin branch. It shaved off like 15 minutes of the build time, which was something like 50% of t…

It's frustrating how many ways there are for a git clone to get out of sync, especially when it's an automation-managed one that is supposed to be long-lived (think stuff like gracefully handling force-pushed branches and tags that are deleted). I've dealt with a bit of this with my company's Hound (code search engine) instance. Currently there's a big snarl of fallback logic in there that tries a shallow clone, but then unshallows and pulls refs if it can't find what it's looking for, culminating in this ridiculousness:

    git fetch --prune --no-tags --depth 1 origin+{ref}:remotes/origin/{ref}
See the whole thing here: https://github.com/mikepurvis/hound/blob/6b0b44db489f9aeff39...

The pipeline I manage is many repos rather than a monorepo, and maintaining long-licheckouts in this context is not really realistic, but what does work and is very fast is just grabbing tarballs— GitLab and Github both cache them, so they don't don't cost additional compute after the first time, and downloading them is strictly less transfer and fewer round trips than the git protocol.

The only real cost is that anything at build time which needs VCS info (eg, to embed it in the binary) will need an alternate path, for example having it be able to be passed in via an envvar.

Re: A one-line change decreased our build times by 99%

#48

Post by mixmastamyk »

chinhodado wrote: post #26

When I first joined one of my previous jobs, the build process had a checkout stage where it was blowing away the git folder and checked out from scratch the whole repo every time (!). Since the build machine was reserved for that build job I simply made some changes to do git clean -dfx & git reset --hard & git checkout origin branch. It shaved off like 15 minutes of the build time, which was something like 50% of t…

A new checkout is good practice. Using refspec and depth options can make it quick.

Re: A one-line change decreased our build times by 99%

#49

Post by pavel_lishin »

hashkb wrote: post #45
jedberg wrote: post #33

Earlier quoted context omitted.

Why are you so angry about this? You've commented throughout this post about how this is boring and the Pinterest team is incompetent. Why? I found it quite interesting. I've been working in deployments for over 20 years at some pretty big places, and never really though about this before. I now have a new tool in my toolbox, and I'm quite happy about it.

In general I'm frustrated that rigor, standards, etc are out the window in favor of all this warm fuzziness. I guess I might be angry... The culture change in our industry, towards warm fuzzies and away from tech screens, results in calculable waste. Time, money, electricity, customers. We lose good engineers and tell ourselves they were a bad fit. We push crap on users just to sell ads. Then we write engineering pos…

> The culture change in our industry, towards warm fuzzies and away from tech screens, results in calculable waste.

I have not noticed, in the past decade, any move away from tech screens whatsoever.

Re: A one-line change decreased our build times by 99%

#50

Post by phildenhoff »

lytedev wrote: post #46
CGamesPlay wrote: post #32

Earlier quoted context omitted.

They set out to reduce build times, not to reduce git checkout times. It turns out that 99% of the entire build was spent downloading code.

This isn't true either, as the article says that builds went from 40 minutes to 30 minutes. The time spent cloning was presumably about 10 minutes and came down very far, presumably by 99%.

It says from 40 minutes to 30 seconds, not minutes.
Post reply on HN