Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A one-line change decreased our build times by 99%

medium.com

Post reply on HN
Print view View original

101–110 of 256 posts

Re: A one-line change decreased our build times by 99%

#101

Post by phist_mcgee »

sytringy05 wrote: post #67
phist_mcgee wrote: post #55

Earlier quoted context omitted.

Well I learned something from this article, and I thought I had a good handle on CI/CD, so either I am incredibly stupid and shouldn't be reading these 'nothing' articles, or maybe there is so much to learn it's impossible to know it all.

If you dont have massive repos, then this is the sort of thing that not often a big problem. Also - if you are using things like gitlab runners, you might be in the same AZ and even large repos clones are fast. And it is impossible to know it all, I like these articles just for the differing ways people work

It sounds like GP has a bit of a chip on their shoulder when it comes to feeling like you're not worthy if you don't know everything.

Re: A one-line change decreased our build times by 99%

#102

Post by unishark »

phist_mcgee wrote: post #51
hashkb wrote: post #45

Earlier quoted context omitted.

In general I'm frustrated that rigor, standards, etc are out the window in favor of all this warm fuzziness. I guess I might be angry... The culture change in our industry, towards warm fuzzies and away from tech screens, results in calculable waste. Time, money, electricity, customers. We lose good engineers and tell ourselves they were a bad fit. We push crap on users just to sell ads. Then we write engineering pos…

1. You seem to be projecting a lot of your own thoughts and biases onto this article, and this discussion. 2. RTFM made sense when you wrote code in one terminal window and compiled it in another and then shipped a CD. There is no way you can RTFM for every tool you use at a modern software company. NO ifs ands or buts, it just is not happening. It sounds like you are looking at a nostalgic view of the past, and not…

I personally do think there is a systemic problem in companies trying to hire the bare minimum in skills/experience for technical roles and ending up with people operating right at the limit of their abilities (and intermittently being in over their heads).

But do I agree RTFM is frustrating advice for a lot of things. Especially if you aren't going to use such information often enough, so will keep forgetting it anyway and have to start over with the manual each time.

Re: A one-line change decreased our build times by 99%

#104

Post by ibains »

Thus just shows how poor visibility into git is, I hope it gets better.

Building a product with poor visibility and ridiculing users for not knowing internals is the worst practice in Computer Science.

Hadoop did the same, and has set a record of fastest software to become legacy.

Super nice to see great comments here and the nice article.

Re: A one-line change decreased our build times by 99%

#106

Post by raverbashing »

AdamJacobMuller wrote: post #61

I expected this to be some micro-optimization of moving a thing from taking 10 seconds to 100ms. > Cloning our largest repo, Pinboard went from 40 minutes to 30 seconds. This is both very impressive as well as very disheartening. If a process in my CI was taking 40 minutes I would be investigating sooner than a 40-minute delay. I don't mean to throw shade on the pintrest engineering team, but, it speaks to an institu…

This is the real complacency

Did someone really think "well it takes 40min, what can you do about it?" and just left it as such?

I knew people who would have that mentality in companies that are not around anymore. Take it as you want.

Yes, git is hard, but you know, maybe someone else has a better idea, or you can check SO, etc. (I don't even know why they were adding the refspecs there)

Re: A one-line change decreased our build times by 99%

#107

Post by masklinn »

pushrax wrote: post #83
marta_morena_28 wrote: post #76

Earlier quoted context omitted.

It really doesn't sound complicated to find, unless you just have a handsoff approach to building things and just don't care as long as "something" comes out on the other end. What makes me wonder however is this: 40 min made them look into this? I mean 40 min is crazy long. What builds this long? Chrome, Windows, Linux Kernel on a single core? This should have been raising red flags much earlier. The only explanatio…

It takes >12h to build Windows on the MS build platform. On a single core, Chromium surely takes hours to build. Though I agree that 40min for the repository in question is highly suspect.

> On a single core, Chromium surely takes hours to build.

Earlier this year Bruce Dawson had a post indicating that it took about a CPU-day, though coalescing files (“jumbo builds”) significantly reduces build time (we’re talking down to 5h), however that’s at the expense of incremental building, and it constrains the code as you can get symbol collisions between the coalesced files.

Re: A one-line change decreased our build times by 99%

#108

Post by fn1 »

AdamJacobMuller wrote: post #61

I expected this to be some micro-optimization of moving a thing from taking 10 seconds to 100ms. > Cloning our largest repo, Pinboard went from 40 minutes to 30 seconds. This is both very impressive as well as very disheartening. If a process in my CI was taking 40 minutes I would be investigating sooner than a 40-minute delay. I don't mean to throw shade on the pintrest engineering team, but, it speaks to an institu…

> I wonder what 'institutional complacencies' we have. Problems we assume are unsolvable but are actually very trivial to solve.

I spend a lot of time optimizing builds, because the effect is a multiplicator for everything else in development.

But it is not an easy task. One issue with performance-monitoring is that you have to carefully plan your work, or you will sit around and wait for results a lot:

Try the build: 40 minutes. Maybe add profiling statements, because you forgot them: another 40 minutes. Change something and try it out: no change, 40 minutes. Find another optimization which decreases time locally and try it out: 39.5 minutes, because on the build-server that optimization does not work that well. etc.

You just spent 160 minutes and shaved 0.5 minutes off the build.

I'm not saying it's not worth it, but that line of work is not often rewarding.

On the flip-side I once took two hours to write a java-agent which caches File.exists for class-loading and managed to decrease local startup time by 500% because the corporate virus-scanner got active less often.

Re: A one-line change decreased our build times by 99%

#109

Post by dagmx »

manojlds wrote: post #95
innagadadavida wrote: post #74

Earlier quoted context omitted.

Considering the build host does this hundreds of times every day, a better solution would be to simply have a git repo cache locally, should be secure and reliable given git’s object store design? Any simple wrappers for git that can do this transparently?

Build servers don't git clone everytime though. They do a git clean if needed followed a git fetch / git pull equivalent. GoCD for example maintains a single copy of the repo on the server for every pipeline that refers to it and the agents have the repos that they work on checked out. Any local changes or untracked files are by default cleaned. There are settings to force reclone etc, but it's not the default.

That depends very heavily on the build infrastructure being used however

Re: A one-line change decreased our build times by 99%

#110

Post by jacquesm »

hashkb wrote: post #17
Aeolun wrote: post #13

Earlier quoted context omitted.

I don’t know, I consider myself fairly competent but I’d never even considered that. It’s just not so relevant until your repo is multiple gigabytes big. Still, I’ll see if it works for our pipelines, and we can get our clone from 20s to 1s

Good teams profile everything. This team's only goal is to support other engineers. Build time is a huge issue for every ops team. Missing this for so long is wasted money that's easy to calculate. We can be nice to people while still having high standards. It's a missed opportunity for a deeper postmortem, and it's bland content at best.

I think you live in a highly theoretical parallel universe. The one I occupy is the one where 'good teams' profile those things that take too long.

Take yourself as an example: in spite of the wide availability of free certificates you are still hosting your domain without using a secure transport layer. Some would take that as incompetence. Others would assume you have more stuff on your plate rather than that you don't have high standards.

Post reply on HN