Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A one-line change decreased our build times by 99%

medium.com

Post reply on HN
Print view View original

211–220 of 256 posts

Re: A one-line change decreased our build times by 99%

#211

Post by bluedino »

projektfu wrote: post #27
JOnAgain wrote: post #2

I think it takes some real humility to post this. No doubt someone will follow up with an “of course...” or “if you don’t understand the tech you use...” comment. But thank you for this. It takes a bit of courage to point out you’ve been doing something grotesquely inefficient for years and years.

I'd be interested to know how they came to realize what was missing. Did they read the Jenkins docs more thoroughly? Post on a mailing list? See something on StackOverflow? Hire a consultant?

I see this kind of stuff at companies where only one or two developers work on a project, or the team working on it hasn't had much experience on other projects.

An example would be a company I worked for who ran a pretty standard LAMP setup but had never heard of memcached. Simply adding that reduced the database load by like 90%.

Re: A one-line change decreased our build times by 99%

#212

Post by bluedino »

AdamJacobMuller wrote: post #61

I expected this to be some micro-optimization of moving a thing from taking 10 seconds to 100ms. > Cloning our largest repo, Pinboard went from 40 minutes to 30 seconds. This is both very impressive as well as very disheartening. If a process in my CI was taking 40 minutes I would be investigating sooner than a 40-minute delay. I don't mean to throw shade on the pintrest engineering team, but, it speaks to an institu…

People probably did complain, but they were met with, "We're cloning a 20GB repo! It's not going to happen in an instant!"

Re: A one-line change decreased our build times by 99%

#213

Post by richardwhiuk »

dahfizz wrote: post #200
SamuelAdams wrote: post #189

> In the case of Pinboard, that operation would be fetching more than 2,500 branches. Ok, I'll ask: why does a single repository have over 2,500 branches? Why not delete the ones you no longer use?

Where I work doesn't delete branches, because there is no reason to. Git branches have essentially zero overhead and deleting them is just extra complexity in the CI toolchain. Deleting branches also deletes context in some scenarios. When dealing with an old codebase its nice to be able to checkout the exact version of the code at some point without having to dig through the log to get hashes and then dealing with a…

If you are doing squash merges, git branches have a cost.

Re: A one-line change decreased our build times by 99%

#214

Post by bluedino »

My similar story goes like this: We had CRM software that let you setup user defined menu options. Someone at our organization decided to make a set of nested menu options where you could configure a product, with every possible combination being assigned a value!

So if you had a large, blue second generation widget with a foo accessory and option buzz, you were value 30202, and if was the same one except red, it was 26420...

Every time the CRM software started up, it cycled through the options, generated a new XML file with all the results, this took about a minute and created like a 60MB file.

The fix was to basically version the XML file and the options definition file. If someone had already generated that file, just load the XML file instead of parsing and looping through the options file. Started up in 5 seconds!

What was the excuse that it took so long in the first place? "The CRM software is written in Java, so it's slow."

Re: A one-line change decreased our build times by 99%

#217

Post by Cthulhu_ »

holtalanm wrote: post #205
mcv wrote: post #161

On my first job, 20 years ago, we used a custom Visual C framework that generated one huge .h file that connected all sorts of stuff together. Amongst other things, that .h file contained a list of 10,000 const uints, which were included in every file, and compiled in every file. Compiling that project took hours. At some point I wrote a script that changed all those const uints to #define, which cut our build time t…

> In my current project, the slowest part of our build pipeline is the Cypress tests Oh man, I feel your pain.

Personally I think longer tests (like a full Cypress run) should not be a boundary to merging in prod if they take more than 10 minutes, but should be run nightly or continuously in the background.

I've not yet had the opportunity of having a large Cypress suite (working on it as we speak), but is it still more stable than e.g. Selenium is? Honestly 80% of issues we had with that were 'unstable' tests.

Re: A one-line change decreased our build times by 99%

#218

Post by dahfizz »

richardwhiuk wrote: post #213
dahfizz wrote: post #200

Earlier quoted context omitted.

Where I work doesn't delete branches, because there is no reason to. Git branches have essentially zero overhead and deleting them is just extra complexity in the CI toolchain. Deleting branches also deletes context in some scenarios. When dealing with an old codebase its nice to be able to checkout the exact version of the code at some point without having to dig through the log to get hashes and then dealing with a…

If you are doing squash merges, git branches have a cost.

A git branch is literally a file with a commit hash in it. It's conceptually a pointer to a commit. Creating, destroying, and maintaining a branch has all the overhead of a ~40 byte file.

Squash merges leave a ton of commits just floating in your old branch. If you delete the branch (the 40B file), all those commits are still there. Doing lots of squash merges brings you into this case I mentioned:

> If you have a weird branching strategy where you maintain multiple, significantly diverged branches at once, but only care about one of those branches at build time, then this optimization would save you time.

Re: A one-line change decreased our build times by 99%

#219

Post by dmurray »

jacquesm wrote: post #110
hashkb wrote: post #17

Earlier quoted context omitted.

Good teams profile everything. This team's only goal is to support other engineers. Build time is a huge issue for every ops team. Missing this for so long is wasted money that's easy to calculate. We can be nice to people while still having high standards. It's a missed opportunity for a deeper postmortem, and it's bland content at best.

I think you live in a highly theoretical parallel universe. The one I occupy is the one where 'good teams' profile those things that take too long. Take yourself as an example: in spite of the wide availability of free certificates you are still hosting your domain without using a secure transport layer. Some would take that as incompetence. Others would assume you have more stuff on your plate rather than that you d…

> The one I occupy is the one where 'good teams' profile those things that take too long.

Deciding which things take too long is profiling. Maybe you do it in your head or with pencil and paper instead of using a software approach but I think your position aligns with "good teams profile everything".

Re: A one-line change decreased our build times by 99%

#220

Post by holtalanm »

Cthulhu_ wrote: post #217
holtalanm wrote: post #205

Earlier quoted context omitted.

> In my current project, the slowest part of our build pipeline is the Cypress tests Oh man, I feel your pain.

Personally I think longer tests (like a full Cypress run) should not be a boundary to merging in prod if they take more than 10 minutes, but should be run nightly or continuously in the background. I've not yet had the opportunity of having a large Cypress suite (working on it as we speak), but is it still more stable than e.g. Selenium is? Honestly 80% of issues we had with that were 'unstable' tests.

Unfortunately, the issues we had with Cypress were with the framework itself, not the tests.

I used to write automation, and I can say that Selenium tests can be written to be very stable. Just depends on how they are written.

Post reply on HN