Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A one-line change decreased our build times by 99%

medium.com

Post reply on HN
Print view View original

181–190 of 256 posts

Re: A one-line change decreased our build times by 99%

#181

Post by fenomas »

alkonaut wrote: post #144
saagarjha wrote: post #130

Earlier quoted context omitted.

Can we not do the thing where we pick an organization from an article and then bring up the most generic complaint you can about it in a way that is entirely irrelevant to the post? We get it, you don't like Pinterest showing up in search results, nobody does. But this has absolutely nothing to do with the article other than it being pattern matching on the word "Pinterest", which is about the least informative comme…

Sorry no. If an article is paywalled, on Pinterest, or similar, then please let's discuss the source instead, even if it ruins the discussion, so people learn not to post such links.

TFA isn't paywalled, or on Pinterest, or similar.

Re: A one-line change decreased our build times by 99%

#182

Post by user5994461 »

bald42 wrote: post #154
JOnAgain wrote: post #2

I think it takes some real humility to post this. No doubt someone will follow up with an “of course...” or “if you don’t understand the tech you use...” comment. But thank you for this. It takes a bit of courage to point out you’ve been doing something grotesquely inefficient for years and years.

I don't get why they have to clone their repo frequently in the first place - seems to me as a brute force usage of a version control system prone to high cost in the first place.

Jenkins has a setting to keep the checkout directory (default) or to clear the directory between builds.

At last job, the default was letting broken changes pass the build, they break some step of the setup/run process that's not run on a partial build. New joiners came in and they couldn't build because the build was broken.

Had to fix it by setting up two jobs, one running from scratch (30 minutes) and one incremental (10 minutes). The build from scratch was catching a broken change or two every week.

Re: A one-line change decreased our build times by 99%

#183

Post by ForHackernews »

DougBTX wrote: post #180
ForHackernews wrote: post #157

Earlier quoted context omitted.

I don't know about a big org like Pinterest, but it's pretty common for "clone the repo" to be the first step of a CI/CD pipeline when using something like CircleCI or GitlabCI. It's an easy (if inefficient) way to always get the latest changes and if you have disposable build-runners then it all gets thrown away at the end of the pipeline.

It is interesting that we trust our tools so little. A git hash is a pretty robust way to know whether the code in the repo is what it is supposed to be, so a "git fetch" rather than a fresh "git clone" should be safe, but we can't trust the build steps to not trash the build-runner so the entire thing needs to be thrown away. Edit: for context, I wrote this comment while waiting for `npm ci` to run. Its first step i…

> we can't trust the build steps to not trash the build-runner so the entire thing needs to be thrown away.

I think it's partly this, and partly that everything is shared infrastructure now. I don't want to pay to keep a machine up 24/7 just to use it to run a build for 10 minutes half a dozen times per day.

So instead I lease time on shared hardware with ephemeral "containers" or "virtual machines" or whatever.

Re: A one-line change decreased our build times by 99%

#184

Post by mcherm »

Dylan16807 wrote: post #160
Thorrez wrote: post #89

Earlier quoted context omitted.

Where does the article say "99% of the entire build was spent downloading code"?

The title. If they reduced the build time by that much, then at least that much of the build time must have been spent downloading code. If the title is a lie (which it probably is), then nevermind that number, but it's clear where it came from.

The text of the article clearly states that clone time was reduced by 99%.

The only way build time could have been reduced by 99% is if every part of the build other than cloning is negligible. It is far more plausible to assume that the title is simply wrong.

Re: A one-line change decreased our build times by 99%

#185

Post by user5994461 »

andrelaszlo wrote: post #118

We recently reduced our build times by 5-10% or so by changing the default bcrypt iteration count (for tests). It also felt silly once we found it.

You should consider doing the same thing in production.

It's a trope at this point how the modern slow hashing algorithms are utterly misconfigured. Stopped counting how many times I've seen it.

Take a whole second to compute a hash on the production machine because "hashing is supposed to be slow", noting the production server is a low frequency Xeon that has many core but they're half as slow as your development with a 4GHz i7-9999.

Hashing is supposed to take milliseconds, not seconds. If it's taking longer than 100 ms you need to make it faster.

edit: found the problem, this bad stackoverflow answer that's been spreading bad recommendations for years https://security.stackexchange.com/questions/17207/recommend...

Re: A one-line change decreased our build times by 99%

#186

Post by dada78641 »

This reminds me of my first programming job in 2005, working with Macromedia Flash. They had one other Flash programmer who only worked there every once in a while because he was actually studying in college, and he was working on some kind of project from hell that, among other problems, took about two minutes to build to SWF.

Eventually they stopped asking him to come because he couldn't get anything done, and so I had a look at it. In the Movie Clip library of the project I found he had an empty text field somewhere that was configured to include a copy of almost the entire Unicode range, including thousands of CJK characters, so each time you built the SWF it would collect and compress numerous different scripts from different fonts as vectors for use by the program. And it wasn't even being used by anything.

Once I removed that one empty text field, builds went down to about ~3 seconds.

Re: A one-line change decreased our build times by 99%

#187

Post by NikhilVerma »

AdamJacobMuller wrote: post #61

I expected this to be some micro-optimization of moving a thing from taking 10 seconds to 100ms. > Cloning our largest repo, Pinboard went from 40 minutes to 30 seconds. This is both very impressive as well as very disheartening. If a process in my CI was taking 40 minutes I would be investigating sooner than a 40-minute delay. I don't mean to throw shade on the pintrest engineering team, but, it speaks to an institu…

I doubt that they started off with a 40 mins delay. It probably crept slowly as the repo got bigger and no one noticed it because of the gentle gradient. And they didn't have the time/resources to look into it.

Re: A one-line change decreased our build times by 99%

#188

Post by greesil »

ravishi wrote: post #167
mcv wrote: post #161

On my first job, 20 years ago, we used a custom Visual C framework that generated one huge .h file that connected all sorts of stuff together. Amongst other things, that .h file contained a list of 10,000 const uints, which were included in every file, and compiled in every file. Compiling that project took hours. At some point I wrote a script that changed all those const uints to #define, which cut our build time t…

At my second job in the industry I worked on a Python project that had to be deployed in a kind of sandboxed production environment where we had no internet access. Deploys were painful, as any missing dependency had to be searched in our notebooks over 3G, then copied to an external storage, then plugged into a Windows machine, uploaded to the production server through SCP and then deployed manually over SSH. Someti…

I suppose no Docker containers were allowed in prod either?

Re: A one-line change decreased our build times by 99%

#190

Post by SamuelAdams »

temporallobe wrote: post #29
gpapilion wrote: post #9

I’ve found as an industry we’ve moved to more complex tools, but haven’t built the expertise in them to truly engineer solutions using them. I think lots of organizations could find major optimizations, but it requires really learning about the technology you’re utilizing.

This, this, so much this. When we build more complexity into a system, the less we understand it, similar to how development frameworks create multiple layers of abstraction to the point where the developers have no idea what actual code the framework produces, much less how to fix it.

Another way of looking at it is this is the current golden age of infosec.

Think of all these complex systems developers and SysAdmins need to maintain at a company. Then think of how well each person knows each technology. Most of them will be "T" shaped, ie know one tech well but surface-level on all the others.

If I know several tools really well (or better than the company's sysadmins / devs) I can probably find some security issues with them.

Post reply on HN