Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A one-line change decreased our build times by 99%

medium.com

Post reply on HN
Print view View original

161–170 of 256 posts

Re: A one-line change decreased our build times by 99%

#161

Post by mcv »

On my first job, 20 years ago, we used a custom Visual C framework that generated one huge .h file that connected all sorts of stuff together. Amongst other things, that .h file contained a list of 10,000 const uints, which were included in every file, and compiled in every file. Compiling that project took hours. At some point I wrote a script that changed all those const uints to #define, which cut our build time to a much more manageable half hour.

Project lead called it the biggest productivity improvement in the project; now we could build over lunch instead of over the weekend.

If there's a step in your build pipeline that takes an unreasonable amount of time, it's worth checking why. In my current project, the slowest part of our build pipeline is the Cypress tests. (They're also the most unreliable part.)

Re: A one-line change decreased our build times by 99%

#162

Post by mcv »

amelius wrote: post #137
randunel wrote: post #119

Earlier quoted context omitted.

The most frequent search keyword that I use is "-pinterest"

Yes, there seems to be no way to make it clear to Google that we want to never see certain websites in our search results. Yet, Google claims they need our information to "improve our experience".

If Google wants my information to improve my experience, I'd love to be able to vote search results up or down. Or entire sites, like pinterest and content farms.

Re: A one-line change decreased our build times by 99%

#163

Post by krageon »

inopinatus wrote: post #146
amelius wrote: post #141

Earlier quoted context omitted.

So if Monsanto/Bayer had a post about their bio informatics stack, you'd expect nobody to complain about the company and its business practices? Sometimes the negative impact of a company is just more interesting to people than what the article brings to the table.

It’s not surprising when certain firms evoke a strong personal feeling, but it’d be terribly exhausting if every article about, say, React, attracted the annotation that Facebook is the Philip Morris of media. The subsequent discussion then tends toward the divisive and derisive rather than the illuminating and informative. Hard to tell anyone they should suppress what they feel, but overall I’d tip the balance towar…

I think it's valuable to keep saying it because otherwise we start thinking it's okay to fetishise a company's products just because they're technologically interesting. If a company made them on the back of incredibly shady and unethical dealings, they shouldn't be getting free advertising here.

Re: A one-line change decreased our build times by 99%

#164

Post by robjan »

manojlds wrote: post #95
innagadadavida wrote: post #74

Earlier quoted context omitted.

Considering the build host does this hundreds of times every day, a better solution would be to simply have a git repo cache locally, should be secure and reliable given git’s object store design? Any simple wrappers for git that can do this transparently?

Build servers don't git clone everytime though. They do a git clean if needed followed a git fetch / git pull equivalent. GoCD for example maintains a single copy of the repo on the server for every pipeline that refers to it and the agents have the repos that they work on checked out. Any local changes or untracked files are by default cleaned. There are settings to force reclone etc, but it's not the default.

In many cases the build agent is a stateless container which is destroyed as soon as the build is finished. In cases like this the repo needs to be (shallow) cloned each time.

Re: A one-line change decreased our build times by 99%

#165

Post by andromeduck »

amelius wrote: post #137
randunel wrote: post #119

Earlier quoted context omitted.

The most frequent search keyword that I use is "-pinterest"

Yes, there seems to be no way to make it clear to Google that we want to never see certain websites in our search results. Yet, Google claims they need our information to "improve our experience".

-pintrest should be a search extension.

Re: A one-line change decreased our build times by 99%

#166

Post by Dylan16807 »

mehrdadn wrote: post #64
sxp wrote: post #38

> Even though we’re telling Git to do a shallow clone, to not fetch any tags, and to fetch the last 50 commits ... What is the reason for cloning 50 commits? Whenever I clone a repo off GitHub for a quick build and don't care about sending patches back, I always use --depth=1 to avoid any history or stale assets. Is there a reason to get more commits if you don't care about having a local copy of the history? Do auto…

Some tools (like linters) might need to look at the actual changes that occurred for various reasons, such as to avoid doing redundant work on unmodified files. To do that, you need all the merge bases... which can present a kind of a chicken-and-egg problem because, to figure this out with git, you need the commits to be there locally to begin with. I'm sure you can find a way around it if you put enough effort into…

If you're interested in metadata, you can use --filter=blob:none to get the commit history but without any file contents.

Re: A one-line change decreased our build times by 99%

#167

Post by ravishi »

mcv wrote: post #161

On my first job, 20 years ago, we used a custom Visual C framework that generated one huge .h file that connected all sorts of stuff together. Amongst other things, that .h file contained a list of 10,000 const uints, which were included in every file, and compiled in every file. Compiling that project took hours. At some point I wrote a script that changed all those const uints to #define, which cut our build time t…

At my second job in the industry I worked on a Python project that had to be deployed in a kind of sandboxed production environment where we had no internet access.

Deploys were painful, as any missing dependency had to be searched in our notebooks over 3G, then copied to an external storage, then plugged into a Windows machine, uploaded to the production server through SCP and then deployed manually over SSH. Sometimes we spent hours doing this again and again until all dependencies were finally resolved.

I worked there for almost a year, did many cool gigs and learned a lot. But my most valuable contribution came when at some point, tired of that unpredictable torture that were the deploys, started researching into solutions. I set up a pypi proxy into one of our spare office machines and routed all my daily package installs through that. Then I copied that entire proxy content into the production machine before every deploy, and voila, no more surprises.

I left this job a few weeks later, but have heard that this solution was very useful for many devs that joined the team afterwards.

Re: A one-line change decreased our build times by 99%

#168

Post by [deleted] »

saagarjha wrote: post #130
dusted wrote: post #117

This is the most I've ever gotten out of pinterest, other than this, it's just the "wrong site that google turns up, that I can't use because it wants me to create an account just to watch the image I searched for"

Can we not do the thing where we pick an organization from an article and then bring up the most generic complaint you can about it in a way that is entirely irrelevant to the post? We get it, you don't like Pinterest showing up in search results, nobody does. But this has absolutely nothing to do with the article other than it being pattern matching on the word "Pinterest", which is about the least informative comme…

[deleted]

Re: A one-line change decreased our build times by 99%

#169

Post by mehrdadn »

Dylan16807 wrote: post #166
mehrdadn wrote: post #64

Earlier quoted context omitted.

Some tools (like linters) might need to look at the actual changes that occurred for various reasons, such as to avoid doing redundant work on unmodified files. To do that, you need all the merge bases... which can present a kind of a chicken-and-egg problem because, to figure this out with git, you need the commits to be there locally to begin with. I'm sure you can find a way around it if you put enough effort into…

If you're interested in metadata, you can use --filter=blob:none to get the commit history but without any file contents.

Did not know, that's great, thanks! Seems this is a relatively recent feature?

Re: A one-line change decreased our build times by 99%

#170

Post by 1123581321 »

krageon wrote: post #163
inopinatus wrote: post #146

Earlier quoted context omitted.

It’s not surprising when certain firms evoke a strong personal feeling, but it’d be terribly exhausting if every article about, say, React, attracted the annotation that Facebook is the Philip Morris of media. The subsequent discussion then tends toward the divisive and derisive rather than the illuminating and informative. Hard to tell anyone they should suppress what they feel, but overall I’d tip the balance towar…

I think it's valuable to keep saying it because otherwise we start thinking it's okay to fetishise a company's products just because they're technologically interesting. If a company made them on the back of incredibly shady and unethical dealings, they shouldn't be getting free advertising here.

Who here is fetishizing products because they learned something from the engineering blog? This is not happening.
Post reply on HN