Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A one-line change decreased our build times by 99%

medium.com

Post reply on HN
Print view View original

171–180 of 256 posts

Re: A one-line change decreased our build times by 99%

#171

Post by mcintyre1994 »

nathan_f77 wrote: post #125

Is there any way to do this for GitLab CI [1]? I'm using GIT_DEPTH=1, but I'm not sure how to set refspecs. It's not too important right now since it only takes about 11 seconds to clone the git repo, but maybe it's a quick win as well. [1] https://docs.gitlab.com/ee/ci/large_repositories/

The docs seem to give the impression that they already do this, but it'd be great if someone from Gitlab could confirm because it doesn't use the refspec term or show the resulting git command.

> The following example makes the runner shallow clone to fetch only a given branch; it does not fetch any other branches nor tags.

https://docs.gitlab.com/ee/ci/large_repositories/#shallow-cl...

Re: A one-line change decreased our build times by 99%

#172

Post by some_furry »

dusted wrote: post #117

This is the most I've ever gotten out of pinterest, other than this, it's just the "wrong site that google turns up, that I can't use because it wants me to create an account just to watch the image I searched for"

This fact has forced people to write browser extensions to filter Pinterest out.

I opt for the "teach non-tech people how to dork" route instead: https://soatok.blog/2020/07/21/dorking-your-way-to-search-re...

Re: A one-line change decreased our build times by 99%

#173

Post by renke1 »

mcv wrote: post #161

On my first job, 20 years ago, we used a custom Visual C framework that generated one huge .h file that connected all sorts of stuff together. Amongst other things, that .h file contained a list of 10,000 const uints, which were included in every file, and compiled in every file. Compiling that project took hours. At some point I wrote a script that changed all those const uints to #define, which cut our build time t…

> If there's a step in your build pipeline that takes an unreasonable amount of time, it's worth checking why. In my current project, the slowest part of our build pipeline is the Cypress tests. (They're also the most unreliable part.)

Would you say the (slow and unreliable) Cypress tests are worth it still?

Re: A one-line change decreased our build times by 99%

#174

Post by alkonaut »

saagarjha wrote: post #148
alkonaut wrote: post #144

Earlier quoted context omitted.

Sorry no. If an article is paywalled, on Pinterest, or similar, then please let's discuss the source instead, even if it ruins the discussion, so people learn not to post such links.

Paywall complaints are explicitly off-topic: https://news.ycombinator.com/item?id=10178989 . I am not a moderator, but I think I've made it clear that I personally consider comments like the one I responded to be as well. FWIW, in the all years I have been on this site, I have seen this happen regularly and I have yet to see any reduction in such links or these kinds of discussions. Seeing as you've been here longer,…

I just don’t mind repeating myself whether it changes anything or not I guess. Simply because discussing paywalled links or Pinterest linked is invariably more interesting than whatever is found (or not found) when following those links.

Re: A one-line change decreased our build times by 99%

#176

Post by dzhiurgis »

renke1 wrote: post #173
mcv wrote: post #161

On my first job, 20 years ago, we used a custom Visual C framework that generated one huge .h file that connected all sorts of stuff together. Amongst other things, that .h file contained a list of 10,000 const uints, which were included in every file, and compiled in every file. Compiling that project took hours. At some point I wrote a script that changed all those const uints to #define, which cut our build time t…

> If there's a step in your build pipeline that takes an unreasonable amount of time, it's worth checking why. In my current project, the slowest part of our build pipeline is the Cypress tests. (They're also the most unreliable part.) Would you say the (slow and unreliable) Cypress tests are worth it still?

That's the nature of UI tests for most part. IIRC Cypress are written declarative tools which would make them even more unreliable and slow, albeit easier to fix.

Personally I've recently started using Playwright and I'm quite happy with it. There was occasional misunderstanding of their API, but 95% of time it's great. Microsoft is kicking butt these days.

Re: A one-line change decreased our build times by 99%

#177

Post by rightbyte »

bald42 wrote: post #154
JOnAgain wrote: post #2

I think it takes some real humility to post this. No doubt someone will follow up with an “of course...” or “if you don’t understand the tech you use...” comment. But thank you for this. It takes a bit of courage to point out you’ve been doing something grotesquely inefficient for years and years.

I don't get why they have to clone their repo frequently in the first place - seems to me as a brute force usage of a version control system prone to high cost in the first place.

It is a nice and fool proof way to get a clean working environment to just download everything from nothing. And you want different working folder for different jobs anyway so they don't mess with eachother or build of state between jobs due to scripting messups.

Re: A one-line change decreased our build times by 99%

#178

Post by mcv »

renke1 wrote: post #173
mcv wrote: post #161

On my first job, 20 years ago, we used a custom Visual C framework that generated one huge .h file that connected all sorts of stuff together. Amongst other things, that .h file contained a list of 10,000 const uints, which were included in every file, and compiled in every file. Compiling that project took hours. At some point I wrote a script that changed all those const uints to #define, which cut our build time t…

> If there's a step in your build pipeline that takes an unreasonable amount of time, it's worth checking why. In my current project, the slowest part of our build pipeline is the Cypress tests. (They're also the most unreliable part.) Would you say the (slow and unreliable) Cypress tests are worth it still?

I don't know. We need some sort of e2e tests, and all e2e test frameworks are terrible in one way or another. Cypress is okay. I would prefer to only run it on production or the dev server and have alarms go off when they fail, but either the requirement is, or other developers have decided that it's necessary to pass all e2e tests before a feature branch can be merged into the master branch.

And I get the reason for it; you don't want to accidentally merge breaking changes. But it does make our build pipelines very slow and unreliable.

So are they worth it? I don't know. If I had my way, we'd only run them on master, and not make it a requirement for feature branches to pass them. Because if you fix one tiny thing, you now have to wait 15 minutes again for the Cypress tests to run. I think they'd be better in a different setup than what we're doing.

Re: A one-line change decreased our build times by 99%

#179

Post by dctoedt »

nabaraz wrote: post #138
saagarjha wrote: post #130

Earlier quoted context omitted.

Can we not do the thing where we pick an organization from an article and then bring up the most generic complaint you can about it in a way that is entirely irrelevant to the post? We get it, you don't like Pinterest showing up in search results, nobody does. But this has absolutely nothing to do with the article other than it being pattern matching on the word "Pinterest", which is about the least informative comme…

HN has always been very predictive. Praise Microsoft for turning the corner, Dislike Google for ads and snooping, Praise Apple for privacy, Dislike Zoom for privacy, Dislike Pinterest for middlewaring Google Image, and so on.

Friendly amendment: *predictable

Re: A one-line change decreased our build times by 99%

#180

Post by DougBTX »

ForHackernews wrote: post #157
bald42 wrote: post #154

Earlier quoted context omitted.

I don't get why they have to clone their repo frequently in the first place - seems to me as a brute force usage of a version control system prone to high cost in the first place.

I don't know about a big org like Pinterest, but it's pretty common for "clone the repo" to be the first step of a CI/CD pipeline when using something like CircleCI or GitlabCI. It's an easy (if inefficient) way to always get the latest changes and if you have disposable build-runners then it all gets thrown away at the end of the pipeline.

It is interesting that we trust our tools so little. A git hash is a pretty robust way to know whether the code in the repo is what it is supposed to be, so a "git fetch" rather than a fresh "git clone" should be safe, but we can't trust the build steps to not trash the build-runner so the entire thing needs to be thrown away.

Edit: for context, I wrote this comment while waiting for `npm ci` to run. Its first step is to delete the node_modules folder, as otherwise it can't be trusted to update correctly.

Post reply on HN