Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bring your monorepo down to size with sparse-checkout

github.blog

Post reply on HN
Print view View original

61–68 of 68 posts

Re: Bring your monorepo down to size with sparse-checkout

#61

Post by peterwwillis »

gen220 wrote: post #54
peterwwillis wrote: post #53

Earlier quoted context omitted.

If you can do that with 3rd party dependencies, can't you do that with all the code? This is what confuses me about monorepos. Their design requires an array of confusing processes and complex software to make the process of merging, testing, and releasing code manageable at scale (and "scale" can even be 6 developers working on 2 separate features each across 10 services, in one repo). But it turns out that you can…

These are great questions!! :) > Can you treat all code like 3rd party dependencies? Yes, but there are trade-offs. Discoverability, enforcing hard deadlines on global changes, style consistency, etc. > Is it impossible to do these things with multi-repo? No, but there are trade-offs to consider. > If it's hard, is it "so hard" that it justifies the complexity? Hitting the nail on the head; there are trade-offs :) >…

I don't think we're talking past each other, and thank you for your responses.

> Does your team want to treat other teams as a 3rd party dependency?

From what I recall, 'true' microservices are supposed to operate totally independent from each other, so one team's microservice really is a 3rd party dependency of another team's (if one depends on the other). OTOH, monolithic services would require much tighter integration between teams. But there's also architecture like SOA that sort of sits in the middle.

To my mind, if the repo structure mimics the communication and workflow of the people writing the code, it feels like the tradeoffs might fit better. But I'd need to make a matrix of all the things (repos, architectures, SDLCs, tradeoffs, etc) and see some white papers to actually know. If someone feels like writing that book, I'd read it!

Re: Bring your monorepo down to size with sparse-checkout

#62

Post by gdxhyrd »

krupan wrote: post #32
gdxhyrd wrote: post #22

Earlier quoted context omitted.

Not really, because commits don't go across the entire SVN, which is what makes monorepos so powerful.

What do you mean? When you commit to svn the whole repository goes up in version number.

You are right, I was thinking of CVS.

In any case, with SVN you usually do not want to give write perms to everyone in all the tree, so you end up with effectively partitioned spaces, or you make several repos instead, or you put another layer on top. With Git, anyone can easily develop global commits.

Re: Bring your monorepo down to size with sparse-checkout

#63

Post by [deleted] »

krupan wrote: post #31

How does a sparse checkout not defeat the purpose of a monorepo? I thought monorepos existed so it was easy to make changes that affect the whole codebase and to test those changes. If you only checkout a portion of the files, how are you going to test against the whole repo? EDIT: my overall concern is that it looks like people are reinventing clearcase. Please speak to an older developer who worked at an HP/IBM typ…

[deleted]

Re: Bring your monorepo down to size with sparse-checkout

#64

Post by MayeulC »

scarejunba wrote: post #30
monocasa wrote: post #13

Earlier quoted context omitted.

Sometimes you don't own the other repo.

Doesn't that seem like a build tool situation? At that point the other piece of code isn't part of source, it's a source dependency, and no different from a binary dependency at some version so you don't really want the tree, you want the file at some revision and if it's `github` based then you have the natural HTTP endpoint and otherwise it's trivial to proxy as an artifact.

Well, you are right, but there would still be some advantages to submodules:

1. Check the files hash themselves: while you can definitely put the commit ID in the URL, nothing prevents the remote server (though unlikely if github) to answer with another version of the file (and could even do so selectively for your build server).

2. Simple upgrade path: with submodules, you can just `cd` into them and run `git pull` or `git checkout v11.5.2`, and git itself could inform you that a newer version is available if tracking a branch.

I also agree with the contribution aspect, though it is less important in some cases.

I take the latest example I have in mind where this could have been useful: For integration into F-Droid, RiotX needed not to include binary artifacts of a library, but the source itself. The source repository is quite big (multiple languages), but the thing of interest is a single java file [1]. They ended up simply copy-pasting the file [2] in their repo, which makes its origin less obvious, and more subject to bit-rot and vulnerabilities.

[1]: https://github.com/google/diff-match-patch/blob/master/java/...

[2]: https://github.com/vector-im/riotX-android/pull/760

Re: Bring your monorepo down to size with sparse-checkout

#65

Post by gdxhyrd »

peterwwillis wrote: post #60
gdxhyrd wrote: post #57

Earlier quoted context omitted.

> This is what confuses me about monorepos. Their design requires an array of confusing processes and complex software to make the process of merging, testing, and releasing code manageable at scale (and "scale" can even be 6 developers working on 2 separate features each across 10 services, in one repo). False. It is having multiple repos what creates those problems and a huge graph of versions and dependencies. Wha…

> It is having multiple repos what creates those problems and a huge graph of versions and dependencies. Bazel, the open source version of Google's CI tool, is built specifically to handle "build dependencies in complex build graphs" . With monorepos. If it didn't do that, you'd never know what to test, what to deploy, what service depends on what other thing, etc. Versions and dependencies are inherent to any collec…

You are conflating language/build issues with VCS issues.

Everything you discuss also applies to multirepo, but worse, because there no one enforces consistency across all the project and you will end up with a broken interdependency.

Re: Bring your monorepo down to size with sparse-checkout

#66

Post by gen220 »

gdxhyrd wrote: post #59
gen220 wrote: post #50

Earlier quoted context omitted.

I understand the reasoning, and agree that it’s not always abuse. At first blush it’s a good idea, but I’d maintain that it’s one of the things that balloons your repo size quite quickly. Plus, one have to draw a line somewhere on what to include (a Python interpreter? A Go version? awk and grep?), and third party vs in-house is a fairly robust one imo. We host a private mirror for third party dependencies, so that “…

> Plus, one have to draw a line somewhere on what to include (a Python interpreter? A Go version? awk and grep?), and third party vs in-house is a fairly robust one imo. If your code/project/company uses the dependency in any way in production and it is not a part of the base system (which should be reproducibly installed), you include it; either in source or binary form. Why is the size a problem? Developers should…

It's a problem if the first step of your build system is a fresh `git pull` :)

Not unsolvable of course, just necessitates an extra layer of complexity.

Re: Bring your monorepo down to size with sparse-checkout

#67

Post by andrewg »

marcell wrote: post #33
krupan wrote: post #31

How does a sparse checkout not defeat the purpose of a monorepo? I thought monorepos existed so it was easy to make changes that affect the whole codebase and to test those changes. If you only checkout a portion of the files, how are you going to test against the whole repo? EDIT: my overall concern is that it looks like people are reinventing clearcase. Please speak to an older developer who worked at an HP/IBM typ…

Continuous integration tools still check out and test the whole repo. Google has used this approach for over a decade.

This would be impractical for really large monorepos like the ones Google and Microsoft have. They have virtual file system layers on top (MS open sourced theirs) to prevent checking out the whole repo.

In fact, it’s not just useful for the CI/CD pipeline - any developers making significant changes to base libraries or core infrastructure should be able to use the VFS in combination with a system like Bazel to run all (or a significant sample of) affected tests across the company.

Re: Bring your monorepo down to size with sparse-checkout

#68

Post by krupan »

qznc wrote: post #47
krupan wrote: post #31

How does a sparse checkout not defeat the purpose of a monorepo? I thought monorepos existed so it was easy to make changes that affect the whole codebase and to test those changes. If you only checkout a portion of the files, how are you going to test against the whole repo? EDIT: my overall concern is that it looks like people are reinventing clearcase. Please speak to an older developer who worked at an HP/IBM typ…

They are hard to find. Do you know some? All I have is this thread: https://lobste.rs/s/fosip5/should_version_control_build_syst...

I used clearcase many years ago, and this thread on lobste.rs is pretty accurate and interesting. They point out that the biggest problems were exclusive checkouts, file versioning instead of changesets, and the baked in out of date assumptions about networking. Getting your configspec wrong was a common problem too.

At HP we had some in-house perl-script wrappers around the raw clearcase tools that fixed many of these problems. The developers of those scripts had all left to go work for Rational (makers of clearcase), and I don't think anyone really knew how they worked. We also had a full-time clearcase engineer that kept the servers running. Fortunately our smallish projects didn't need the full power of clearcase and those perl scripts kept working fine for us. I did alway wonder what would happen if the one guy who understood the servers left the company.

In short, it's a complex and powerful tool that very few people really understood. Very few projects need all that power and complexity. I'm sure Microsoft and Google benefit from complex version control tools and have engineers to spare for managing and understanding them, but I don't think any open source projects or smaller companies are really going to benefit from "clearcase for the modern age" type tools.

Post reply on HN