Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bring your monorepo down to size with sparse-checkout

github.blog

Post reply on HN
Print view View original

51–60 of 68 posts

Re: Bring your monorepo down to size with sparse-checkout

#51

Post by phonebanshee »

scarejunba wrote: post #30
monocasa wrote: post #13

Earlier quoted context omitted.

Sometimes you don't own the other repo.

Doesn't that seem like a build tool situation? At that point the other piece of code isn't part of source, it's a source dependency, and no different from a binary dependency at some version so you don't really want the tree, you want the file at some revision and if it's `github` based then you have the natural HTTP endpoint and otherwise it's trivial to proxy as an artifact.

Not really. You often need to make extensive changes in those kinds of external dependencies, so you really do want them in your source tree.

Re: Bring your monorepo down to size with sparse-checkout

#52

Post by scarejunba »

phonebanshee wrote: post #51
scarejunba wrote: post #30

Earlier quoted context omitted.

Doesn't that seem like a build tool situation? At that point the other piece of code isn't part of source, it's a source dependency, and no different from a binary dependency at some version so you don't really want the tree, you want the file at some revision and if it's `github` based then you have the natural HTTP endpoint and otherwise it's trivial to proxy as an artifact.

Not really. You often need to make extensive changes in those kinds of external dependencies, so you really do want them in your source tree.

Oh, interesting. That explains why you want to retain the history, so you can easily merge and stuff too.

Re: Bring your monorepo down to size with sparse-checkout

#53

Post by peterwwillis »

gen220 wrote: post #50
Hackbraten wrote: post #48

Earlier quoted context omitted.

Checking in third-party dependencies is not always abuse. It can be a useful habit for certain kinds of reproducible builds. The Buck documentation even endorses keeping your dependencies in your monorepo along with your own sources.

I understand the reasoning, and agree that it’s not always abuse. At first blush it’s a good idea, but I’d maintain that it’s one of the things that balloons your repo size quite quickly. Plus, one have to draw a line somewhere on what to include (a Python interpreter? A Go version? awk and grep?), and third party vs in-house is a fairly robust one imo. We host a private mirror for third party dependencies, so that “…

If you can do that with 3rd party dependencies, can't you do that with all the code?

This is what confuses me about monorepos. Their design requires an array of confusing processes and complex software to make the process of merging, testing, and releasing code manageable at scale (and "scale" can even be 6 developers working on 2 separate features each across 10 services, in one repo).

But it turns out that you can also develop individual components, version their releases, link their dependencies, and still have a usable system. That's literally how all Linux distros have worked for decades, and how most other language-specific packaging systems work. None of which requires a monorepo.

So what I'd like to know is, of the 3 actual reasons I've heard companies claim are why they need a monorepo, is it impossible to do these things with multirepo? If it is indeed "hard" to do, is it "so hard" that it justifies all the complexity inherent to the monorepo? Or is it really just a meme? And are these things even necessary at all, if other systems seem to get away without it?

Re: Bring your monorepo down to size with sparse-checkout

#54

Post by gen220 »

peterwwillis wrote: post #53
gen220 wrote: post #50

Earlier quoted context omitted.

I understand the reasoning, and agree that it’s not always abuse. At first blush it’s a good idea, but I’d maintain that it’s one of the things that balloons your repo size quite quickly. Plus, one have to draw a line somewhere on what to include (a Python interpreter? A Go version? awk and grep?), and third party vs in-house is a fairly robust one imo. We host a private mirror for third party dependencies, so that “…

If you can do that with 3rd party dependencies, can't you do that with all the code? This is what confuses me about monorepos. Their design requires an array of confusing processes and complex software to make the process of merging, testing, and releasing code manageable at scale (and "scale" can even be 6 developers working on 2 separate features each across 10 services, in one repo). But it turns out that you can…

These are great questions!! :)

> Can you treat all code like 3rd party dependencies?

Yes, but there are trade-offs. Discoverability, enforcing hard deadlines on global changes, style consistency, etc.

> Is it impossible to do these things with multi-repo?

No, but there are trade-offs to consider.

> If it's hard, is it "so hard" that it justifies the complexity?

Hitting the nail on the head; there are trade-offs :)

> Are these things necessary, if other systems get away without it?

There are many stable equilibria; open source ecosystem evolved one solution and large companies evolved another, because they have been subject to very different constraints. The organization of the open source projects is extremely different from the organization of 100+ engineer companies, even if the contributor headcounts are similar.

For me, the the semantic distinction between monorepos and multirepos is the same as the distinction between internal and 3rd party dependencies. Does your team want to treat other teams as a 3rd party dependency? The correct answer depends on company culture, etc. It's a set of tradeoffs, including transparency over privacy, consistency over freedom, collaboration over compartmentalization.

With monorepos, you can gain a little privacy, freedom, and compartmentalization by being clever, but get the rest for cheap; vice versa for multirepos. It's trading one set of problems for another. I'd challenge the base assumption that multirepos are "simpler", they're just more tolerant of chaos, in a way that's very valuable for the open source community.

I hope we've not been talking past each other, I really like the ideas your raising! :)

Re: Bring your monorepo down to size with sparse-checkout

#55

Post by gdxhyrd »

gen220 wrote: post #41
gdxhyrd wrote: post #21

Earlier quoted context omitted.

What problems did you encounter with just a few services? Monorepos should be straightforward unless you are managing the code of >1k engineers.

We’ve run into some nontrivial but totally solvable issues at about 100-200 engineers. IME, most consternation comes from people adopting a mono repo without adopting a build/dependency graph tool (like Bazel, buck or pants). An additional source of strain is from people abusing the repo (checking in large binaries, third party dependencies, etc). A third is when people try to do branch-based feature development, ins…

> IME, most consternation comes from people adopting a mono repo without adopting a build/dependency graph tool (like Bazel, buck or pants).

That seems like a build problem, not a Git problem.

> An additional source of strain is from people abusing the repo (checking in large binaries, third party dependencies, etc).

That is not necessarily abuse. In fact, it is a good practice in many cases!

> A third is when people try to do branch-based feature development, instead of the “correct” practice of only deploying master (or weekly cuts of master).

I am not sure what you mean by branch-based development, but I don't see why that would be a specific problem of monorepos.

Re: Bring your monorepo down to size with sparse-checkout

#56

Post by gdxhyrd »

peterwwillis wrote: post #43
gdxhyrd wrote: post #21

Earlier quoted context omitted.

What problems did you encounter with just a few services? Monorepos should be straightforward unless you are managing the code of >1k engineers.

How are they straightforward? Like rebuilding a car's engine is straightforward? If you know how they're built, it's easy...

What? I don't understand what that means.

A monorepo is just 1 repo. There is nothing more straightforward than that.

Re: Bring your monorepo down to size with sparse-checkout

#57

Post by gdxhyrd »

peterwwillis wrote: post #53
gen220 wrote: post #50

Earlier quoted context omitted.

I understand the reasoning, and agree that it’s not always abuse. At first blush it’s a good idea, but I’d maintain that it’s one of the things that balloons your repo size quite quickly. Plus, one have to draw a line somewhere on what to include (a Python interpreter? A Go version? awk and grep?), and third party vs in-house is a fairly robust one imo. We host a private mirror for third party dependencies, so that “…

If you can do that with 3rd party dependencies, can't you do that with all the code? This is what confuses me about monorepos. Their design requires an array of confusing processes and complex software to make the process of merging, testing, and releasing code manageable at scale (and "scale" can even be 6 developers working on 2 separate features each across 10 services, in one repo). But it turns out that you can…

> This is what confuses me about monorepos. Their design requires an array of confusing processes and complex software to make the process of merging, testing, and releasing code manageable at scale (and "scale" can even be 6 developers working on 2 separate features each across 10 services, in one repo).

False. It is having multiple repos what creates those problems and a huge graph of versions and dependencies.

What "processes" are you talking about?

Re: Bring your monorepo down to size with sparse-checkout

#58

Post by war1025 »

pcl wrote: post #46
war1025 wrote: post #19

Earlier quoted context omitted.

I guess it's possible you don't get it all, but I've definitely ran `git grep` before on that repo and had results come back that weren't in my worktree. Edit: wrowclif@wrowclif-desktop:~/Taccs2/p5_deps$ git grep "def returnValue" twisted/install_linux_gcc54/lib/python2.7/site-packages/twisted/internet/defer.py:1350:def returnValue(val): twisted/install_linux_gcc54/lib/python2.7/site-packages/twisted/internet/test/te…

I would speculate that the partial-clone implementation pulls down all the commits that touch any files that are required. Some of these commits would presumably include changes to other parts of the source tree. Perhaps `git grep` still matches on such commits?

partial clone is different from sparse checkout.

We are using sparse checkout. Partial clone is the one that only pulls down objects that are needed by the store.

Re: Bring your monorepo down to size with sparse-checkout

#59

Post by gdxhyrd »

gen220 wrote: post #50
Hackbraten wrote: post #48

Earlier quoted context omitted.

Checking in third-party dependencies is not always abuse. It can be a useful habit for certain kinds of reproducible builds. The Buck documentation even endorses keeping your dependencies in your monorepo along with your own sources.

I understand the reasoning, and agree that it’s not always abuse. At first blush it’s a good idea, but I’d maintain that it’s one of the things that balloons your repo size quite quickly. Plus, one have to draw a line somewhere on what to include (a Python interpreter? A Go version? awk and grep?), and third party vs in-house is a fairly robust one imo. We host a private mirror for third party dependencies, so that “…

> Plus, one have to draw a line somewhere on what to include (a Python interpreter? A Go version? awk and grep?), and third party vs in-house is a fairly robust one imo.

If your code/project/company uses the dependency in any way in production and it is not a part of the base system (which should be reproducibly installed), you include it; either in source or binary form.

Why is the size a problem? Developers should only be checking out once. If your repo hits the many-GiB mark, then you can apply more complex solutions (LFS, sparse, etc.) if it is a burden.

Re: Bring your monorepo down to size with sparse-checkout

#60

Post by peterwwillis »

gdxhyrd wrote: post #57
peterwwillis wrote: post #53

Earlier quoted context omitted.

If you can do that with 3rd party dependencies, can't you do that with all the code? This is what confuses me about monorepos. Their design requires an array of confusing processes and complex software to make the process of merging, testing, and releasing code manageable at scale (and "scale" can even be 6 developers working on 2 separate features each across 10 services, in one repo). But it turns out that you can…

> This is what confuses me about monorepos. Their design requires an array of confusing processes and complex software to make the process of merging, testing, and releasing code manageable at scale (and "scale" can even be 6 developers working on 2 separate features each across 10 services, in one repo). False. It is having multiple repos what creates those problems and a huge graph of versions and dependencies. Wha…

> It is having multiple repos what creates those problems and a huge graph of versions and dependencies.

Bazel, the open source version of Google's CI tool, is built specifically to handle "build dependencies in complex build graphs". With monorepos. If it didn't do that, you'd never know what to test, what to deploy, what service depends on what other thing, etc. Versions and dependencies are inherent to any collection of independently changing "things".

Even if you build every service you have every time you commit a single line of code to any service, and run every test for every service any time you change a single line of code, the end result of all those newly-built services is still a new version. A change in that line of code still reflects the service it belongs to, and so thinking about "this change to this service" involves things like "other changes to other services", and so you need to be able to refer to one change when you talk about a different change. But they are different changes, with different implications. You may need to go back to a previous "version" of a line of code for one service, so it doesn't negatively impact another "version" of a different line of code in a different service. Every line of code, compared to every other line of code, is a unique version, and you have to track them somehow. You can use commit hashes or you can use semantic versions, it doesn't matter.

So because versions and dependencies are inherent to any collection of code, regardless of whether it's monorepo or multirepo, I don't buy this "it's easier to handle versions/dependencies" claim. In practice it doesn't seem to matter at all.

> What "processes" are you talking about?

Developer A and developer B are working on changes A1 and B1. Both are in review. Change A1 is merged. Now B1 needs to merge A1: it becomes B1.1. Fixing conflicts, running tests, and fixing anything changed finally results in B1.2, which goes into review. Now A develops and merges A2, so B1.2 goes through it all over again to become B1.4.

You can do all of that manually, but it's time-consuming, and the more people and services involved, the more time it takes to manage it all. So you add automated processes to try to speed up as much of it as you can: automatically merging the mainline into any open PRs and running tests, and doing this potentially with a dozen different merged items at once. Hence tools like Bazel, Zuul, etc. So, those processes.

Post reply on HN