Bring your monorepo down to size with sparse-checkout • Hacker News Board

Re: Bring your monorepo down to size with sparse-checkout

#41

Post by gen220 » Sat, Jan 18, 2020, 4:18 AM UTC

gdxhyrd wrote: post #21

0xbadcafebee wrote: post #12

I wish somebody'd write a book on monorepos. I've run into only a handful of their problems when trying to manage production pipelines using just a dozen services, so I'm sure there's tons more (like the purpose behind this command). Nobody mentions the massive investment in time, technical expertise, compute resource, and money required to run large monorepos in production. Also, would emulating this command with a…

What problems did you encounter with just a few services? Monorepos should be straightforward unless you are managing the code of >1k engineers.

We’ve run into some nontrivial but totally solvable issues at about 100-200 engineers.

IME, most consternation comes from people adopting a mono repo without adopting a build/dependency graph tool (like Bazel, buck or pants).

An additional source of strain is from people abusing the repo (checking in large binaries, third party dependencies, etc).

A third is when people try to do branch-based feature development, instead of the “correct” practice of only deploying master (or weekly cuts of master).

I think even a simple list of these sort of “gotchas” would be valuable for the aspirational mono repo company.

My impression is that a lot of teams hit these early and painful roadblocks, and imagine that they’ll never go away (they do!!).

Re: Bring your monorepo down to size with sparse-checkout

#42

Post by fuckyah » Sat, Jan 18, 2020, 5:00 AM UTC

monorepos are just a disaster

Re: Bring your monorepo down to size with sparse-checkout

#43

Post by peterwwillis » Sat, Jan 18, 2020, 5:50 AM UTC

gdxhyrd wrote: post #21

0xbadcafebee wrote: post #12

I wish somebody'd write a book on monorepos. I've run into only a handful of their problems when trying to manage production pipelines using just a dozen services, so I'm sure there's tons more (like the purpose behind this command). Nobody mentions the massive investment in time, technical expertise, compute resource, and money required to run large monorepos in production. Also, would emulating this command with a…

What problems did you encounter with just a few services? Monorepos should be straightforward unless you are managing the code of >1k engineers.

How are they straightforward? Like rebuilding a car's engine is straightforward? If you know how they're built, it's easy...

Re: Bring your monorepo down to size with sparse-checkout

#44

Post by potatochup » Sat, Jan 18, 2020, 6:26 AM UTC

krupan wrote: post #34

MayeulC wrote: post #5

Interesting. I've been wanting something like that for submodules. Can the two features be combined? For instance, if you need a single file/directory from another project in your repository.

I thought one of the big benefits of monorepos was that you didn't mess with submodules anymore?

It still might be needed for external dependencies. the code an organization writes might be in one repo, but if you want to bring in some other library, like libssl (assuming there is no better package manager for your language) submodules are often used

Re: Bring your monorepo down to size with sparse-checkout

#45

Post by [deleted] » Sat, Jan 18, 2020, 7:48 AM UTC

kyrra wrote: post #29

wikibob wrote: post #26

Earlier quoted context omitted.

I'm surprised that VFS for Git isn't yet available on GitHub. Surely they are working on adding it? Anyone have an inside scoop?

2 reasons: VFS is a fork of normal Git. There is no Linux client. Also remember that there are many Git clients that work with normal Git repos. Like libgit and others. I doubt you'll see wide spread support for it unless MS can upstream it into the main Git implementation, and maybe some the the primary libraries. This is one nice argument for Mercurial, where there is only a single implementation, so adding big new…

[deleted]

Re: Bring your monorepo down to size with sparse-checkout

#46

Post by pcl » Sat, Jan 18, 2020, 8:37 AM UTC

war1025 wrote: post #19

vlovich123 wrote: post #18

Earlier quoted context omitted.

Are you sure about that? > This combination speeds up the data transfer process since you don’t need every reachable Git object, and instead, can download only those you need to populate your cone of the working directory If you're only downloading what you need to populate the working directory how is it that `.git` will have the entire repository?

I guess it's possible you don't get it all, but I've definitely ran `git grep` before on that repo and had results come back that weren't in my worktree. Edit: wrowclif@wrowclif-desktop:~/Taccs2/p5_deps$ git grep "def returnValue" twisted/install_linux_gcc54/lib/python2.7/site-packages/twisted/internet/defer.py:1350:def returnValue(val): twisted/install_linux_gcc54/lib/python2.7/site-packages/twisted/internet/test/te…

I would speculate that the partial-clone implementation pulls down all the commits that touch any files that are required. Some of these commits would presumably include changes to other parts of the source tree. Perhaps `git grep` still matches on such commits?

Re: Bring your monorepo down to size with sparse-checkout

#47

Post by qznc » Sat, Jan 18, 2020, 9:20 AM UTC

krupan wrote: post #31

How does a sparse checkout not defeat the purpose of a monorepo? I thought monorepos existed so it was easy to make changes that affect the whole codebase and to test those changes. If you only checkout a portion of the files, how are you going to test against the whole repo? EDIT: my overall concern is that it looks like people are reinventing clearcase. Please speak to an older developer who worked at an HP/IBM typ…

They are hard to find. Do you know some?

All I have is this thread: https://lobste.rs/s/fosip5/should_version_control_build_syst...

Re: Bring your monorepo down to size with sparse-checkout

#48

Post by Hackbraten » Sat, Jan 18, 2020, 1:49 PM UTC

gen220 wrote: post #41

gdxhyrd wrote: post #21

Earlier quoted context omitted.

What problems did you encounter with just a few services? Monorepos should be straightforward unless you are managing the code of >1k engineers.

We’ve run into some nontrivial but totally solvable issues at about 100-200 engineers. IME, most consternation comes from people adopting a mono repo without adopting a build/dependency graph tool (like Bazel, buck or pants). An additional source of strain is from people abusing the repo (checking in large binaries, third party dependencies, etc). A third is when people try to do branch-based feature development, ins…

Checking in third-party dependencies is not always abuse. It can be a useful habit for certain kinds of reproducible builds. The Buck documentation even endorses keeping your dependencies in your monorepo along with your own sources.

Re: Bring your monorepo down to size with sparse-checkout

#49

Post by mark_l_watson » Sat, Jan 18, 2020, 3:46 PM UTC

Partial checkout efficiency improvements makes mono repos more compelling for large projects and organizations.

As an individual, I have switched to a mono repo for all of my Common Lisp code and with some adjustments to my Quicklisp configuration I am very happy with my setup.

I am a programming language junkie, and I have it on my low priority todo list to switch to a mono repo for Haskell, Racket, and Hy language (Lisp with a Clojure syntax that sits on top of Python).

I worked as a contractor at Google in 2013 and I absolutely loved their mono repo and web based development environment. I really miss that.

Re: Bring your monorepo down to size with sparse-checkout

#50

Post by gen220 » Sat, Jan 18, 2020, 4:16 PM UTC

Hackbraten wrote: post #48

gen220 wrote: post #41

Earlier quoted context omitted.

We’ve run into some nontrivial but totally solvable issues at about 100-200 engineers. IME, most consternation comes from people adopting a mono repo without adopting a build/dependency graph tool (like Bazel, buck or pants). An additional source of strain is from people abusing the repo (checking in large binaries, third party dependencies, etc). A third is when people try to do branch-based feature development, ins…

Checking in third-party dependencies is not always abuse. It can be a useful habit for certain kinds of reproducible builds. The Buck documentation even endorses keeping your dependencies in your monorepo along with your own sources.

I understand the reasoning, and agree that it’s not always abuse. At first blush it’s a good idea, but I’d maintain that it’s one of the things that balloons your repo size quite quickly. Plus, one have to draw a line somewhere on what to include (a Python interpreter? A Go version? awk and grep?), and third party vs in-house is a fairly robust one imo.

We host a private mirror for third party dependencies, so that “pip install”/“go get” fail on our CI system if the dependency isn’t hosted by us. This gives us reproducible builds, while allowing us to hold 3rd party libraries to a higher standard of entry than source code. For certain libraries we pin version numbers in our build system, but in general it allows us to update dependencies transparently. It also keeps our source repo size small, for developers, and allows for conflicting versions (example Kafka X.Y and X.Z) without cluttering the repo with duplicates.

It’s definitely a smaller gotcha than the others I listed, maybe to the point where it’s not a gotcha, but I stand by it :)