Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Put Half a Million Files in One Git Repository, Here’s What We Learned

canvatechblog.com

Post reply on HN
Print view View original

151–160 of 270 posts

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#151

Post by hamandcheese »

urban_winter wrote: post #39

What's the value of a monorepo if developers only ever check out a small subset of it? Wouldn't multiple repos allow greater scale without any practical reduction in utility? For example, all the localisation files could live in a separate project (if we accept the need to commit them at all). Some tools would be needed to deal with the inevitable problem that developer working sets would not align with project bound…

> without any practical reduction in utility?

There is a massive loss in dependency management if you move to multiple repos.

Do polyrepo build systems exist that give you the same capabilities as bazel? Particularly with regard to querying your dependency graph.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#152

Post by necovek »

prmoustache wrote: post #121
haizzz wrote: post #56

Hey everyone, author here, the article is a bit misleading in that .xlf files aren't really generated files, they're created through our translation pipeline by real humans. I considered them generated in the sense that they're not directly worked on by engineers who have to deal with them in the repository. The content of these translation files are snapshot in time aligned with the text in our product so simply rem…

Can't those xlf files be stored in a separate repo or in an object storage and let the build system fetch them from there?

That wouldn't be called a "monorepo" then :D

Obviously, the problems they are solving (and admitting to solving) are due to their dedication to the monorepo.

With all the effort spent on working around the drawbacks, I really wonder what advantages they are seeing that make it worth their while?

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#153

Post by sigio »

dotancohen wrote: post #94
anitil wrote: post #76

Earlier quoted context omitted.

... are we no longer doing 'works on my machine' ?

Ostensibly if it works in Alice's Docker instance, it will run in Bob's Docker instance too.

Well... unless some dev's have M1-macs, and some of the docker layers are not available for arm, or the other way around, not available for amd64. Gives interesting issues.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#154

Post by necovek »

urban_winter wrote: post #39

What's the value of a monorepo if developers only ever check out a small subset of it? Wouldn't multiple repos allow greater scale without any practical reduction in utility? For example, all the localisation files could live in a separate project (if we accept the need to commit them at all). Some tools would be needed to deal with the inevitable problem that developer working sets would not align with project bound…

> For example, all the localisation files could live in a separate project (if we accept the need to commit them at all).

That's the wrong way to split files: it's as if you said let's split a monorepo so all the .sh files are in one repo, all the Makefiles are in another, all the .py files in yet another...

What you want is to split into "natural" repositories instead. Having 50 or 150 localisation files in an otherwise 40-file repo is not a big deal for anyone. Of course, how the split happens would have an outsized influence on the ergonomics.

Also note that localisation files are tightly linked to source code (the way they use them, similar to GNU gettext model, though they do use XLIFF): you put English strings in the source code, and when you change them (reword, fix typos, or outright change them), all translations need to get their English version updated and translations potentially needing updates marked as such. In short, they are managing their translations as source code (even if translators would be using translation tools akin to IDEs for development).

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#155

Post by trasz »

dotancohen wrote: post #115
trasz wrote: post #103

Earlier quoted context omitted.

From my experience SVN isn’t significantly slower than git.

My experience is that anything dealing with a branch, especially but not exclusively creating branches, is very slow in SVN for a repo of any real size, basically anything with a framework. I do not remember if "stat" was particularly slow, but SVN in general is slow.

Yeah, branches are slow. Otoh blame is fast compared to git.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#156

Post by hamandcheese »

Thorentis wrote: post #80

I don't understand the logic of combining microservices with a monorepo. The whole point of microservices is that you don't care what is beyond the external contract of the service. Who cares how each individual team decides to name their stuff. Why do microservice teams need to care about having every single service checked out? Who or what would be bulk applying changes to all services? This is madness.

Suppose you want to deprecate an API you wrote in favor of something else for $valid_reasons.

In a monorepo with the right tooling, I can make a branch where I delete the API and get pretty immediate feedback as to every module I have broken. From there I can update all the call sites and I also know which teams/engineers I should give a heads-up to.

In a multi-repo world, this is much more difficult. Even learning what all the reverse dependencies are could be a challenge. Most likely, other teams have to pull in my changes on their own schedule, and other teams have very different incentives than my own. The cost of mistakes is therefore higher because they are more difficult to undo.

Monorepos are very important if you want to empower engineers to achieve broad changes across the org. Some people fundamentally disagree with this methodology. And a fair number of people just don’t care enough - marking the old thing deprecated, calling it a day, and letting it rot for eternity is good enough for them. These are the same people who you’ll find saying “not my job” a lot, in my experience.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#157

Post by necovek »

nsoonhui wrote: post #44
urban_winter wrote: post #39

What's the value of a monorepo if developers only ever check out a small subset of it? Wouldn't multiple repos allow greater scale without any practical reduction in utility? For example, all the localisation files could live in a separate project (if we accept the need to commit them at all). Some tools would be needed to deal with the inevitable problem that developer working sets would not align with project bound…

One of the benefits of monorepo is refactoring. You can just apply a renaming command across all of the files in the solution and all the related names are properly updated. Not that easy to get this to work on multiple repos.

I think it's important to understand how often is the OP's company benefiting from this: is it worth the trouble they've gone through through the years? I've checked the rest of engineering posts, and none of them talk of the benefits of using a monorepo explicitly.

Monorepos, like basically all solutions, solves some problems and introduces new ones which you didn't have before. It depends on each individual case which drawbacks are more worthwhile.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#158

Post by blackoil »

kanu666 wrote: post #133
nsoonhui wrote: post #44

Earlier quoted context omitted.

One of the benefits of monorepo is refactoring. You can just apply a renaming command across all of the files in the solution and all the related names are properly updated. Not that easy to get this to work on multiple repos.

How would you do refactoring over monorepo if you have sparse checkout?

CI/CD build the entire project. So if you make a breaking change in library the build will fail.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#159

Post by strken »

urban_winter wrote: post #39

What's the value of a monorepo if developers only ever check out a small subset of it? Wouldn't multiple repos allow greater scale without any practical reduction in utility? For example, all the localisation files could live in a separate project (if we accept the need to commit them at all). Some tools would be needed to deal with the inevitable problem that developer working sets would not align with project bound…

Atomic linearisable updates to your code.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#160

Post by hamandcheese »

nvarsj wrote: post #102
barbs wrote: post #28

I feel like I've read about several big companies using monorepos, but I've never understood why. It feels like the source-control equivalent of writing your code in one big file. Does anyone have any good resources for why and how best to implement a monorepo?

It’s mostly about avoiding code and work duplication. At scale, the waste on duplicate work across teams can be massive (think about setting up CI tooling for example). Mono repo let’s you solve tooling/build problems once and for all. The main drawback is scalability of the tools involved like git.

> The main drawback is scalability of the tools involved like git.

And if you can employ enough engineers to break git, you can probably afford a team to work on scaling git.

Post reply on HN