Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Put Half a Million Files in One Git Repository, Here’s What We Learned

canvatechblog.com

Post reply on HN
Print view View original

211–220 of 270 posts

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#211

Post by pphysch »

stevenjgarner wrote: post #207

I get a headache imagining everything Google in a monorepo. So how do you reconcile this single GIT repository approach with Infoworld's "The case against monorepos"? [1]: Reason #1. Monorepos go against single-team ownership principles Reason #2. Monorepos encourage bad practices involving massive refactoring Reason #3: Small repositories are better than large ones [1] https://www.infoworld.com/article/3638860/the-c…

One person's STOP is another's silo

One person's massive refactoring is another's tech debt reduction

One person's multirepo is another's inability to find that broken bit of upstream code

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#212

Post by pessimizer »

AshamedCaptain wrote: post #175
codeyperson wrote: post #119

Earlier quoted context omitted.

It’s actually even worse than that I think. If something takes over a certain amount of time, then I’m more likely to go do something else while I wait, like check Hackernews. And there goes 20 minutes.

I'm hoping at least you acknowledge this is _your_ problem, rather than a tooling problem or the like. You just can't expect everything to give you immediate feedback after a couple seconds. This is something that's really a degradation of the newer generations of engineers, since I clearly remember the time where these "somethings" would never take less than a couple minutes, and people did not immediately flee to t…

That's a lot of drama, existential angst, and nostalgia around the equivalent of "sometimes during the commercials I go to the bathroom, and when I do I sometimes miss a minute of the show because I don't get back in time."

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#213

Post by rkangel »

urban_winter wrote: post #39

What's the value of a monorepo if developers only ever check out a small subset of it? Wouldn't multiple repos allow greater scale without any practical reduction in utility? For example, all the localisation files could live in a separate project (if we accept the need to commit them at all). Some tools would be needed to deal with the inevitable problem that developer working sets would not align with project bound…

> What's the value of a monorepo if developers only ever check out a small subset of it?

There's a couple of different views of this:

If the subsets are overlapping then the monorepo has had great value. Let's say you've got modules A, B, C and D. Dev 1 is interested in A and B, Dev 2 is interested in B and C, etc. In a multirepo world you have to draw a line somewhere and if someone has concerns overlapping that line then they're going to have to play the 'updating two projects with versioning' game.

The other way of looking at it is "data model" vs "presentation". Too often with git we confuse the two. Spare checkout is a way of presenting the relevant subset to each user. It is nice to be able to consider that separately from whether we want to store all that data together.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#214

Post by hamandcheese »

wewtyflakes wrote: post #206
hamandcheese wrote: post #156

Earlier quoted context omitted.

Suppose you want to deprecate an API you wrote in favor of something else for $valid_reasons. In a monorepo with the right tooling, I can make a branch where I delete the API and get pretty immediate feedback as to every module I have broken. From there I can update all the call sites and I also know which teams/engineers I should give a heads-up to. In a multi-repo world, this is much more difficult. Even learning w…

The perceived ease of that operation in a monorepo is what makes it dangerous. Unless these changes all map to a single monolith service, then even though you have updated all call sites, these callers will not be deployed all at the same time, meaning as the change rolls out you may see random breakage. By using polyrepo, the deployment boundary can align with the code boundary, making the rollout problem obvious.

People point this out often, but in practice I have never seen this cause issues. Remove callers first, then remove endpoint... its pretty obvious the order in which things need to be done.

What I have seen as a real problem, time and time again, is having trouble locating all usages of an API in a multi-repo scenario.

Anyone who fucks this up in a monoreppo will probably fuck it up worse with multiple repos.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#215

Post by jeffbee »

maccard wrote: post #98

I appreciate this post. It's nice to see that there are other teams that feel some of the pain points of git, (and unsurprising that most of the responses are "you're holding it wrong"). The fact is that git doesn't scale to _very large_ repos, We've seen it time and time again, but there isn't really a great alternative. Perforce is.... Perforce (centralized, very expensive to license, branches are incredibly expens…

Just saying that Perforce "is Perforce" is pretty lazy. Perforce is great in my experience. It scales up to repo sizes you are unlikely to ever reach and has a UX that makes actual sense.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#216

Post by compiler-guy »

kanu666 wrote: post #134
rtpg wrote: post #64

Earlier quoted context omitted.

Right, it's "I have to send 5 PRs to 5 different repos, get them all cross merged, and then at the end it's wrong anyways so I have to start all over". Multirepo management is extremely frustrating compared to "it's all in the same folder".

But in the monorepo you almost never can do the change in a single commit as it will cause incompatibilities during gradual deployment

Google does such a thing in its monorepo quite routinely.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#217

Post by jeffbee »

kanu666 wrote: post #139

If you don't have a couple of hundreds engineers who would write a custom file system for git (Microsoft), or who would take an existing source control system and nearly fully rewrite it (Facebook), or who would write a custom source control system from scratch (Google, Yandex, etc) DON'T USE MONOREPO. Otherwise you are risking to end up in a situation when hundreds of your engineers have to spend tens of minutes eve…

Google did not need to write their own SCM to reach the scale mentioned in the article. Off-the-shelf Perforce served that company well enough when it had tens of thousands of engineers and millions of files. They had 11 million files in a monorepo with one 256GB machine which most of you would consider a bare bones desktop computer at this point.

https://www.perforce.com/sites/default/files/still-all-one-s...

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#219

Post by cmcconomy »

tasn wrote: post #185
cnity wrote: post #120

Earlier quoted context omitted.

One alternative that enables you to keep generated files out but still feel like there's an explicit human check in place is to add a gated confirmation step in CI to confirm that the changes to the generated spec match expectations. Something like: "This change will result in the following new API endpoints: ... do you wish to continue?"

Hm... An interesting thought! What does it compare against though? Need to add more state to the CI? We kinda like the interface be part of the version control and having an audit chain that's part of the code.

you could put it in the pre-commit githook
Post reply on HN