Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Put Half a Million Files in One Git Repository, Here’s What We Learned

canvatechblog.com

Post reply on HN
Print view View original

131–140 of 270 posts

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#131

Post by etripe »

rtpg wrote: post #64
Galanwe wrote: post #55

Earlier quoted context omitted.

Sure, how did we ever manage to rename something without monorepos. Oh wait, maybe that's what this "versionning" thing is all about.

Right, it's "I have to send 5 PRs to 5 different repos, get them all cross merged, and then at the end it's wrong anyways so I have to start all over". Multirepo management is extremely frustrating compared to "it's all in the same folder".

How many scenarios are there where the rename both matters (beyond taste and philosophy) and is across interface boundaries?

Surely if it is an advantage to rename once in a ginormous, single code base there must also be leaky abstractions, poorly defined interfaces, god objects, etc present at the same time?

Whenever I find I need to rename anything across domains, it's a matter of updating the "core" repository and then just pulling the newest version.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#132

Post by [deleted] »

the_biot wrote: post #73

They made a bad design decision 10 years ago, have been fighting the fallout for years, and will be doing so forever and ever because things will only ever grow. They wrote a blog post on how clever they think all their workarounds are, at least one of which involves sparse-checkout -- which is perilously close to chopping up your monorepo into several, while still pretending monorepo is fine. I feel like somebody's…

[deleted]

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#133

Post by kanu666 »

nsoonhui wrote: post #44
urban_winter wrote: post #39

What's the value of a monorepo if developers only ever check out a small subset of it? Wouldn't multiple repos allow greater scale without any practical reduction in utility? For example, all the localisation files could live in a separate project (if we accept the need to commit them at all). Some tools would be needed to deal with the inevitable problem that developer working sets would not align with project bound…

One of the benefits of monorepo is refactoring. You can just apply a renaming command across all of the files in the solution and all the related names are properly updated. Not that easy to get this to work on multiple repos.

How would you do refactoring over monorepo if you have sparse checkout?

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#134

Post by kanu666 »

rtpg wrote: post #64
Galanwe wrote: post #55

Earlier quoted context omitted.

Sure, how did we ever manage to rename something without monorepos. Oh wait, maybe that's what this "versionning" thing is all about.

Right, it's "I have to send 5 PRs to 5 different repos, get them all cross merged, and then at the end it's wrong anyways so I have to start all over". Multirepo management is extremely frustrating compared to "it's all in the same folder".

But in the monorepo you almost never can do the change in a single commit as it will cause incompatibilities during gradual deployment

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#135

Post by mariusmg »

>across half a million files

>we found that .xlf files made up almost 70% of the total number of files

If one xlf file is used to keep the translation for 1 language, how the heck do you have so many then ? Makes no sense.

If you have many xlf files for 1 language, may God save your soul.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#136

Post by maccard »

Ygg2 wrote: post #117
maccard wrote: post #98

I appreciate this post. It's nice to see that there are other teams that feel some of the pain points of git, (and unsurprising that most of the responses are "you're holding it wrong"). The fact is that git doesn't scale to _very large_ repos, We've seen it time and time again, but there isn't really a great alternative. Perforce is.... Perforce (centralized, very expensive to license, branches are incredibly expens…

There was pijul. Which allowed partial pulls iirc. I haven't used it, so I can't really recommended it.

I tried pijul [0] last time this conversation came up and despite its claims it is not fast.

[0] https://news.ycombinator.com/item?id=29992875

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#137

Post by kanu666 »

maccard wrote: post #98

I appreciate this post. It's nice to see that there are other teams that feel some of the pain points of git, (and unsurprising that most of the responses are "you're holding it wrong"). The fact is that git doesn't scale to _very large_ repos, We've seen it time and time again, but there isn't really a great alternative. Perforce is.... Perforce (centralized, very expensive to license, branches are incredibly expens…

But that's what those responses are all about: git doesn't scale, so don't use it if you need scale. If there are no tools that scale available, then change the approach so you don't need face this scaling problem at all.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#138

Post by isaacremuant »

abrookewood wrote: post #116
charcircuit wrote: post #114

Earlier quoted context omitted.

You can put microservices in a monorepo. You just put them in their own folder.

Of course, but can you split a monolith across multiple repos?

Yes. With git submodules.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#139

Post by kanu666 »

If you don't have a couple of hundreds engineers who would write a custom file system for git (Microsoft), or who would take an existing source control system and nearly fully rewrite it (Facebook), or who would write a custom source control system from scratch (Google, Yandex, etc) DON'T USE MONOREPO.

Otherwise you are risking to end up in a situation when hundreds of your engineers have to spend tens of minutes every time they need to simply push their code changes.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#140

Post by diegoveralli »

raverbashing wrote: post #101
jamesfinlayson wrote: post #42

Earlier quoted context omitted.

Ah - that would explain why at my current job there was a node_modules directory in git with nearly 2 million lines of Javascript within. It is gone now.

Ugh The price of letting less experienced people "go crazy" in the repo

Npm didn't support lockfiles until version 5, released in 2017, Yarn had them at launch in 2016. Before that committing node_modules was often used as a form of vendoring, to get reproducible builds.

If a new project these days commits node_modules to git, it's likely a mistake, but for legacy projects started before 2017 it was the lesser of two evils.

Edit: spelling.

Post reply on HN