Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Put Half a Million Files in One Git Repository, Here’s What We Learned

canvatechblog.com

Post reply on HN
Print view View original

111–120 of 270 posts

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#111

Post by rtpg »

Galanwe wrote: post #106
rtpg wrote: post #64

Earlier quoted context omitted.

Right, it's "I have to send 5 PRs to 5 different repos, get them all cross merged, and then at the end it's wrong anyways so I have to start all over". Multirepo management is extremely frustrating compared to "it's all in the same folder".

The process you're describing looks like some trial and error PRs... Multirepo also allows you to roll out that change incrementally instead of big banging all the time.

Well for trivial changes it's even worse, cuz instead of "change 3 files across this boundary" it's "send 2 sets of changes to different places, babysit it until merging, then send a third PR in the integration point to use the updated vesrion and then get it merged".

Meanwhile reviewers don't have context about changes, so it's easier to get lost in the weeds.

It's not always this, of course. But I think that way too many tools are based on "repo" being the largest element, so things like cross-repo review are just miserable.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#112

Post by robertlagrant »

arinlen wrote: post #40
robertlagrant wrote: post #30

Earlier quoted context omitted.

I'm not saying don't have the translations at all. I'm saying: 1) caching things in git in general is a bad idea; why is it not in this case? 2) these are not - to my understanding - the raw resource files, but rather machine-generated intermediate files. This is why it's about caching, rather than minimal source files. Additionally, to respond to your comment, if string translations don't change much then it may be…

> I'm not saying don't have the translations at all. I'm saying: 1) caching things in git in general is a bad idea (...) You're missing the point. Storing translated files is caching things in git, and it is not a bad idea. It's a standard practice that saves your neck. You either place faith on a build step working deterministically when it was not designed to work like that, or you track your generated files in you…

> Just because you place faith in some build step to always work deterministically that does not mean you are following a good practice and everyone else around you is wrong.

You're also doing that everywhere else. How do you think anything works? Why do you think Git is deterministic somehow? Why more so than including some files in a build?

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#113

Post by the_biot »

pech0rin wrote: post #107
the_biot wrote: post #73

They made a bad design decision 10 years ago, have been fighting the fallout for years, and will be doing so forever and ever because things will only ever grow. They wrote a blog post on how clever they think all their workarounds are, at least one of which involves sparse-checkout -- which is perilously close to chopping up your monorepo into several, while still pretending monorepo is fine. I feel like somebody's…

So every company ever? I love these comments like hindsight is not 20/20 and it would have been so simple for you to have made all of the right engineering decisions as an armchair CTO. Give me a break. Git repos get out of control, bad decisions are made, this is an interesting solution and write up. So over the whole “just make perfect decisions all the time and you would never need X comments”. Guess what? Everybo…

This isn't just a matter of hindsight. They don't even have that, as demonstrated by the fact they're putting ever more work into their workaround instead of fixing the glaringly obvious problem: it's a monorepo with half a million files growing ever bigger.

Certainly I wouldn't have decided to put all that junk into a single git repo from the start, since I'm not an idiot. But, even if I did, or I inherited something like that, I would fix it, not double down as they're doing.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#114

Post by charcircuit »

abrookewood wrote: post #100

I don't think I've ever seen "because your mono-repo is too big for git" used as an argument for micro-services, but maybe at this point it's valid.

You can put microservices in a monorepo. You just put them in their own folder.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#115

Post by dotancohen »

trasz wrote: post #103
dotancohen wrote: post #91

Earlier quoted context omitted.

> it takes 10 seconds to run git status People coming from the SVN world do not think that this is unusual or problematic. And unfortunately even recently I've seen SVN still in use at large legacy companies.

From my experience SVN isn’t significantly slower than git.

My experience is that anything dealing with a branch, especially but not exclusively creating branches, is very slow in SVN for a repo of any real size, basically anything with a framework.

I do not remember if "stat" was particularly slow, but SVN in general is slow.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#116

Post by abrookewood »

charcircuit wrote: post #114
abrookewood wrote: post #100

I don't think I've ever seen "because your mono-repo is too big for git" used as an argument for micro-services, but maybe at this point it's valid.

You can put microservices in a monorepo. You just put them in their own folder.

Of course, but can you split a monolith across multiple repos?

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#117

Post by Ygg2 »

maccard wrote: post #98

I appreciate this post. It's nice to see that there are other teams that feel some of the pain points of git, (and unsurprising that most of the responses are "you're holding it wrong"). The fact is that git doesn't scale to _very large_ repos, We've seen it time and time again, but there isn't really a great alternative. Perforce is.... Perforce (centralized, very expensive to license, branches are incredibly expens…

There was pijul. Which allowed partial pulls iirc. I haven't used it, so I can't really recommended it.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#118

Post by tasn »

rsanheim wrote: post #20

It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…

One of my biggest pet peeves: tracking generated files in version control.

The only exception is our generated OpenAPI spec, because we want people to be explicit about modifying the API, and have a CI task that verifies that the API and OpenAPI spec match.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#119

Post by codeyperson »

nomilk wrote: post #65

> git status takes 10 seconds on average > running these commands multiple times a day reduces the total productive time engineers have every day I love the attention paid to this. Often opportunities to prioritise seemingly small efficiency gains are neglected. At 10 seconds per command, an engineer that uses git status 50 times per day spends ~10 minutes per day waiting; an entire work week per year!! Well above th…

It’s actually even worse than that I think. If something takes over a certain amount of time, then I’m more likely to go do something else while I wait, like check Hackernews. And there goes 20 minutes.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#120

Post by cnity »

tasn wrote: post #118
rsanheim wrote: post #20

It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…

One of my biggest pet peeves: tracking generated files in version control. The only exception is our generated OpenAPI spec, because we want people to be explicit about modifying the API, and have a CI task that verifies that the API and OpenAPI spec match.

One alternative that enables you to keep generated files out but still feel like there's an explicit human check in place is to add a gated confirmation step in CI to confirm that the changes to the generated spec match expectations.

Something like: "This change will result in the following new API endpoints: ... do you wish to continue?"

Post reply on HN