Live data from Hacker News

We Put Half a Million Files in One Git Repository, Here’s What We Learned

canvatechblog.com

41–50 of 270 posts

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#41
post #28

I feel like I've read about several big companies using monorepos, but I've never understood why. It feels like the source-control equivalent of writing your code in one big file. Does anyone have any good resources for why and how best to implement a monorepo?

Here are a few reasons that I've heard and experienced

1. Standardized tooling

2. Fewer issues related to dependencies

3. Hermetic tests

4. Reduced code duplication and easier code sharing

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#42
post #33

It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…

Remember back when people recommended commiting node_modules into git?

Ah - that would explain why at my current job there was a node_modules directory in git with nearly 2 million lines of Javascript within.

It is gone now.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#43
post #10
post #8

Earlier quoted context omitted.

Reading the article they are not generated files, but files that are never touched by developers. Translators will work with those files. I expect that for translators they have a different sparse checkout that only fetches .xlf files for their target languages.

xlf files are usually generated. They're XML. No one wants to write that by hand.

.docx files are archives of xml-files. No one wants to write that by hand.

Or, in more words: The format of the files is just the representation on disk - it’s not directly connected to how the files are generated or edited. XML files can be written by hand with suitable editor support.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#44

What's the value of a monorepo if developers only ever check out a small subset of it? Wouldn't multiple repos allow greater scale without any practical reduction in utility? For example, all the localisation files could live in a separate project (if we accept the need to commit them at all). Some tools would be needed to deal with the inevitable problem that developer working sets would not align with project bound…

One of the benefits of monorepo is refactoring. You can just apply a renaming command across all of the files in the solution and all the related names are properly updated.

Not that easy to get this to work on multiple repos.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#46
post #34

The fact that Canva has a `Source Control team` with at least 5 people on it (going by the thanks at the bottom of the article), means they should probably try a different approach. I think it's a cool company, with a good product, but they're WAY too small to be having a "source control team" on staff. That's at least 1.2MM a year salary / benefits cost.

Yeah... also once such a team exists, they are naturally going to resist ideas like not having a monorepo but just using git submodules, not keeping the history of every generated file forever etc.

Because then they won't need a 5-man team to handle the fallout any more.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#48

> These .xlf files are generated and contain translated strings for each locale ... and they make up 70% of their repo Why would you include generated file in a repo? Do they take to long to remake? [EDIT]: especially given the fact that they're using bazel which is supposed to be the bee's knees of build system?

The article uses the term generated, but a more precise term could be "produced by translators". The files can't be simply regenerated. We've recently published [1] a separate post that explains the localisation process in more detail.

[1]: https://news.ycombinator.com/item?id=28931601

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#49

It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…

Canva gives users without design tools expertise the ability to make fairly polished looking graphics with a super easy and intuitive interface. (As a designer, I can assure you that polished looking is not the same thing as designed.) It’s a very popular service, so they’re dealing with huge scale. Intuitive interfaces often come with complex mechanisms and lots of assets, and they have clients on every major mobile and desktop platform, and the web. They also do a ton of heavy graphics processing that is likely done in lower-level languages than the interface.

The likelihood of a 2000 employee software company simply not considering that they could streamline their build process is pretty slim.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#50
post #24

Earlier quoted context omitted.

The bigger issues for me are it makes history impossible to read (every change is hidden in an avalanche of crap), merges are a mess (you definitely want to spend forever merging autogened files, right?), PR reviews are annoying, etc.

Depends how much generated stuff is there. We have our graphql schema in git even though its auto generated via a library. But its useful in PRs to see exactly how the schema changed as a result of the root change.

Yeah if there's not a lot of it, and if it's easy to regenerate, it can be fine.
Post reply on HN