Live data from Hacker News

We Put Half a Million Files in One Git Repository, Here’s What We Learned

canvatechblog.com

191–200 of 270 posts

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#191
post #34

The fact that Canva has a `Source Control team` with at least 5 people on it (going by the thanks at the bottom of the article), means they should probably try a different approach. I think it's a cool company, with a good product, but they're WAY too small to be having a "source control team" on staff. That's at least 1.2MM a year salary / benefits cost.

Really weird. But maybe this team is in a country where developers cost 100x less? (: Still weird though.

[deleted]

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#192

>across half a million files >we found that .xlf files made up almost 70% of the total number of files If one xlf file is used to keep the translation for 1 language, how the heck do you have so many then ? Makes no sense. If you have many xlf files for 1 language, may God save your soul.

One xlf per language _per component_.

5 components, 5 languages, would be 25 xlf files.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#194
post #53

Earlier quoted context omitted.

Ah - that would explain why at my current job there was a node_modules directory in git with nearly 2 million lines of Javascript within. It is gone now.

The days before lock files were a thing and 'it works on my machine!' was rampant.

Lock files protect you from the version changing out from under you, but modules disappearing from NPM is a thing that happens. Yes, you can use artifactory or similar as a proxy but that requires infrastructure that you may not want to run. That is all to say: there are situations where committing node_modules is the least evil.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#195
post #69
post #34

The fact that Canva has a `Source Control team` with at least 5 people on it (going by the thanks at the bottom of the article), means they should probably try a different approach. I think it's a cool company, with a good product, but they're WAY too small to be having a "source control team" on staff. That's at least 1.2MM a year salary / benefits cost.

It's not exactly a great name for the team but it gets across the point - we handle developer experience from the point you want to push your code to when you merge it, so we also work on communicating with CI/CD, code review/ownership tooling, gitops bots etc. Git Performance is a big issue right now but there's no end in sight to all the other scaling problems we could work on so if it stops making sense, we can wo…

I am curious why git modules couldn't be used here, and perhaps scripted to match particular workflows.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#196

What's the value of a monorepo if developers only ever check out a small subset of it? Wouldn't multiple repos allow greater scale without any practical reduction in utility? For example, all the localisation files could live in a separate project (if we accept the need to commit them at all). Some tools would be needed to deal with the inevitable problem that developer working sets would not align with project bound…

If you can checkout your monorepo as if it's multiple repos, but then also check it out as a monorepo when you want it, that seems to me more utility than splitting into multiple repos, then you can never check it out as a monorepo.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#197
post #155

Earlier quoted context omitted.

My experience is that anything dealing with a branch, especially but not exclusively creating branches, is very slow in SVN for a repo of any real size, basically anything with a framework. I do not remember if "stat" was particularly slow, but SVN in general is slow.

Yeah, branches are slow. Otoh blame is fast compared to git.

These days mercurial has a cached blame/annotate called "fast annotate" which I love because of one particular awesome feature --deleted which must be seen to be appreciated I think.

I have this alias in my .hgrc file fad=fastannotate -u -n -wbB --deleted

It's by the Facebook engineer Jun Wu who also made the even more awesome "absorb"

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#198
post #23

It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…

> Didn't someone on the team say "hey, it takes 10 seconds to run git status, can we move this junk out and do this another way??" Why do you assume they didn't? Just because they arrived at a different conclusion than you that doesn't mean they didn't thought about it. I might very well mean you did not considered the tradeoffs they had to take into account, mainly because you're out of the loop.

They’re not assuming anything, but stating that the post itself does not cover this well/at all.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#199
post #83

They made a bad design decision 10 years ago, have been fighting the fallout for years, and will be doing so forever and ever because things will only ever grow. They wrote a blog post on how clever they think all their workarounds are, at least one of which involves sparse-checkout -- which is perilously close to chopping up your monorepo into several, while still pretending monorepo is fine. I feel like somebody's…

What would be a better structure?

In a word, Sqlite. You can always put your files in a file. ("Yo dawg, I heard you like files so I...")

Depending on your use cases a zip file can work, e.g. Python packages can be imported from within a zip file, and the standard lib is distributed that way.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#200

Earlier quoted context omitted.

Except for weird Docker edge cases (extremely rare, but does happen).

Not rare at all. Docker is a congregation of technologies held together with duct tape and glue. Eg. permissions handling is completely different on Macs with Docker Desktop from the Linux dockerd stuff: on Macs, it automatically translates user ownership for any "mounted" local storage (like your code repository), whereas on Linux user IDs and host system permissions are preserved. Have some developers use Macs and…

> Docker is a congregation of technologies held together with duct tape and glue.

No, it's not. What a wild conclusion to reach from the example you gave.

Post reply on HN