The fact that Canva has a `Source Control team` with at least 5 people on it (going by the thanks at the bottom of the article), means they should probably try a different approach. I think it's a cool company, with a good product, but they're WAY too small to be having a "source control team" on staff. That's at least 1.2MM a year salary / benefits cost.
Really weird. But maybe this team is in a country where developers cost 100x less? (: Still weird though.
We Put Half a Million Files in One Git Repository, Here’s What We Learned
191–200 of 270 posts
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#192>across half a million files >we found that .xlf files made up almost 70% of the total number of files If one xlf file is used to keep the translation for 1 language, how the heck do you have so many then ? Makes no sense. If you have many xlf files for 1 language, may God save your soul.
5 components, 5 languages, would be 25 xlf files.
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#193Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#194Earlier quoted context omitted.
Ah - that would explain why at my current job there was a node_modules directory in git with nearly 2 million lines of Javascript within. It is gone now.
The days before lock files were a thing and 'it works on my machine!' was rampant.
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#195The fact that Canva has a `Source Control team` with at least 5 people on it (going by the thanks at the bottom of the article), means they should probably try a different approach. I think it's a cool company, with a good product, but they're WAY too small to be having a "source control team" on staff. That's at least 1.2MM a year salary / benefits cost.
It's not exactly a great name for the team but it gets across the point - we handle developer experience from the point you want to push your code to when you merge it, so we also work on communicating with CI/CD, code review/ownership tooling, gitops bots etc. Git Performance is a big issue right now but there's no end in sight to all the other scaling problems we could work on so if it stops making sense, we can wo…
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#196What's the value of a monorepo if developers only ever check out a small subset of it? Wouldn't multiple repos allow greater scale without any practical reduction in utility? For example, all the localisation files could live in a separate project (if we accept the need to commit them at all). Some tools would be needed to deal with the inevitable problem that developer working sets would not align with project bound…
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#197Earlier quoted context omitted.
My experience is that anything dealing with a branch, especially but not exclusively creating branches, is very slow in SVN for a repo of any real size, basically anything with a framework. I do not remember if "stat" was particularly slow, but SVN in general is slow.
Yeah, branches are slow. Otoh blame is fast compared to git.
I have this alias in my .hgrc file fad=fastannotate -u -n -wbB --deleted
It's by the Facebook engineer Jun Wu who also made the even more awesome "absorb"
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#198It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…
> Didn't someone on the team say "hey, it takes 10 seconds to run git status, can we move this junk out and do this another way??" Why do you assume they didn't? Just because they arrived at a different conclusion than you that doesn't mean they didn't thought about it. I might very well mean you did not considered the tradeoffs they had to take into account, mainly because you're out of the loop.
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#199They made a bad design decision 10 years ago, have been fighting the fallout for years, and will be doing so forever and ever because things will only ever grow. They wrote a blog post on how clever they think all their workarounds are, at least one of which involves sparse-checkout -- which is perilously close to chopping up your monorepo into several, while still pretending monorepo is fine. I feel like somebody's…
What would be a better structure?
Depending on your use cases a zip file can work, e.g. Python packages can be imported from within a zip file, and the standard lib is distributed that way.
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#200Earlier quoted context omitted.
Except for weird Docker edge cases (extremely rare, but does happen).
Not rare at all. Docker is a congregation of technologies held together with duct tape and glue. Eg. permissions handling is completely different on Macs with Docker Desktop from the Linux dockerd stuff: on Macs, it automatically translates user ownership for any "mounted" local storage (like your code repository), whereas on Linux user IDs and host system permissions are preserved. Have some developers use Macs and…
No, it's not. What a wild conclusion to reach from the example you gave.