Live data from Hacker News

We Put Half a Million Files in One Git Repository, Here’s What We Learned

canvatechblog.com

31–40 of 270 posts

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#31
post #2

>just under 60 million lines of code in 2022. I'm always surprised at the fact that almost every product has more lines of code than the entire Linux repo. The scale of these products is astounding.

From the rest of the article, it sounds like a big chunk of these lines are from generated files. What I don't understand is why they're checking in generated files into Git.

We check-in some generated CSS files. That are generated by external theme cli. Just to be sure, that after version update we can track all changes in the generated CSS files.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#32
post #28

I feel like I've read about several big companies using monorepos, but I've never understood why. It feels like the source-control equivalent of writing your code in one big file. Does anyone have any good resources for why and how best to implement a monorepo?

You kind of have to experience the worst of boths worlds to understand where and how each method works and breaks down.

With multiple repos it's harder for teams to share code and collaborate. Each team has a repo that becomes a little fiefdom where they are oblivious to who is using their code and how they're using it. Suddenly they'll push out what they think is an innocuous refactor and inadvertently break core functionality other teams took a dependency on for better or worse.

So what happens is the team with a dependency now copies the old code into their repo and take on all the extra burden of maintaining this old version, trying to backport fixes, etc. It becomes an enormous mess and time sink. No one ever has time to go back and fix things, and when they eventually are forced to do so it costs more time and effort than it would have taken to do it right from the start. You'll also run into horrible versioning problems where you're stuck on old version X but depend on widget foo which needs current version Y of that dependency.

You might say well bad on that team they should have engaged the product managers, made sure their dependencies and usage were well tracked with them, been looped in the process of changes, etc... but in the real world when your boss says X feature needs to be shipped in a few days all of that process goes out the window.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#33

It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…

Remember back when people recommended commiting node_modules into git?

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#34
The fact that Canva has a `Source Control team` with at least 5 people on it (going by the thanks at the bottom of the article), means they should probably try a different approach.

I think it's a cool company, with a good product, but they're WAY too small to be having a "source control team" on staff. That's at least 1.2MM a year salary / benefits cost.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#36
post #34

The fact that Canva has a `Source Control team` with at least 5 people on it (going by the thanks at the bottom of the article), means they should probably try a different approach. I think it's a cool company, with a good product, but they're WAY too small to be having a "source control team" on staff. That's at least 1.2MM a year salary / benefits cost.

[deleted]

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#37
post #31

Earlier quoted context omitted.

From the rest of the article, it sounds like a big chunk of these lines are from generated files. What I don't understand is why they're checking in generated files into Git.

We check-in some generated CSS files. That are generated by external theme cli. Just to be sure, that after version update we can track all changes in the generated CSS files.

If they're generated you can just re-generate from source every time you need to track changes.

You're using git as a cache. You don't need to version a cache.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#38
> These .xlf files are generated and contain translated strings for each locale

... and they make up 70% of their repo

Why would you include generated file in a repo?

Do they take to long to remake?

[EDIT]: especially given the fact that they're using bazel which is supposed to be the bee's knees of build system?

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#39
What's the value of a monorepo if developers only ever check out a small subset of it? Wouldn't multiple repos allow greater scale without any practical reduction in utility?

For example, all the localisation files could live in a separate project (if we accept the need to commit them at all). Some tools would be needed to deal with the inevitable problem that developer working sets would not align with project boundaries, but that seems like an easier job than making git scale while maintaining response times.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#40
post #21

Earlier quoted context omitted.

> It's still surprising to have any generated things there. E.g. you could make the same case for keeping built binaries in Git as well. This is not surprising at all. In fact, it's quite standard to commit string translations. Just because you can run the code generation/string replacement step as part of the build that does not mean it's a good idea to generate everything from scratch at every single build. String…

I'm not saying don't have the translations at all. I'm saying: 1) caching things in git in general is a bad idea; why is it not in this case? 2) these are not - to my understanding - the raw resource files, but rather machine-generated intermediate files. This is why it's about caching, rather than minimal source files. Additionally, to respond to your comment, if string translations don't change much then it may be…

> I'm not saying don't have the translations at all. I'm saying: 1) caching things in git in general is a bad idea (...)

You're missing the point. Storing translated files is caching things in git, and it is not a bad idea. It's a standard practice that saves your neck.

You either place faith on a build step working deterministically when it was not designed to work like that, or you track your generated files in your version control system.

If you decide to put faith on your ability to run deterministic builds with a potentially non-deterministic system, you waste minutes with each build regenerating files that you could very well have checked out and in the process risk sneaking in hard to track bugs. Then you need to have internationalization test steps for each localization running as part of your integration tests to verify if your build worked, which consume even more resources.

Or... you stash them in git?

You use git to track changes, regardless of where they came from. Just because you place faith in some build step to always work deterministically that does not mean you are following a good practice and everyone else around you is wrong.

Post reply on HN