Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Put Half a Million Files in One Git Repository, Here’s What We Learned

canvatechblog.com

Post reply on HN
Print view View original

61–70 of 270 posts

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#61

Post by haizzz »

rsanheim wrote: post #20

It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…

hey hey author here, xlf files are translations that are coupled with the texts we set in the code so they're not really generated I admit that was misleading. What I wanted to get across is they're not touched directly by engineers but they're still created through our translation pipeline where real humans translate them

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#62

Post by milosmns »

syspec wrote: post #34

The fact that Canva has a `Source Control team` with at least 5 people on it (going by the thanks at the bottom of the article), means they should probably try a different approach. I think it's a cool company, with a good product, but they're WAY too small to be having a "source control team" on staff. That's at least 1.2MM a year salary / benefits cost.

Really weird. But maybe this team is in a country where developers cost 100x less? (: Still weird though.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#63

Post by ratww »

arinlen wrote: post #40
robertlagrant wrote: post #30

Earlier quoted context omitted.

I'm not saying don't have the translations at all. I'm saying: 1) caching things in git in general is a bad idea; why is it not in this case? 2) these are not - to my understanding - the raw resource files, but rather machine-generated intermediate files. This is why it's about caching, rather than minimal source files. Additionally, to respond to your comment, if string translations don't change much then it may be…

> I'm not saying don't have the translations at all. I'm saying: 1) caching things in git in general is a bad idea (...) You're missing the point. Storing translated files is caching things in git, and it is not a bad idea. It's a standard practice that saves your neck. You either place faith on a build step working deterministically when it was not designed to work like that, or you track your generated files in you…

It is definitely possible to have determinism in a CI build step, and it's possible to have checks for it. If one needs determinism and a cache, they can store the files on S3 or some other place instead of git. Re-generating the files every time on the build isn't the only alternative. Instead of generate-and-commit, generate and upload. The difficulty is the same for developers.

If one has to be more granular than that, and have versioning and verification against the repository, they can still store the multiple versions on another service and store the hashes on git. Even though I'm not a fan of this for translation (especially if you have lots of languages/lots of strings), since there's an advantage of decoupling the translation process from the development process.

The problem with storing those files on git is that it can cause more problems, including developer experience issues.

It depends on how much you're storing on git. Some CSS files? Fine. 70% of files of the project, like in this case, slowing down everyone's workflow? Definitely not.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#64

Post by rtpg »

Galanwe wrote: post #55
nsoonhui wrote: post #44

Earlier quoted context omitted.

One of the benefits of monorepo is refactoring. You can just apply a renaming command across all of the files in the solution and all the related names are properly updated. Not that easy to get this to work on multiple repos.

Sure, how did we ever manage to rename something without monorepos. Oh wait, maybe that's what this "versionning" thing is all about.

Right, it's "I have to send 5 PRs to 5 different repos, get them all cross merged, and then at the end it's wrong anyways so I have to start all over".

Multirepo management is extremely frustrating compared to "it's all in the same folder".

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#65

Post by nomilk »

> git status takes 10 seconds on average

> running these commands multiple times a day reduces the total productive time engineers have every day

I love the attention paid to this. Often opportunities to prioritise seemingly small efficiency gains are neglected.

At 10 seconds per command, an engineer that uses git status 50 times per day spends ~10 minutes per day waiting; an entire work week per year!! Well above the threshold warranting optimisation, and that doesn't even factor in distractions and context switching.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#66

Post by haizzz »

zoomablemind wrote: post #29

> "...we found that .xlf files made up almost 70% of the total number of files. These .xlf files are generated... they are never manually edited by engineers..." First thought is why not to zip/tar away all of these "convenience" files per generation and add a line into build/install script to unpack them after checkout? Additionally, add the .xlf into .gitignore to exclude them from untracked. Noone cares to diff th…

I made another comment as well though tldr is these xlf files are translations tied to texts in code so we can't simply ignore them from the repository. The changes have to be kept so that if we say revert to a certain commit, all the translations match with the texts of headers, buttons, etc...

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#67

Post by haizzz »

onion2k wrote: post #9
mherrmann wrote: post #3

Earlier quoted context omitted.

From the rest of the article, it sounds like a big chunk of these lines are from generated files. What I don't understand is why they're checking in generated files into Git.

They're using git as a cache. Having generated files stored there means they're available if they're needed (eg in CI) without needing further access controls, they're versioned, and it's a simple and understandable strategy. As the article states, most devs are set up to ignore those files so they're not much of a source of the slowness. It's a common pattern for apps that have to serve lots of different locales.

Sorry I've been a bit misleading. These xlf files aren't generated, they're just not interacted with by engineers but they're still created and edited by humans as translations. We want to keep track of them so that if we deploy a different commit, the texts and translations in other languages will match

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#68

Post by haizzz »

syspec wrote: post #34

The fact that Canva has a `Source Control team` with at least 5 people on it (going by the thanks at the bottom of the article), means they should probably try a different approach. I think it's a cool company, with a good product, but they're WAY too small to be having a "source control team" on staff. That's at least 1.2MM a year salary / benefits cost.

Author here, not everyone on the team works on git and git performance. Our team work on a variety of things that touch "code" in general like our ownership system and tooling, access control, bots and automation, etc...

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#69

Post by Maxious »

syspec wrote: post #34

The fact that Canva has a `Source Control team` with at least 5 people on it (going by the thanks at the bottom of the article), means they should probably try a different approach. I think it's a cool company, with a good product, but they're WAY too small to be having a "source control team" on staff. That's at least 1.2MM a year salary / benefits cost.

It's not exactly a great name for the team but it gets across the point - we handle developer experience from the point you want to push your code to when you merge it, so we also work on communicating with CI/CD, code review/ownership tooling, gitops bots etc. Git Performance is a big issue right now but there's no end in sight to all the other scaling problems we could work on so if it stops making sense, we can work on something else :)

disclaimer: canva staff working on (for now) source control

Post reply on HN