It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…
We Put Half a Million Files in One Git Repository, Here’s What We Learned
181–190 of 270 posts
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#182Earlier quoted context omitted.
hey hey author here, xlf files are translations that are coupled with the texts we set in the code so they're not really generated I admit that was misleading. What I wanted to get across is they're not touched directly by engineers but they're still created through our translation pipeline where real humans translate them
How do you split your XLIFF files? Does each project get one big one and the proliferation is simply due to number of languages, or do you have a more granular split (eg. if you've got one component, it will have dozens of XLIFF files for every language, instead of one per language)? By the numbers you mention, 70% of the files make a ratio of code files to translation files 1-3, so unless you only support 3-5 langua…
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#183Earlier quoted context omitted.
The problem is not the repository, it’s having the translation data in the repo. We have the same issue at $dayjob, the repo is quite large and 80% of it is translation data. Even if they compress ridiculously well (99% last I checked) the number of translation files and the number of exports makes them the vast majority of the cost. However that structure remains a convenient nuisance, and more importantly removing…
> The problem is not the repository, it’s having the translation data in the repo. Does this data change as often as the code does? If not then get it out of the repo.
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#184Earlier quoted context omitted.
CI/CD build the entire project. So if you make a breaking change in library the build will fail.
Sure, and what next? So considering that on big projects CI/CD may take up to an hour (in one of my projects it took 4 hours) the feedback loop would be great
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#185Earlier quoted context omitted.
One of my biggest pet peeves: tracking generated files in version control. The only exception is our generated OpenAPI spec, because we want people to be explicit about modifying the API, and have a CI task that verifies that the API and OpenAPI spec match.
One alternative that enables you to keep generated files out but still feel like there's an explicit human check in place is to add a gated confirmation step in CI to confirm that the changes to the generated spec match expectations. Something like: "This change will result in the following new API endpoints: ... do you wish to continue?"
What does it compare against though? Need to add more state to the CI? We kinda like the interface be part of the version control and having an audit chain that's part of the code.
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#186It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#187Earlier quoted context omitted.
> The main drawback is scalability of the tools involved like git. And if you can employ enough engineers to break git, you can probably afford a team to work on scaling git.
Git staring to break at 200-300 engineers pushing into it. Scaling git would take 200 more :)
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#188Earlier quoted context omitted.
Author here, not everyone on the team works on git and git performance. Our team work on a variety of things that touch "code" in general like our ownership system and tooling, access control, bots and automation, etc...
Sorry, there is a quote from the article: "Kudos to other members of the Source Control team — Alex Sadleir, Wesley Li, Adam Murray, Matthew Chhoeu — who work on improving git performance at Canva" So I assume 5 people working specifically on Git performance ;-P
That's the entire Source Control team (of 5 people) today. Among other things, they work on improving Git performance ;)
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#189Earlier quoted context omitted.
CI/CD build the entire project. So if you make a breaking change in library the build will fail.
Sure, and what next? So considering that on big projects CI/CD may take up to an hour (in one of my projects it took 4 hours) the feedback loop would be great
No, it may not. Perhaps occasionally it does. That is a bug that you must fix - a pipeline that takes even 30 minutes is horrifically slow.
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#190These engineering blogs are intended to promote the engineering culture of the company for recruitment process, and as a correlate an opportunity for the engineers to self-promote and boost their resume/promotion-worthiness internally. The sentiment on this thread, if it is indicative of the greater talent pool, suggests this blog post is having the complete opposite effect. I remember when Uber was proud of their th…