Live data from Hacker News

We Put Half a Million Files in One Git Repository, Here’s What We Learned

canvatechblog.com

161–170 of 270 posts

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#161

These engineering blogs are intended to promote the engineering culture of the company for recruitment process, and as a correlate an opportunity for the engineers to self-promote and boost their resume/promotion-worthiness internally. The sentiment on this thread, if it is indicative of the greater talent pool, suggests this blog post is having the complete opposite effect. I remember when Uber was proud of their th…

I checked out their careers page after reading this.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#162
post #64
post #55

Earlier quoted context omitted.

Sure, how did we ever manage to rename something without monorepos. Oh wait, maybe that's what this "versionning" thing is all about.

Right, it's "I have to send 5 PRs to 5 different repos, get them all cross merged, and then at the end it's wrong anyways so I have to start all over". Multirepo management is extremely frustrating compared to "it's all in the same folder".

Monorepo is not necessarily synced deployment, and even if it was, each deployment of a single component is usually racy with itself (as long as you're deploying to at least two nodes).

Which means that you've got to do independent backwards-compatible changes anyway, and that for anything remotely complex, you are better off with separate branches (and PR/MRs) anyway.

Monorepos mostly have a benefit for trivial changes across all repos (eg. we've decided to rename our "Shop" to "Shoppe"), where it doesn't really take much to explain with multiple repos, but is mostly a lot of tedious work to get all the PRs up and such.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#163

Earlier quoted context omitted.

How would you do refactoring over monorepo if you have sparse checkout?

CI/CD build the entire project. So if you make a breaking change in library the build will fail.

Sure, and what next? So considering that on big projects CI/CD may take up to an hour (in one of my projects it took 4 hours) the feedback loop would be great

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#164

If you don't have a couple of hundreds engineers who would write a custom file system for git (Microsoft), or who would take an existing source control system and nearly fully rewrite it (Facebook), or who would write a custom source control system from scratch (Google, Yandex, etc) DON'T USE MONOREPO. Otherwise you are risking to end up in a situation when hundreds of your engineers have to spend tens of minutes eve…

There is no point in making any argument with someone who has the conviction of their religion behind them. Followers of the monorepo religion are no different.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#165
post #56

Hey everyone, author here, the article is a bit misleading in that .xlf files aren't really generated files, they're created through our translation pipeline by real humans. I considered them generated in the sense that they're not directly worked on by engineers who have to deal with them in the repository. The content of these translation files are snapshot in time aligned with the text in our product so simply rem…

Can't those xlf files be stored in a separate repo or in an object storage and let the build system fetch them from there?

Having them under version control seems very important. Monorepo makes that easy

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#166

Earlier quoted context omitted.

The problem is not the repository, it’s having the translation data in the repo. We have the same issue at $dayjob, the repo is quite large and 80% of it is translation data. Even if they compress ridiculously well (99% last I checked) the number of translation files and the number of exports makes them the vast majority of the cost. However that structure remains a convenient nuisance, and more importantly removing…

But isnt it sort of correct to have translation data, which is text data, and there might be bug fixes, in the repo?

Which is why it commonly is there, but at the same time the translation data is kinda independent: you can run the software with no or part of the translations, as far as the software is concerned it’s usually close to configuration data or assets.

Not to mention transgressions often have their own lifecycle e.g. is common to find translatable strings which are untranslated or incorrectly translated, and want to ship updates independently from the software’s.

As such keeping the translations outside the source is also perfectly defensible.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#167

Earlier quoted context omitted.

The problem is not the repository, it’s having the translation data in the repo. We have the same issue at $dayjob, the repo is quite large and 80% of it is translation data. Even if they compress ridiculously well (99% last I checked) the number of translation files and the number of exports makes them the vast majority of the cost. However that structure remains a convenient nuisance, and more importantly removing…

But isnt it sort of correct to have translation data, which is text data, and there might be bug fixes, in the repo?

Yes

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#168
post #66

Earlier quoted context omitted.

I made another comment as well though tldr is these xlf files are translations tied to texts in code so we can't simply ignore them from the repository. The changes have to be kept so that if we say revert to a certain commit, all the translations match with the texts of headers, buttons, etc...

Could these translations be moved to another repository? Maybe they could be published as a separate NPM package that devs could install if they needed to look at the translations?

Then you get the annoying problem of having to push an update to that repo and wait for the new version before you can merge changes into the new repo which use the new version. It’s tightly coupled, so they should be co-located.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#169
post #96

Earlier quoted context omitted.

monorepo until it becomes too big -> then, splitting it into 2-3 repos, until each one also gets too big to manage...

The problem is not the repository, it’s having the translation data in the repo. We have the same issue at $dayjob, the repo is quite large and 80% of it is translation data. Even if they compress ridiculously well (99% last I checked) the number of translation files and the number of exports makes them the vast majority of the cost. However that structure remains a convenient nuisance, and more importantly removing…

> The problem is not the repository, it’s having the translation data in the repo.

Does this data change as often as the code does? If not then get it out of the repo.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#170
post #34

The fact that Canva has a `Source Control team` with at least 5 people on it (going by the thanks at the bottom of the article), means they should probably try a different approach. I think it's a cool company, with a good product, but they're WAY too small to be having a "source control team" on staff. That's at least 1.2MM a year salary / benefits cost.

That bit surprised me also. I don't have much experience with companies of this size so I have no real idea; how many people would you expect to work on source control at a company of this size? I had a look at Linkedin, they have 4,345 employees.

Usually it is considered good proportion to start with to be 10% to work on a Developer Productivity and it should increase to 15% as company grow. But regarding source control it really depends on the approach: if you are on polyrepo - then 0 is enough. If you decide to write you own source control system like Facebook or Google, then maybe couple of hundred engineers full time until the project is finished :)
Post reply on HN