Live data from Hacker News

We Put Half a Million Files in One Git Repository, Here’s What We Learned

canvatechblog.com

261–270 of 270 posts

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#261

Earlier quoted context omitted.

Of course, but can you split a monolith across multiple repos?

Sure. It'd drastically complicate the build process and CI for little benefit compared to other approaches (e.g. sparse checkout), but you could even in principle create a repo for each code directory and stitch them all together.

>It'd drastically complicate the build process and CI

This is a part of most monorepos. I figure having only a single build artifact is rare.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#262

Earlier quoted context omitted.

That wouldn't be called a "monorepo" then :D Obviously, the problems they are solving (and admitting to solving) are due to their dedication to the monorepo. With all the effort spent on working around the drawbacks, I really wonder what advantages they are seeing that make it worth their while?

Well, there is monorepo and monorepo. Git was primarily made to host code, not necessary artifacts. I would categorize those files as artifacts and in my opinion it would still be a code monorepo to have everything else on a single repo.

I think it's relatively normal to include build artifacts in a monorepo when you don't want users to have to be able to build every single one. Especially if you don't want to have to buy a license for every developer for specialist software that only needs to be used by a few people.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#263

It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…

> Given that 70% of their repo is generated files, that discussion and the tradeoffs involved don't get nearly enough attention from OP.

It's perfectly fine to use Git to track things other than sourcecode. In fact, right on the manpage, Git calls itself "the stupid content tracker".

I've been using Git with git-annex to track archival files with their associated metadata. We keep our data separate from our sourcecode, and segment our data into individual Git repositories for each collection. Git gives us many features that we would have had to build into our app in other ways (data integrity, fixity, etc), though this came with costs.

To my eye it probably would have been better for Canva to use multiple separate repositories instead of a monorepo, but I'm not them and their use-case is not mine.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#264

Earlier quoted context omitted.

Well, there is monorepo and monorepo. Git was primarily made to host code, not necessary artifacts. I would categorize those files as artifacts and in my opinion it would still be a code monorepo to have everything else on a single repo.

They are code files managed by a different set of people (translators), not generated artifacts (as they explain elsewhere in this discussion). If we are being pedantic, git was not designed to host multiple projects in a single repo (otherwise, git would have been a subdirectory in the kernel tree). But tools are made without knowing how they'll be used, and that's ok, so I wouldn't stress on what the purpose for mo…

>They are code files managed by a different set of people (translators), not generated artifacts (as they explain elsewhere in this discussion

Translations do not look like code to me. Rather human generated artifacts.

What I understand is there is a strong tie/match needed between versions of these translations files and the code itself, so I believe this is where having all in the same repo would make sense, having the translators update those file when code has been modified...

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#265

Earlier quoted context omitted.

Also plain text files are usually generated. They are arrays of 1s and 0s. No one wants to write that by hand. I think that this is not a sensible definition of a generated file. A more sensible definition is that a generated file is created automatically from some source, which is not user input (i.e. an other file). This means generated files do not need to be kept under git, as long as their source is checked in.…

For the app I work at the moment we use https://lokalise.com/ . We add translation strings to a SaaS app, and then the translation team translate them. I've written a build tool that downloads the translation JSON files from the API using the CLI, or as part of our CI process. Other teams have tools that download their language packs for different iOS and Android apps. The translations are versioned in Lokalise and w…

This still doesn't make them generated files!

You opted to keep translation files out of version control; you could also keep images there, or source files. All this stuff is the (pretty direct) output of non-deterministic human intervention.

(BTW, how do you build an old version of your application? Is lokalise able to give you the appropriate translations for a specific git commit / app version?)

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#266
post #65

> git status takes 10 seconds on average > running these commands multiple times a day reduces the total productive time engineers have every day I love the attention paid to this. Often opportunities to prioritise seemingly small efficiency gains are neglected. At 10 seconds per command, an engineer that uses git status 50 times per day spends ~10 minutes per day waiting; an entire work week per year!! Well above th…

It’s actually even worse than that I think. If something takes over a certain amount of time, then I’m more likely to go do something else while I wait, like check Hackernews. And there goes 20 minutes.

I created a `beep` alias that just does `echo "\x07\x07\x07"` which triggers the system notification sound three times.

Then if I have a command that will take a while, like a stupidly long `git status`, I do `git status && beep`.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#267

Earlier quoted context omitted.

> it takes 10 seconds to run git status People coming from the SVN world do not think that this is unusual or problematic. And unfortunately even recently I've seen SVN still in use at large legacy companies.

I don't think it's unfortunate. We use subversion for development in our team and it does everything we need it to. We looked into git and didn't find it offered any features that would significantly improve our process, but found 1000 more ways to shoot ourselves in the foot For many processes I think SVN is (and has been for many many years) been an absolutely fine method of version control

  > didn't find it offered any features that would significantly improve our process, but found 1000 more ways to shoot ourselves in the foot
You're not wrong about this.

I really like git for the cheap branching, which encourages branching and merging often. But SVN might have cheap branching now, as another commenter implies.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#268

It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…

> it takes 10 seconds to run git status People coming from the SVN world do not think that this is unusual or problematic. And unfortunately even recently I've seen SVN still in use at large legacy companies.

What is wrong with SVN?

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#269
post #185

Earlier quoted context omitted.

Hm... An interesting thought! What does it compare against though? Need to add more state to the CI? We kinda like the interface be part of the version control and having an audit chain that's part of the code.

you could put it in the pre-commit githook

That's not enforced centrally (or visible by the reviewer) so hard to trust it that everyone will remember having it.

Though even without that, I'm not sure how it'll even mechanically work.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#270
post #117

Earlier quoted context omitted.

There was pijul. Which allowed partial pulls iirc. I haven't used it, so I can't really recommended it.

I tried pijul [0] last time this conversation came up and despite its claims it is not fast. [0] https://news.ycombinator.com/item?id=29992875

Just seeing this now. If you look at the conversation, this was fixed within a few hours.
Post reply on HN