Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Put Half a Million Files in One Git Repository, Here’s What We Learned

canvatechblog.com

Post reply on HN
Print view View original

141–150 of 270 posts

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#141

Post by danielheath »

kanu666 wrote: post #139

If you don't have a couple of hundreds engineers who would write a custom file system for git (Microsoft), or who would take an existing source control system and nearly fully rewrite it (Facebook), or who would write a custom source control system from scratch (Google, Yandex, etc) DON'T USE MONOREPO. Otherwise you are risking to end up in a situation when hundreds of your engineers have to spend tens of minutes eve…

I mean, if you have hundreds of engineers, that might be sound advice; if you have 20, you’ll probably be fine.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#142

Post by rossmohax »

eatonphil wrote: post #130
SenHeng wrote: post #33

Earlier quoted context omitted.

Remember back when people recommended commiting node_modules into git?

Based on how brittle Github actions is I'd be ready to commit node_modules except for that I'm building cross-platform software with native dependencies.

`npm rebuild` should rebuild native code in your committed node_modules

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#143

Post by kanu666 »

danielheath wrote: post #141
kanu666 wrote: post #139

If you don't have a couple of hundreds engineers who would write a custom file system for git (Microsoft), or who would take an existing source control system and nearly fully rewrite it (Facebook), or who would write a custom source control system from scratch (Google, Yandex, etc) DON'T USE MONOREPO. Otherwise you are risking to end up in a situation when hundreds of your engineers have to spend tens of minutes eve…

I mean, if you have hundreds of engineers, that might be sound advice; if you have 20, you’ll probably be fine.

Of course, you shouldn't start a project from 10 repos. But the downside is while you live in a single repo all the tooling get tuned to work with a single repo. And then there is no way to go multi-repo when you need to.

So advice would be: start small but think big :)

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#145

Post by Beltiras »

dotancohen wrote: post #94
anitil wrote: post #76

Earlier quoted context omitted.

... are we no longer doing 'works on my machine' ?

Ostensibly if it works in Alice's Docker instance, it will run in Bob's Docker instance too.

Except for weird Docker edge cases (extremely rare, but does happen).

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#146

Post by IshKebab »

urban_winter wrote: post #39

What's the value of a monorepo if developers only ever check out a small subset of it? Wouldn't multiple repos allow greater scale without any practical reduction in utility? For example, all the localisation files could live in a separate project (if we accept the need to commit them at all). Some tools would be needed to deal with the inevitable problem that developer working sets would not align with project bound…

The point is that developers can check out bigger subsets if they need.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#147

Post by necovek »

ElemenoPicuares wrote: post #49
rsanheim wrote: post #20

It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…

Canva gives users without design tools expertise the ability to make fairly polished looking graphics with a super easy and intuitive interface. (As a designer, I can assure you that polished looking is not the same thing as designed.) It’s a very popular service, so they’re dealing with huge scale. Intuitive interfaces often come with complex mechanisms and lots of assets, and they have clients on every major mobile…

They have obviously invested a lot over time into streamlining their build process: so much so that they're putting an article about it.

All of the problems they are having are basically due to their use of a monorepo: they do explain that they made the decision early, but I wonder what are the advantages over multiple repos they are seeing that it was worth it all this trouble?

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#148

Post by necovek »

haizzz wrote: post #61
rsanheim wrote: post #20

It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…

hey hey author here, xlf files are translations that are coupled with the texts we set in the code so they're not really generated I admit that was misleading. What I wanted to get across is they're not touched directly by engineers but they're still created through our translation pipeline where real humans translate them

How do you split your XLIFF files? Does each project get one big one and the proliferation is simply due to number of languages, or do you have a more granular split (eg. if you've got one component, it will have dozens of XLIFF files for every language, instead of one per language)?

By the numbers you mention, 70% of the files make a ratio of code files to translation files 1-3, so unless you only support 3-5 languages, it's definitely not one XLIFF file per source file, so I wonder at what granularity it is?

(My experience is mostly with localizations using GNU gettext tools, and you usually do a small-finite-number of PO files per project per language, where that small-finite number is exactly one for like 99% of projects)

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#149

Post by ChrisRR »

dotancohen wrote: post #91
rsanheim wrote: post #20

It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…

> it takes 10 seconds to run git status People coming from the SVN world do not think that this is unusual or problematic. And unfortunately even recently I've seen SVN still in use at large legacy companies.

I don't think it's unfortunate. We use subversion for development in our team and it does everything we need it to. We looked into git and didn't find it offered any features that would significantly improve our process, but found 1000 more ways to shoot ourselves in the foot

For many processes I think SVN is (and has been for many many years) been an absolutely fine method of version control

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#150

Post by necovek »

Beltiras wrote: post #145
dotancohen wrote: post #94

Earlier quoted context omitted.

Ostensibly if it works in Alice's Docker instance, it will run in Bob's Docker instance too.

Except for weird Docker edge cases (extremely rare, but does happen).

Not rare at all.

Docker is a congregation of technologies held together with duct tape and glue.

Eg. permissions handling is completely different on Macs with Docker Desktop from the Linux dockerd stuff: on Macs, it automatically translates user ownership for any "mounted" local storage (like your code repository), whereas on Linux user IDs and host system permissions are preserved. Have some developers use Macs and others use Linux, and attempt to do proper permissions setup (otherwise known as "don't run as root"), and you are looking for some fun debugging sessions.

Post reply on HN