Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Put Half a Million Files in One Git Repository, Here’s What We Learned

canvatechblog.com

Post reply on HN
Print view View original

11–20 of 270 posts

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#11

Post by fomine3 »

onion2k wrote: post #9
mherrmann wrote: post #3

Earlier quoted context omitted.

From the rest of the article, it sounds like a big chunk of these lines are from generated files. What I don't understand is why they're checking in generated files into Git.

They're using git as a cache. Having generated files stored there means they're available if they're needed (eg in CI) without needing further access controls, they're versioned, and it's a simple and understandable strategy. As the article states, most devs are set up to ignore those files so they're not much of a source of the slowness. It's a common pattern for apps that have to serve lots of different locales.

I don't think it's good idea to store cache in Git. Any file remains forever in the repository after once committed. Local/remote repository become unnecessary big.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#12

Post by robertlagrant »

onion2k wrote: post #9
mherrmann wrote: post #3

Earlier quoted context omitted.

From the rest of the article, it sounds like a big chunk of these lines are from generated files. What I don't understand is why they're checking in generated files into Git.

They're using git as a cache. Having generated files stored there means they're available if they're needed (eg in CI) without needing further access controls, they're versioned, and it's a simple and understandable strategy. As the article states, most devs are set up to ignore those files so they're not much of a source of the slowness. It's a common pattern for apps that have to serve lots of different locales.

It's still surprising to have any generated things there. E.g. you could make the same case for keeping built binaries in Git as well.

Is there a reason why that type of file couldn't be better place into an artifact repository, or just generated and consumed in CI as part of generating a final build output?

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#13

Post by epage »

mherrmann wrote: post #3
Gigachad wrote: post #2

>just under 60 million lines of code in 2022. I'm always surprised at the fact that almost every product has more lines of code than the entire Linux repo. The scale of these products is astounding.

From the rest of the article, it sounds like a big chunk of these lines are from generated files. What I don't understand is why they're checking in generated files into Git.

I've been switching a lot of generated files to being checked in (with CI verifying they haven't drifted from the source). The primary motivation has been performance. For example, in Rust code, it means I don't need to foist the code-gen process and all the dependencies needed for it on dependent crates. I've seen this play out similar in other build systems and circumstances. The key is the data needs to be independent of other factors (like the system doing the generation) and the rate of change of the code-gen source and generator has to be relatively low.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#15

Post by kstenerud »

robertlagrant wrote: post #12
onion2k wrote: post #9

Earlier quoted context omitted.

They're using git as a cache. Having generated files stored there means they're available if they're needed (eg in CI) without needing further access controls, they're versioned, and it's a simple and understandable strategy. As the article states, most devs are set up to ignore those files so they're not much of a source of the slowness. It's a common pattern for apps that have to serve lots of different locales.

It's still surprising to have any generated things there. E.g. you could make the same case for keeping built binaries in Git as well. Is there a reason why that type of file couldn't be better place into an artifact repository, or just generated and consumed in CI as part of generating a final build output?

> place into an artifact repository

This adds yet another moving part to the system, and another place things can go wrong.

> generated and consumed in CI as part of generating a final build output

This can get quite slow, and on larger projects you have to expend a lot of effort to keep build times reasonable.

Also, if you're serving a library for public consumption, you generally don't want to add the burden of extra build steps for the user to follow before they can use it. If it can all be automated to the point of invisibility to the user that's fine, but often it can't.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#17

Post by Bluecobra »

37 wrote: post #6

Pretty interesting.... I don't know that much about git, but still fun to read. I guess the main takeaway is don't put all your eggs in one basket? Although it kinda seems like they are going to stick with the monorepo, ("Here’s how we solve them at Canva") Also, I looked up .xlf files and I still don't understand. It's xml, that part makes sense, but it's basically a config file? To tell what process to read which f…

Putting my rusty sysadmin hat on, I would wager a guess that if it keeps on growing exponentially something will break. For example, hitting some hard limit like maximum number of files possible in a file system. Seems like you’re playing with fire to me.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#18

Post by iqkznnft »

dymk wrote: post #16
nindalf wrote: post #14

git status scales with the number of files in the repo. Ask HN - Are there any common git operations that scale with the number of commits?

git log ;)

I believe git log is limited by the number of lines on the screen...

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#19

Post by duskwuff »

nindalf wrote: post #14

git status scales with the number of files in the repo. Ask HN - Are there any common git operations that scale with the number of commits?

Repository maintenance operations like "git gc" scale with the total number of objects in the repo.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#20

Post by rsanheim »

It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool.

To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise.

But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure, we can throw all these generated files into git!". Didn't someone on the team say "hey, it takes 10 seconds to run git status, can we move this junk out and do this another way??"

Given that 70% of their repo is generated files, that discussion and the tradeoffs involved don't get nearly enough attention from OP.

Post reply on HN