Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Put Half a Million Files in One Git Repository, Here’s What We Learned

canvatechblog.com

Post reply on HN
Print view View original

21–30 of 270 posts

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#21

Post by arinlen »

robertlagrant wrote: post #12
onion2k wrote: post #9

Earlier quoted context omitted.

They're using git as a cache. Having generated files stored there means they're available if they're needed (eg in CI) without needing further access controls, they're versioned, and it's a simple and understandable strategy. As the article states, most devs are set up to ignore those files so they're not much of a source of the slowness. It's a common pattern for apps that have to serve lots of different locales.

It's still surprising to have any generated things there. E.g. you could make the same case for keeping built binaries in Git as well. Is there a reason why that type of file couldn't be better place into an artifact repository, or just generated and consumed in CI as part of generating a final build output?

> It's still surprising to have any generated things there. E.g. you could make the same case for keeping built binaries in Git as well.

This is not surprising at all. In fact, it's quite standard to commit string translations. Just because you can run the code generation/string replacement step as part of the build that does not mean it's a good idea to generate everything from scratch at every single build.

String translations hardly change once they are introduced, running the build step takes significant amounts of time, and if anything fails then your product can break in critical and hard to notice ways.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#23

Post by arinlen »

rsanheim wrote: post #20

It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…

> Didn't someone on the team say "hey, it takes 10 seconds to run git status, can we move this junk out and do this another way??"

Why do you assume they didn't?

Just because they arrived at a different conclusion than you that doesn't mean they didn't thought about it. I might very well mean you did not considered the tradeoffs they had to take into account, mainly because you're out of the loop.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#24

Post by kadoban »

fomine3 wrote: post #11
onion2k wrote: post #9

Earlier quoted context omitted.

They're using git as a cache. Having generated files stored there means they're available if they're needed (eg in CI) without needing further access controls, they're versioned, and it's a simple and understandable strategy. As the article states, most devs are set up to ignore those files so they're not much of a source of the slowness. It's a common pattern for apps that have to serve lots of different locales.

I don't think it's good idea to store cache in Git. Any file remains forever in the repository after once committed. Local/remote repository become unnecessary big.

The bigger issues for me are it makes history impossible to read (every change is hidden in an avalanche of crap), merges are a mess (you definitely want to spend forever merging autogened files, right?), PR reviews are annoying, etc.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#25

Post by alexdowad »

Gigachad wrote: post #2

>just under 60 million lines of code in 2022. I'm always surprised at the fact that almost every product has more lines of code than the entire Linux repo. The scale of these products is astounding.

The natural tendency in almost any software is to keep adding and adding, while rarely throwing anything out. More features, more code, more supported platforms, more supported languages, more this, more that, more, more, more.

If it was a physical product, you couldn't keep making it bigger and more complex ad infinitum, because making a physical thing bigger takes more material, and bounded physical resources would be consumed. With software, it's all just bits, and computers can hold a lot of bits.

This leads to bigger problems than just git running slowly.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#26

Post by Gigachad »

kadoban wrote: post #24
fomine3 wrote: post #11

Earlier quoted context omitted.

I don't think it's good idea to store cache in Git. Any file remains forever in the repository after once committed. Local/remote repository become unnecessary big.

The bigger issues for me are it makes history impossible to read (every change is hidden in an avalanche of crap), merges are a mess (you definitely want to spend forever merging autogened files, right?), PR reviews are annoying, etc.

Depends how much generated stuff is there. We have our graphql schema in git even though its auto generated via a library. But its useful in PRs to see exactly how the schema changed as a result of the root change.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#27

Post by kadoban »

iqkznnft wrote: post #18
dymk wrote: post #16

Earlier quoted context omitted.

git log ;)

I believe git log is limited by the number of lines on the screen...

It may operate in a streaming fashion if it's going to a pager, but naw it'll output however much you want.

Technically it has modes which need to scan all commits in the current branch as well, like the ones that grep for certain changes.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#28

Post by barbs »

I feel like I've read about several big companies using monorepos, but I've never understood why. It feels like the source-control equivalent of writing your code in one big file.

Does anyone have any good resources for why and how best to implement a monorepo?

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#29

Post by zoomablemind »

> "...we found that .xlf files made up almost 70% of the total number of files. These .xlf files are generated... they are never manually edited by engineers..."

First thought is why not to zip/tar away all of these "convenience" files per generation and add a line into build/install script to unpack them after checkout?

Additionally, add the .xlf into .gitignore to exclude them from untracked.

Noone cares to diff them as long as their contents is consistent with the checkout. Text compresses quite efficiently so this should not introduce any unreasonable build/install delays.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#30

Post by robertlagrant »

arinlen wrote: post #21
robertlagrant wrote: post #12

Earlier quoted context omitted.

It's still surprising to have any generated things there. E.g. you could make the same case for keeping built binaries in Git as well. Is there a reason why that type of file couldn't be better place into an artifact repository, or just generated and consumed in CI as part of generating a final build output?

> It's still surprising to have any generated things there. E.g. you could make the same case for keeping built binaries in Git as well. This is not surprising at all. In fact, it's quite standard to commit string translations. Just because you can run the code generation/string replacement step as part of the build that does not mean it's a good idea to generate everything from scratch at every single build. String…

I'm not saying don't have the translations at all. I'm saying: 1) caching things in git in general is a bad idea; why is it not in this case? 2) these are not - to my understanding - the raw resource files, but rather machine-generated intermediate files. This is why it's about caching, rather than minimal source files.

Additionally, to respond to your comment, if string translations don't change much then it may be possible to push them out as an internal 3rd-party library, and then they're even quicker to build.

Post reply on HN