Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Put Half a Million Files in One Git Repository, Here’s What We Learned

canvatechblog.com

Post reply on HN
Print view View original

91–100 of 270 posts

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#91

Post by dotancohen »

rsanheim wrote: post #20

It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…

  > it takes 10 seconds to run git status
People coming from the SVN world do not think that this is unusual or problematic. And unfortunately even recently I've seen SVN still in use at large legacy companies.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#92

Post by coding123 »

anitil wrote: post #76
SenHeng wrote: post #53

Earlier quoted context omitted.

The days before lock files were a thing and 'it works on my machine!' was rampant.

... are we no longer doing 'works on my machine' ?

At companies that don't check in node_modules, build folders, and are using standard packaging tooling like maven or yarn or npm or what-have-you. Yes, I haven't experienced that in like 15 years.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#93

Post by gustavohenke »

jokethrowaway wrote: post #37
AlfeG wrote: post #31

Earlier quoted context omitted.

We check-in some generated CSS files. That are generated by external theme cli. Just to be sure, that after version update we can track all changes in the generated CSS files.

If they're generated you can just re-generate from source every time you need to track changes. You're using git as a cache. You don't need to version a cache.

Regenerating certain things might be fast, but some might not be. Hundreds of engineers pushing code and having to wait for these to be regenerated both locally and on CI means that caching is quite cheap after all.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#94

Post by dotancohen »

anitil wrote: post #76
SenHeng wrote: post #53

Earlier quoted context omitted.

The days before lock files were a thing and 'it works on my machine!' was rampant.

... are we no longer doing 'works on my machine' ?

Ostensibly if it works in Alice's Docker instance, it will run in Bob's Docker instance too.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#96

Post by roflulz »

andi999 wrote: post #83
the_biot wrote: post #73

They made a bad design decision 10 years ago, have been fighting the fallout for years, and will be doing so forever and ever because things will only ever grow. They wrote a blog post on how clever they think all their workarounds are, at least one of which involves sparse-checkout -- which is perilously close to chopping up your monorepo into several, while still pretending monorepo is fine. I feel like somebody's…

What would be a better structure?

monorepo until it becomes too big -> then, splitting it into 2-3 repos, until each one also gets too big to manage...

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#98

Post by maccard »

I appreciate this post. It's nice to see that there are other teams that feel some of the pain points of git, (and unsurprising that most of the responses are "you're holding it wrong"). The fact is that git doesn't scale to _very large_ repos, We've seen it time and time again, but there isn't really a great alternative. Perforce is.... Perforce (centralized, very expensive to license, branches are incredibly expensive and streams still feel like a band aid even years and years later). PlasticSCM (which we use at work) is fine, but closed source, mildly expensive, and has a terrible UX

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#99

Post by vinay_ys »

nsoonhui wrote: post #44
urban_winter wrote: post #39

What's the value of a monorepo if developers only ever check out a small subset of it? Wouldn't multiple repos allow greater scale without any practical reduction in utility? For example, all the localisation files could live in a separate project (if we accept the need to commit them at all). Some tools would be needed to deal with the inevitable problem that developer working sets would not align with project bound…

One of the benefits of monorepo is refactoring. You can just apply a renaming command across all of the files in the solution and all the related names are properly updated. Not that easy to get this to work on multiple repos.

All developer activities related to code – refactoring, editing, building etc all happen in a developer's workspace. A workspace can be composed of multiple git repo checkouts. An infrequent activity like renaming a lot of files can be done with minor inconveniences even if they are spread across the workspace in different repos.

Only the code that is closely related – read/modified/built together frequently – should live in the same repo. If two pieces of code that don't have much to do with each other (that is, they are not read/modified/built by a developer in a single developer workflow frequently) live in the same repo, then they are just being a burden to the overall development lifecycle of devs who work on those disjoint sets of code.

The unrelated code in the same repo is a distraction to the developers who checkout that code as it costs storage space, iops, cpu cycles and network bandwidth to lug that code around, load/index in IDEs, track changes, build and discard dependency graphs by build/dependency systems etc. Then, to deal with these issues more complexities are incurred. Instead, it is better to optimise for the common case and deal with the complexity only for the rare cases.

Post reply on HN