Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Put Half a Million Files in One Git Repository, Here’s What We Learned

canvatechblog.com

Post reply on HN
Print view View original

251–260 of 270 posts

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#251

Post by 37 »

Bluecobra wrote: post #17
37 wrote: post #6

Pretty interesting.... I don't know that much about git, but still fun to read. I guess the main takeaway is don't put all your eggs in one basket? Although it kinda seems like they are going to stick with the monorepo, ("Here’s how we solve them at Canva") Also, I looked up .xlf files and I still don't understand. It's xml, that part makes sense, but it's basically a config file? To tell what process to read which f…

Putting my rusty sysadmin hat on, I would wager a guess that if it keeps on growing exponentially something will break. For example, hitting some hard limit like maximum number of files possible in a file system. Seems like you’re playing with fire to me.

I would think the same... and from reading the comments in this thread, it seems like a pretty spicy topic.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#252

Post by sally1620 »

haizzz wrote: post #56

Hey everyone, author here, the article is a bit misleading in that .xlf files aren't really generated files, they're created through our translation pipeline by real humans. I considered them generated in the sense that they're not directly worked on by engineers who have to deal with them in the repository. The content of these translation files are snapshot in time aligned with the text in our product so simply rem…

If it is output of some other tool, then it is a build artifact and there is no need to check it in.

I have this argument with people all the time and the conclusion is always like: "it is too hard to integrate the generator with the build system so we check them in".

The big problem with generated files is merge conflicts. How do you resolve a merge conflict on generated files. especially if they are binary.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#253

Post by necovek »

rtpg wrote: post #250
necovek wrote: post #162

Earlier quoted context omitted.

Monorepo is not necessarily synced deployment, and even if it was, each deployment of a single component is usually racy with itself (as long as you're deploying to at least two nodes). Which means that you've got to do independent backwards-compatible changes anyway, and that for anything remotely complex, you are better off with separate branches (and PR/MRs) anyway. Monorepos mostly have a benefit for trivial chan…

I think that when you have large enough systems that works. I do not believe that "microservice" is the right size for repo splits. Sometimes you have to ship a feature. Shipping that requires changing 3 parts of your app. A lot of times that _entire_ set of changes is less than 100 lines of code. Having a full vision of what is being accomplished across your system in one go is very helpful for reviewing code! It ju…

TBH, I am not a fan of frontend/backend split either: ideally, you'd still be splitting per component/service, so frontend and backend for a single thing could live in the same place: you get the benefit of seeing the API in use with each PR, without the costs of monorepo otherwise.

Most projects start out as monoliths (which is good) and splitting up on this axis is unfortunately very hard/costly.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#254

Post by necovek »

prmoustache wrote: post #242
necovek wrote: post #152

Earlier quoted context omitted.

That wouldn't be called a "monorepo" then :D Obviously, the problems they are solving (and admitting to solving) are due to their dedication to the monorepo. With all the effort spent on working around the drawbacks, I really wonder what advantages they are seeing that make it worth their while?

Well, there is monorepo and monorepo. Git was primarily made to host code, not necessary artifacts. I would categorize those files as artifacts and in my opinion it would still be a code monorepo to have everything else on a single repo.

They are code files managed by a different set of people (translators), not generated artifacts (as they explain elsewhere in this discussion).

If we are being pedantic, git was not designed to host multiple projects in a single repo (otherwise, git would have been a subdirectory in the kernel tree). But tools are made without knowing how they'll be used, and that's ok, so I wouldn't stress on what the purpose for monorepo was, but how it's used and what value it brings.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#255

Post by dotancohen »

capitainenemo wrote: post #171
dotancohen wrote: post #115

Earlier quoted context omitted.

My experience is that anything dealing with a branch, especially but not exclusively creating branches, is very slow in SVN for a repo of any real size, basically anything with a framework. I do not remember if "stat" was particularly slow, but SVN in general is slow.

Huh. 10 gigabyte svn repo at work spanning about 40 projects.. Creating branches is virtually instantaneous. It's just a copy which is a free operation (just a link). Curious as to why it would be slow for you. svn cp https:/ /server/svn/trunk/project/ https:/ /server/svn/branches/project/ticket -m "making a branch here" svn status, even for an entire repo checkout (which is not common) is also fast. And yeah, it has…

  > Creating branches is virtually instantaneous. It's just a copy which is a free operation (just a link).
Copy is not a "free" operation, but a symlink is close to "free" if you're measuring disk space.

What version SVN are you using? I'm certain that older SVN versions would actually copy the entire project's files, not symlinks but real copies. That would take forever and running out of disk space was a real concern.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#256

Post by maccard »

jeffbee wrote: post #225
maccard wrote: post #223

Earlier quoted context omitted.

In my defence, I elaborated in parentheses immediately after saying it. It's _very_ centralised and online only. There's no concept of any local work/branches so all prototypes are checked in (or in my experience people manually shelve things and juggle shelves around). Perforce's branches are a disaster and should have been deprecated years ago and replaced with streams a decade ago. Branches are _incredibly_ slow,…

I used Perforce for a decade and did not create a branch, not even once. People who want to branch in Perforce are often trying to bring a git mindset to a different tool. With a trunk-based edit/sync/submit workflow where you have different p4 clients for your different projects (what you would use various branches for in git) you need not branch.

> I used Perforce for a decade and did not create a branch, not even once.

That's because P4's branches are a pale imitation of what branches can be (and to be fair to P4, branches long predate git and they can't exactly up and change the behaviour, however there's really no excuse for what they did with streams. They bolted a loosely enforced hierarchy onto the existing branch system, created a split in the tooling, and shipped something that has as many footguns as problems it solves.)

> People who want to branch in Perforce are often trying to bring a git mindset to a different tool. With a trunk-based edit/sync/submit workflow where you have different p4 clients for your different projects (what you would use various branches for in git) you need not branch.

"often" is a very nebulous adjective, and a naive view of what git branches do. Perforce and a task branch based workflow is a terrible idea, yes. If you want to do the PR based flow that github and gitlab encourage, you're going to have a bad time. P4's shelves are an excellent tool, but they encourage ad-hoc and self managed version control. Shelves to bring changes across streams (if you're using them), Shelves to share a WIP or a quick change with someone else, and iterating back and forth with shelves with v1 v2 v3 etc in them, shelves for temporary debugging code/non prod features/work in progress feature that's ticking along in the background.

Again, I'm not saying perforce has no place, but git's branches are a force multipler, even with trunk based development.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#257

Post by cnity »

tasn wrote: post #185
cnity wrote: post #120

Earlier quoted context omitted.

One alternative that enables you to keep generated files out but still feel like there's an explicit human check in place is to add a gated confirmation step in CI to confirm that the changes to the generated spec match expectations. Something like: "This change will result in the following new API endpoints: ... do you wish to continue?"

Hm... An interesting thought! What does it compare against though? Need to add more state to the CI? We kinda like the interface be part of the version control and having an audit chain that's part of the code.

Yeah either you'd have to maintain the generated spec as a versioned artifact hosted wherever is most convenient for you, or the CI could actually generate the before and after specs based on the PR diff. If the calculation of the spec is computationally expensive (it shouldn't be) then the latter approach could be a problem.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#258

Post by capitainenemo »

dotancohen wrote: post #255
capitainenemo wrote: post #171

Earlier quoted context omitted.

Huh. 10 gigabyte svn repo at work spanning about 40 projects.. Creating branches is virtually instantaneous. It's just a copy which is a free operation (just a link). Curious as to why it would be slow for you. svn cp https:/ /server/svn/trunk/project/ https:/ /server/svn/branches/project/ticket -m "making a branch here" svn status, even for an entire repo checkout (which is not common) is also fast. And yeah, it has…

> Creating branches is virtually instantaneous. It's just a copy which is a free operation (just a link). Copy is not a "free" operation, but a symlink is close to "free" if you're measuring disk space. What version SVN are you using? I'm certain that older SVN versions would actually copy the entire project's files, not symlinks but real copies. That would take forever and running out of disk space was a real concer…

An svn copy is just a link. It has always been a "free" operation (and yes, the analogy would be to a symlink). I'm not aware of any version of svn that behaved the way you described and I've used it for a couple of decades.

I can perhaps imagine a large repo plus a broken svn client requiring checking out unneeded portions of trees to do a copy, but no client I've used works like that.

Hm. Another theory. Perhaps someone who knew nothing about svn and was using TortoiseSVN's Windows file manager integration was doing a Windows file manager copy, then checking that in as a "branch" with the only link being the commit message instead of using svn's copy which is free and properly links content. That would indeed be an expensive operation, and the wrong thing to do.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#259

Post by pronik »

Somehow, it seems so backwards. We make our repositories more accessible for automated tools and less accessible for people and instead of investing coding time into better tools, we invest it into local workarounds at every single developer's machine. This is probably not a general conclusion on monorepos, but the notion of "let's put auto-generated stuff into version control" sounds like the point in time where it all went awry. Yes, there is ClearCase where this is the way to do it, Perforce probably does this too and there are areas where you are supposed to be archiving your generated binaries with your code and scripts for later audits, but I'd argue most people shouldn't be doing this, since there are ways to separate source and release artifacts once and for all in a way to make both bots and people happy without locally-brewed scripts for checking out code.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#260

Post by rtpg »

necovek wrote: post #253
rtpg wrote: post #250

Earlier quoted context omitted.

I think that when you have large enough systems that works. I do not believe that "microservice" is the right size for repo splits. Sometimes you have to ship a feature. Shipping that requires changing 3 parts of your app. A lot of times that _entire_ set of changes is less than 100 lines of code. Having a full vision of what is being accomplished across your system in one go is very helpful for reviewing code! It ju…

TBH, I am not a fan of frontend/backend split either: ideally, you'd still be splitting per component/service, so frontend and backend for a single thing could live in the same place: you get the benefit of seeing the API in use with each PR, without the costs of monorepo otherwise. Most projects start out as monoliths (which is good) and splitting up on this axis is unfortunately very hard/costly.

This is why I'm a fan of tools like Bazel, where you can still get most of the tooling benefits from a single repo, but get testing speed benefits (and, if you roll that way, the design benefits from the separation) of a multirepo setup.

Unfortunately it's hard for me to recommend Bazel, it's such an uphill climb to get things working within that system.

Post reply on HN