Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Put Half a Million Files in One Git Repository, Here’s What We Learned

canvatechblog.com

Post reply on HN
Print view View original

201–210 of 270 posts

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#201

Post by ptha »

robertlagrant wrote: post #112
arinlen wrote: post #40

Earlier quoted context omitted.

> I'm not saying don't have the translations at all. I'm saying: 1) caching things in git in general is a bad idea (...) You're missing the point. Storing translated files is caching things in git, and it is not a bad idea. It's a standard practice that saves your neck. You either place faith on a build step working deterministically when it was not designed to work like that, or you track your generated files in you…

> Just because you place faith in some build step to always work deterministically that does not mean you are following a good practice and everyone else around you is wrong. You're also doing that everywhere else. How do you think anything works? Why do you think Git is deterministic somehow? Why more so than including some files in a build?

Just an example, I had the non-deterministic case using JAXB to generate java classes from XSD Schema files. Running an ANT jaxb task to generate the classes from the same schema files would generate different class files each time. The class files were functionally the same, however it would reorder methods, the order of the variable definitions etc. Possibly due to some internal code using a Map vs List, so order was not guaranteed. In our case the schema files were in Source Control, the Java/Class files were not, the Java/Class files were generated by the build, packaged to a jar and published to our artifact repository.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#202

Post by hcarvalhoalves »

Thorentis wrote: post #80

I don't understand the logic of combining microservices with a monorepo. The whole point of microservices is that you don't care what is beyond the external contract of the service. Who cares how each individual team decides to name their stuff. Why do microservice teams need to care about having every single service checked out? Who or what would be bulk applying changes to all services? This is madness.

Microservices in a monorepo is the least friction path towards a distributed ball of mud.

Whether this is a pattern or anti-pattern depends if you want a single engineer being able to change the entire architecture to “just ship it” or you if you value conceptual integrity more.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#203

Post by maccard »

sausagefeet wrote: post #196
urban_winter wrote: post #39

What's the value of a monorepo if developers only ever check out a small subset of it? Wouldn't multiple repos allow greater scale without any practical reduction in utility? For example, all the localisation files could live in a separate project (if we accept the need to commit them at all). Some tools would be needed to deal with the inevitable problem that developer working sets would not align with project bound…

If you can checkout your monorepo as if it's multiple repos, but then also check it out as a monorepo when you want it, that seems to me more utility than splitting into multiple repos, then you can never check it out as a monorepo.

In a world where submodules worked (side note: We use PlasticSCM which has xlinks [0] which are substantially better than submodules, but Plastic itself has it's own set of problems), you could have each "subrepo" as an independent repo, and then have a monorepo comprised entirely of submodules.

If submodules worked.

[0] https://www.plasticscm.com/documentation/xlinks/plastic-scm-...

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#204

Post by throwaway892238 »

Galanwe wrote: post #55
nsoonhui wrote: post #44

Earlier quoted context omitted.

One of the benefits of monorepo is refactoring. You can just apply a renaming command across all of the files in the solution and all the related names are properly updated. Not that easy to get this to work on multiple repos.

Sure, how did we ever manage to rename something without monorepos. Oh wait, maybe that's what this "versionning" thing is all about.

It takes X*N amount of work to merge change X across N repos. 1 repo just takes X.

Then there's version management. Do all your repos use the same versioning scheme? "They should", but in the real world, they sometimes don't. Whereas if you only have 1 repo, you are guaranteed 1 versioning scheme, and 1 version for everything.

How do you know which version of what correlates to what else? With N repos, do you maintain a DAG which maps every version of every repo to every other repo, so when you revert a change from 1 repo, you can go back in history and revert all the other repos to their versions from the same time? Most people do not, so reverting a change leads to regressions. With a multirepo, there only is one version of everything and everything is in lock-step with everything else, so you can either revert a single change, or do an entire rollback of everything, with 1 command.

How do you deploy changes? If each repo has an independent deployment process (if your repos even have a deployment process that isn't just waiting for Phil to do something from his laptop), are you going to deploy each one at a time, or all at once? What if one of them fails? How do you find out when they've all passed and deployed successfully? Pull up 5 different CI results in your browser every couple hours, and when one fails, go ask that team to fix something? If you only have 1 repo, there is 1 deploy process (though different jobs) and merging triggers exactly what needs to happen in exactly the right order.

The reason people use multirepos is they don't want to build a fully automated CI/CD pipeline. They don't want to add tests and quality gates, they don't want to set up a deployment system that can handle all the code. They just want to keep their own snowflake repo of code and deal with everything via manual toil. But at scale (not "Google scale", but just "We have 6 different teams working on one product" scale) it becomes incredibly wasteful to have all these divergent processes and not enough automation. Multirepo wastes time, adds complexity, and introduces errors.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#205

Post by jeffhuys »

haizzz wrote: post #66
zoomablemind wrote: post #29

> "...we found that .xlf files made up almost 70% of the total number of files. These .xlf files are generated... they are never manually edited by engineers..." First thought is why not to zip/tar away all of these "convenience" files per generation and add a line into build/install script to unpack them after checkout? Additionally, add the .xlf into .gitignore to exclude them from untracked. Noone cares to diff th…

I made another comment as well though tldr is these xlf files are translations tied to texts in code so we can't simply ignore them from the repository. The changes have to be kept so that if we say revert to a certain commit, all the translations match with the texts of headers, buttons, etc...

The entire workflow could remain the same while tar/zipping goes on in the background...

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#206

Post by wewtyflakes »

hamandcheese wrote: post #156
Thorentis wrote: post #80

I don't understand the logic of combining microservices with a monorepo. The whole point of microservices is that you don't care what is beyond the external contract of the service. Who cares how each individual team decides to name their stuff. Why do microservice teams need to care about having every single service checked out? Who or what would be bulk applying changes to all services? This is madness.

Suppose you want to deprecate an API you wrote in favor of something else for $valid_reasons. In a monorepo with the right tooling, I can make a branch where I delete the API and get pretty immediate feedback as to every module I have broken. From there I can update all the call sites and I also know which teams/engineers I should give a heads-up to. In a multi-repo world, this is much more difficult. Even learning w…

The perceived ease of that operation in a monorepo is what makes it dangerous. Unless these changes all map to a single monolith service, then even though you have updated all call sites, these callers will not be deployed all at the same time, meaning as the change rolls out you may see random breakage. By using polyrepo, the deployment boundary can align with the code boundary, making the rollout problem obvious.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#207

Post by stevenjgarner »

I get a headache imagining everything Google in a monorepo. So how do you reconcile this single GIT repository approach with Infoworld's "The case against monorepos"? [1]:

Reason #1. Monorepos go against single-team ownership principles

Reason #2. Monorepos encourage bad practices involving massive refactoring

Reason #3: Small repositories are better than large ones

[1] https://www.infoworld.com/article/3638860/the-case-against-m...

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#208

Post by yen223 »

syspec wrote: post #34

The fact that Canva has a `Source Control team` with at least 5 people on it (going by the thanks at the bottom of the article), means they should probably try a different approach. I think it's a cool company, with a good product, but they're WAY too small to be having a "source control team" on staff. That's at least 1.2MM a year salary / benefits cost.

Canva is worth around $40 billion, it's not really a small company

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#209

Post by pessimizer »

andi999 wrote: post #183
MisterTea wrote: post #169

Earlier quoted context omitted.

> The problem is not the repository, it’s having the translation data in the repo. Does this data change as often as the code does? If not then get it out of the repo.

What if it changes more often than the code? Throw out the code?

There's no material difference between taking the translations out of the repo and taking the code out of the repo. "She shouldn't divorce him, he should divorce her!"

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#210

Post by ralphb »

necovek wrote: post #147
ElemenoPicuares wrote: post #49

Earlier quoted context omitted.

Canva gives users without design tools expertise the ability to make fairly polished looking graphics with a super easy and intuitive interface. (As a designer, I can assure you that polished looking is not the same thing as designed.) It’s a very popular service, so they’re dealing with huge scale. Intuitive interfaces often come with complex mechanisms and lots of assets, and they have clients on every major mobile…

They have obviously invested a lot over time into streamlining their build process: so much so that they're putting an article about it. All of the problems they are having are basically due to their use of a monorepo: they do explain that they made the decision early, but I wonder what are the advantages over multiple repos they are seeing that it was worth it all this trouble?

We can talk about the advantages of monorepos, but your questions is phrased in a way that makes me think that you don't see any "trouble" in multiple repositories.

I would encourage you to do some research and keep an open mind.

Post reply on HN