Live data from Hacker News

Monorepo: please do

medium.com

51–60 of 166 posts

Re: Monorepo: please do

#51

Please don't. It's just too slow and not efficient. Instead use common open source best practices of shared library architecture. Problem solved! Putting everything into one repo is just lack of organization and creates a huge mess.

It would be quite remarkable if in-house corporate software, which face different constraints and challenges than open source software, turned out to nonetheless have exactly the same best practices.

The idea of using open source styled practices for internal development is not exactly new or remarkable. It's something people have been doing for a long time.

https://en.wikipedia.org/wiki/Inner_source

Re: Monorepo: please do

#52
post #17

Earlier quoted context omitted.

Sure but then you only have some small portion of the total infrastructure, which adds its own layer of complexity for the people reviewing your changes :P It's all trade offs, is all I'm saying - I honestly still can't decide between the two, although for all companies sub 20 people, I'd for sure stick with a single repo.

If I'm working on Application X, wtf do I care about infrastructure code? Or for that matter, as a specific... if someone is working on Google Maps, should they care about the codebase for Google Inbox for Android?

Does Application X rely on particular infrastructure configuration? Or does Google Inbox on Android integrate with Google Maps?

There are dependencies everywhere. Monorepos are one of the tools which can be used to make dealing with them easier in some cases. They’re not an absolute solution not appropriate for all circumstances, but no tool is!

Re: Monorepo: please do

#53

"Shared responsbility" is one of those ideas that sounds good on paper, but doesn't really scale terribly well in the real world. As the old saying goes "when everybody is responsible, nobody is responsible". More to the point, as the author of TFA allows, once a system reaches a certain size, nobody can understand it all. At some point you have to engage division of labor /specialization, and once you do that, it do…

> poly-repo approach, with a designated owner for discrete modules, but where anybody can clone any repo, make a proposed fix, and submit a PR

This is how it works at my company. The issue we run into is that PRs coming from non-core maintainers tend to either get over-scrutinized (e.g. "this diff may work for you but it's not generic enough for X/Y/Z") or flat out ignored at the code review stage and sometimes don't land in a timely enough manner.

Another challenges with this approach is when you have deeply nested dependencies and need to "propagate" an upgrade in some deep dep up the tree. In the JS/Node world, this usually means fixing an issue involves hacking on transpiled files in the node_modules folder of a project to figure out what change needs to be made, and then mirroring said change into the actual repo and then tweaking things until type checking/linting/CI pass. Not really conducive for collaboration.

One other problem is that security/bug fix rollouts are a bit more challenging. We had a case a while back where a crash-inducing bug was fixed and published but people still experienced crashes due to not having upgraded the one out dozens of packages required by their projects.

Re: Monorepo: please do

#54

My org went from polyrepo 10 commit semver dependency hell when updating an internal API to monorepo and it saves a lot of time. Unmigrated semver breaking changes are a form of technical debt, and it takes a lot more total man hours to do the 'proper' one by one many commit poly repo migration than the other way around. If we had the tooling to do multirepo atomic commits and reviews then maybe we would of stuck wit…

My org went from a monorepo where every project had to obey the same CI model and you could not introduce entirely new CI tools for new prototypes over to a polyrepo with separated semver library repos for shared dependencies, and it simplified everything so much.

Adding additional PRs across different repos is functionally no different than the same PR with scattered dependencies in a monorepo, except that separating the PRs makes each isolated set of changes more atomic and focused, which has led to fewer bugs and better quality code review and, the hugest win, each repo is free to use whatever CI & deployment tooling it needs, with absolutely no constraints based on whatever CI or deployment tool another chunk of code in some other repo uses.

The last point is not trivial. Lots of people glibly assume you can create monorepo solutions where arbitrary new projects inside the monorepo can be free to use whatever resource provisioning strategy or language or tooling or whatever, but in reality this not true, both because there is implicit bias to rely on the existing tooling (even if it’s not right for the job) and monorepos beget monopolicies where experimentation that violates some monorepo decision can be wholly prevented due to political blockers in the name of the monorepo.

One example that has frustrated me personally is when working on machine learning projects that require complex runtime environments with custom compiled dependencies, GPU settings, etc.

The clear choice for us was to use Docker containers to deliver the built artifacts to the necessary runtime machines, but the whole project was killed when someone from our central IT monorepo tooling team said no. His reasoning was that all the existing model training jobs in our monorepo worked as luigi tasks executed in hadoop.

We tried explaining that our model training was not amenable to a map reduce style calculation, and our plan was for a luigi task to invoke the entrypoint command of the container to initiate a single, non-distributed training process (I have specific expertise in this type of model training, so I know from experience this is an effective solution and that map reduce would not be appropriate).

But it didn’t matter. The monorepo was set up to assume model training compute jobs had to work one way and only one way, and so it set us back months from training a simple model directly relevant to urgent customer product requests.

Had we been able to set this up as a separate repo where there were no global rules over how all compute jobs must be organized, and used our own choice of deployment (containers) with no concern over whatever other projects were using / doing, we could have solved it in a matter of a few days.

In my experience, this type of policy blocker is uniquely common to monorepos, and easily avoided in polyrepo situations. It’s just a whole class of problem that rarely applies in a polyrepo setting, but almost always causes huge issues with monorepo policies and fixed tooling choices that end up being a poor fit for necessary experiments or innovative projects that happen later.

Re: Monorepo: please do

#55

I think that the reason of the argument is that both authors understand the definition of "large repository" very differently. With 100 engineers a monorepo might seem a good idea. With 500 it becomes nearly impossible to do anything involving a build. Some isolation is needed. Also from my experience many engineers just don't give a shit about architecture. They create entangled mess, that kind of works for the cust…

> With 500 it becomes nearly impossible to do anything involving a build.

Both FB and Google have more than 500 devs and are using a monorepo.

Re: Monorepo: please do

#56
post #40

I think that the reason of the argument is that both authors understand the definition of "large repository" very differently. With 100 engineers a monorepo might seem a good idea. With 500 it becomes nearly impossible to do anything involving a build. Some isolation is needed. Also from my experience many engineers just don't give a shit about architecture. They create entangled mess, that kind of works for the cust…

the fact that essentially 100% of big tech companies use monorepos seems like evidence that it is at least possible to do it in a scalable way...

Amazon does not use a monorepo, so you might want to rethink your "statistic".

Re: Monorepo: please do

#57

"Shared responsbility" is one of those ideas that sounds good on paper, but doesn't really scale terribly well in the real world. As the old saying goes "when everybody is responsible, nobody is responsible". More to the point, as the author of TFA allows, once a system reaches a certain size, nobody can understand it all. At some point you have to engage division of labor /specialization, and once you do that, it do…

That's what we did at my last workplace; 800-odd devs, 3000-odd repos, and a service we wrote to record and gather all the ownership metadata (which team owns this repo? where can I contact them? who should be on the code reviews? how do I page them? which project should tickets be created in?)

It was a gigantic pain trying to find owners for half-dead repos for services still running and in use, where the original authors had left years ago & from teams 4 or 5 restructures ago. The one thing I learned was: never make a user the owner of a repo (unless it is in their personal space), always find a team to accept responsibility for it.

Re: Monorepo: please do

#58
post #29

There aren't good monorepo solutions out there (yet). Git LFS is great for few large files, but it doesn't help with tons of smaller files. Git submodules are crap when it comes to usability, and have been for a long time, it's even mentioned in the famous Torvalds Git Talk. Git had a sparse checkouts feature since a long time, but it only affected the checkout itself, all the blobs would still be synced. Now, Git is…

https://vfsforgit.org/ is another option here - MS-originated and Github is adopting it - https://venturebeat.com/2017/11/15/github-adopts-microsofts-...

Re: Monorepo: please do

#59

My org went from polyrepo 10 commit semver dependency hell when updating an internal API to monorepo and it saves a lot of time. Unmigrated semver breaking changes are a form of technical debt, and it takes a lot more total man hours to do the 'proper' one by one many commit poly repo migration than the other way around. If we had the tooling to do multirepo atomic commits and reviews then maybe we would of stuck wit…

My org went from a monorepo where every project had to obey the same CI model and you could not introduce entirely new CI tools for new prototypes over to a polyrepo with separated semver library repos for shared dependencies, and it simplified everything so much. Adding additional PRs across different repos is functionally no different than the same PR with scattered dependencies in a monorepo, except that separatin…

> each repo is free to use whatever CI & deployment tooling it needs, with absolutely no constraints based on whatever CI or deployment tool another chunk of code in some other repo uses.

Hear, hear. Let teams choose the processes and tools that work best for them. In previous release engineering positions, I resisted the many attempts to instroduce a single standard workflow for all projects. The support burden of letting a thousand flowers bloom was not great, but the benefit was that devs understood their project and were empoiwered to make changes when the business requirements changed faster than standardized tooling could.

We had a few contracts for standard behaviours, but they were low-overhead: must respond to 'make/make test', have a /status endpoint that 500'd when it was unhealthy, register a port in the service conf repo, etc.

Re: Monorepo: please do

#60

My org went from polyrepo 10 commit semver dependency hell when updating an internal API to monorepo and it saves a lot of time. Unmigrated semver breaking changes are a form of technical debt, and it takes a lot more total man hours to do the 'proper' one by one many commit poly repo migration than the other way around. If we had the tooling to do multirepo atomic commits and reviews then maybe we would of stuck wit…

My org went from a monorepo where every project had to obey the same CI model and you could not introduce entirely new CI tools for new prototypes over to a polyrepo with separated semver library repos for shared dependencies, and it simplified everything so much. Adding additional PRs across different repos is functionally no different than the same PR with scattered dependencies in a monorepo, except that separatin…

This has nothing to do with monorepos though. Its entirely a company policy issue. There's nothing about the monorepo that prevents you from writing a script that ran on precommit and built and deployed via docker to a test cluster.

Unless you mean your presubmit test would push to production machines, that's bad and shouldn't be allowed, but again has nothing to do with a monorepo.

A company could just as easily have draconian policy about testing and deployment and multiple repos. Maybe you could break the rules (hell you could have broken the rules in monorepo land), but again, that's just a rules issue, not an issue of the repository.

Post reply on HN