371–380 of 402 posts
I accept the infeasibility of my dream. But I'd like my repo to cover as much as my tooling realistically allows.
I call bullshit on "our repository is too big for one machine". Seriously, you have over 1 TB of code and 100 people wrote it?
adding raw versions of binary assets (designs, video, ...) can quickly lift a repo beyond a TB. Now, you could say "don't do that", but there's valid use cases where you'd want to track all binary assets as part of the development cycle.
Earlier quoted context omitted.
> with a monorepo the "core" or "foundation" or "developer experience" teams tend to act like they're the owners of all the code and everyone else is just visiting. Fwiw, this hasn't at all been my experience with this kind of thing at Google. Certainly developer experience teams and language teams will make broad changes that affect everyone but Those changes are trivial: the change maker has to make a convincing ar…
> Consistent style is important Many things are important. Some things are more important than others, and I'd say that "not making a thousand developers' workflows more cumbersome" is higher on the list than style issues. > If ... I make a change that breaks you but my tests pass, you don't know it. So don't do that. Open-source projects deal with this exact same issue across repos and owners all the time . There ar…
I'd argue that in the long run, not being able to update dependencies because they broke you is going to be much worse than them fixing the incompatibilities for you.
Either way, you need people to act like adults and communicate, but the multirepo problem is worse.
Earlier quoted context omitted.
> we have a Monorepo for each app platform Do you mean two separate mono-repos, one for Android, one for iOS? To me that's not a monorepo. Is there little shared code between the two platforms, or is there a third repo that is depended on?
Yeah, it's two separate monorepos. There is actually some shared code between multiple monorepos. Code like IDL's and some C mapping code are shared, but they are referenced as vendor libraries so each monorepo updates those dependencies when they need to. If you think about Android vs iOS vs a web based dispatch system vs an autonomy system; They are all totally different. All of the code, dependencies, vendor code…
Earlier quoted context omitted.
Microsoft switched all of windows to a mono repo.
I think Windows was already a single repo forever. IMHO, that's exactly what is slowing it right now. Other products have their own repos, sometimes multiple. As of 2016.
Earlier quoted context omitted.
>Exceptions? Nah, multiple returns! Dependency management? Who needs it... Oh, wait. I thought consensus was that exceptions, like OOP, is an antipattern. I guess there's room for different opinions. :-/
There is a largely held opinion (that I share) that exceptions are a dated pattern that is better replaced by the more modern alternatives. They are still much better than the older patterns they were created to replace, like multiple return.
Earlier quoted context omitted.
My current team managed to break a single "component" out into a separate repository. Then that repository broke into two, then those broke into other repositories, until we've eventually have around 10 or so different repositories that we work on every day. An average change touches 4 of them, and touching one of them triggers on average releases on 2 or 3 of them. Even building these locally is super tedious, becau…
even if they are heavily coupled, So don't use polyrepos for heavily coupled projects, then. Or even better... ... try to avoid heavy coupling in the first place.
Q: Why are we debating the merits of mono-repos over poly-repos?
A: Because it's managing dependencies is really hard and needs expertise.
Earlier quoted context omitted.
Number of companies is a good metric, because companies own the repos and if it becomes a pain-point, only the developers working at that point in time will be hit by this. Anyone who leaves before this inflection point or joins after it's been solved will not be hit, so I don't think the percentage of developers in that intersection is large.
> after it's been solved I think a quick perusal of this page will show that it's not really "solved" after all. A far higher percentage of developers continue to be affected by large-repo issues than a Python-specific issue (currently #1 story on the front page) or anything to do with Ethereum (currently #7). Are those "horseshit" topics too?
What you call large-repo issues I call organization issues. From your other comments, it's clear that we draw the lines at different places, but I think I'm right and you're wrong in this case because I've seen engineers try to solve organizational issues with technology enough times that it's a presumable anti-pattern. Why don't we take your own words at face value?
"That hasn't been my experience. Yes, it's a culture thing rather than a technology thing, but with a monorepo the "core" or "foundation" or "developer experience" teams tend to act like they're the owners of all the code and everyone else is just visiting. With multiple repos that's reversed. Each repo has its owner, and the broad-mandate teams are at least aware of their visitor status. That cultural difference has practical consequences, which IMO favor separate repos. The busybodies and style pedants can go jump in a lava lake."
Why are there busybodies and style pedants working in your organization? Because your organization has an issue. Do you think that would be at the root of this pain, or a tool choice? I'll give you a hint, it's not the tool choice.
Earlier quoted context omitted.
I agree, it's not really solved, but solved "enough". You can't have your cake and eat it, there are tradeoffs involved- if you grow large enough to hit monorepo limitations, you are large enough to invest in tooling that manage your workflow (the tradeoff). However, if you're a small organization, you can't afford the tooling and you're wasting time/quality coordinating polyrepo releases, so you are better off with…
> opinion that the article is "horseshit" is unrelated to how well its ranked on HN. When the opinion is not just disagreement but outright dismissal of the topic as worth discussing, I'd say ranking is relevant. So is comment count. Clearly a lot of people do believe it's worth discussion, not irrelevant or a foregone conclusion as yowlingcat tried to imply.
Earlier quoted context omitted.
No, it is horseshit. 99% of companies will never hit big company VCS scaling issues, and once they do, they're on their own. To characterize that scale as common is one of the most embarrassing failures of modern software engineering. People are so embarrassed to use well worn tooling and accept that large scale is both uncommon and something that doesn't invalidate tried and true patterns for smaller scales. It's ut…
In my experience, monorepos cause outrageous problems that have nothing to do with scale. Small or medium monorepos are equally as terrifying. It’s much more about coupling and engendering reliance on pre-existing CI constraints, pipeline constraints, etc. If you work in a monorepo set up to assume a certain model of CI and delivery, but you need to innovate a new project that requires a totally different way to appr…
"The last point is not trivial. Lots of people glibly assume you can create monorepo solutions where arbitrary new projects inside the monorepo can be free to use whatever resource provisioning strategy or language or tooling or whatever, but in reality this not true, both because there is implicit bias to rely on the existing tooling (even if it’s not right for the job) and monorepos beget monopolicies where experimentation that violates some monorepo decision can be wholly prevented due to political blockers in the name of the monorepo.
One example that has frustrated me personally is when working on machine learning projects that require complex runtime environments with custom compiled dependencies, GPU settings, etc.
The clear choice for us was to use Docker containers to deliver the built artifacts to the necessary runtime machines, but the whole project was killed when someone from our central IT monorepo tooling team said no. His reasoning was that all the existing model training jobs in our monorepo worked as luigi tasks executed in hadoop.
We tried explaining that our model training was not amenable to a map reduce style calculation, and our plan was for a luigi task to invoke the entrypoint command of the container to initiate a single, non-distributed training process (I have specific expertise in this type of model training, so I know from experience this is an effective solution and that map reduce would not be appropriate).
But it didn’t matter. The monorepo was set up to assume model training compute jobs had to work one way and only one way, and so it set us back months from training a simple model directly relevant to urgent customer product requests."
What do you think is the cause of your woes, the monorepo, or the disagreement between your colleague in central IT tooling who disagreed with you? Where was your manager in this situation? Where was the conversation about whether GPU accelerated ML jobs were worth the additional business value to change the deployment pipeline? Was that a discussion that could not healthily occur? Perhaps because your organization was siloed and so teams compete with each other rather than cooperate? Perhaps because it's undermanaged anarchy masquerading as a meritocracy? Stop me if this sounds too familiar.
I've been there before. I know what it feels like. But, I also know what the root cause is.