Monorepos: Please don’t • Hacker News Board

Re: Monorepos: Please don’t

#381

Post by yowlingcat » Fri, Jan 04, 2019, 1:05 AM UTC

manigandham wrote: post #299

thanatos_dem wrote: post #289

Earlier quoted context omitted.

I work on CI/CD systems, and that’s one thing that definitely gets harder in a monorepo. So you made a commit. What artifacts change as a result? What do you need to rebuild, retest, and redeploy? It doesn’t take a large amount of scale to make rebuilding and retesting everything impossible. In a poly repo world, the repository is generally the unit of building and deployment. In monorepo it gets more messy. For inst…

Rebuild and deploy everything, what's the actual problem? Like the OP said, that's a scale issue and most projects don't have it. Also building/testing is far more effective at finding dependencies than just going by repo structure. There are numerous package managers available to solve versioning if you need separate components.

100% agree with your entire comment. This is what we do with our monorepo now -- it turns out the rebuilding and deploying everything is actually just fine. If your application services are stateless and decoupled from your state stores, it's completely harmless. If you need to do something fancy, congrats! You're at scale -- enjoy it but remember that it's something rare.

Re: Monorepos: Please don’t

#382

Post by joevandyk » Fri, Jan 04, 2019, 1:07 AM UTC

bpicolo wrote: post #213

forty wrote: post #113

Earlier quoted context omitted.

My rule of thumb is: if you need to do PRs in several repositories to do one features, you should probably merge the repositories. At work, we have code spread among a bunch of repositories, and having to link to the 2/3 related PRs in other repos is a major PITA, and even more so for the reviewers.

At a past job, I had to edit roughly 5 different repositories in order to do some trivial programming task (send an email or some such). It was quite easily the least productive / most demoralizing workflow I've ever experienced. Context switching really sucks. You should aim to reasonably avoid it

Sending an email can have a few different responsibilities:

Who is the email being sent to?

What is the content of the email?

What data does the email content and recipient depend on?

What are you tracking on the email?

How is the email visually formatted?

All those things might be in different apps as the logic gets more complicated.

Re: Monorepos: Please don’t

#383

Post by olingern » Fri, Jan 04, 2019, 1:38 AM UTC

jayd16 wrote: post #121

olingern wrote: post #37

I’ve found monorepos to be extremely valuable in an immature, high-churn codebase. Need to change a function signature or interface? Cool, global find & replace. At some point monorepos outgrow their usefulness. The sheer amount of files in something that’s 10K+ LOC ( not that large, I know ) warrants breaking apart the codebase into packages. Still, I almost err on the side of monorepos because of the convenience th…

Hold on, are we talking about monorepos, ie a set of projects with shared change history (and possibly 'build it all' type tooling) or single monolithic apps? I'm seeing these two things conflated in this thread.

To me, a monorepo exists of a set of related or semi related services or runtimes that can operate autonomously, but have a dependency on their siblings to operate correctly.

In some cases, this could be two separate backend projects where you want to re-use the same deployment pipeline.

Often, I find that API wrappers are something that I share across frontends and backends in the JS world, so it often makes sense to separate my projects into:

- backend

- frontend

- common

In Typescript I really like this pattern and can namespace shared types so that it’s very clear to the future reader that this type is probably used outside of the current context.

So, to reply to your comment — I think the term “monorepo” can encompass a lot of different project types.

I think Dan Luu covers the bases quite well here:

https://danluu.com/monorepo/

Re: Monorepos: Please don’t

#384

Post by olingern » Fri, Jan 04, 2019, 1:52 AM UTC

kazagistar wrote: post #49

olingern wrote: post #37

I’ve found monorepos to be extremely valuable in an immature, high-churn codebase. Need to change a function signature or interface? Cool, global find & replace. At some point monorepos outgrow their usefulness. The sheer amount of files in something that’s 10K+ LOC ( not that large, I know ) warrants breaking apart the codebase into packages. Still, I almost err on the side of monorepos because of the convenience th…

Monorepos and packages are not mutually exclusive. You can and should have many different projects in subfolders I'm your monorepo, each with their own builds and tests and artifacts (though hopefully somewhat standardized). The point is that now it's easy to release changes across multiple projects, integration test between them on a specific global patch, etc, without a whole pile of complex tooling.

Agreed. When I wrote the parent comment, I was thinking of a time I prematurely abstracted an API wrapper to a private git repo and how painful simple, frequent changes were.

Though, as you say and I commented below they’re not mutually exclusive. A wrapper or even an entirely separate service can exist alongside others.

One dark side of this is being able to “reach inside” other parts of the monorepo and blur application boundaries.

Re: Monorepos: Please don’t

#385

Post by rossjudson » Fri, Jan 04, 2019, 3:23 AM UTC

erik_seaberg wrote: post #334

rossjudson wrote: post #163

Earlier quoted context omitted.

Hmm. Monorepos strongly favor "always-good". Core library breakage gets detected and rolled back really fast. It's been months since I've been affected by one. Known-good is in the eye of the beholder, and is just another dimension that generates breakage.

google3 breaking was a bimonthly event that left the entire team aground. Bad enough that it happened, but the worst part is that we couldn't do anything about it but wait.

I guess my end of google3 hasn't been like that in a long time.

Re: Monorepos: Please don’t

#386

Post by rossjudson » Fri, Jan 04, 2019, 3:24 AM UTC

kiallmacinnes wrote: post #164

rossjudson wrote: post #148

Hilariously misguided. Pretty funny to read that the things I do every day are impossible. Monorepo and tight coupling are orthogonal issues. Limits on coupling come from the build system, not from the source repository. Yes, you should assume there is a sophisticated "VFS". What is this "checkout" you speak of? I have no time for that. I am too busy grepping the entire code base, which is apparently not possible. If…

> then why on earth would google invest enormous effort constructing an entire ecosystem around a monorepo? Choices: 1) Google is dumb. 2) Mono and poly are not identical. I think, once you've chosen a path of mono or poly, you have quite a challenge ahead of you to migrate to the other. At that point, the tradeoffs arent based purely on the technical benefits - and "invest in monorepo tooling" may become a perfectly…

I'd refine this to say "it's a good idea the way Google does it".

Re: Monorepos: Please don’t

#387

Post by laumars » Fri, Jan 04, 2019, 8:03 AM UTC

Chris2048 wrote: post #345

laumars wrote: post #189

Earlier quoted context omitted.

That only really matters if your backend developers are a different team to your frontend developers where they'd want to be working concurrently. And even then, they could work in different branches and both teams merge into a development branch when finished. The idealistic discussions for or against monorepos often overlook the most important detail: who's working on the code and how would you want them to version…

> That only really matters if your backend developers are a different team to your frontend developers What if there is one team, but different developers (one working on the frontend, another on the back)? What if QA can test the API while the frontend development is ongoing? What if the front and backends have different toolchains, and ultimately separate execution environments (server app backend vs JS running on…

I’m not sure what your point is. There’s obviously going to be thousands of different scenarios that I didn’t cover; it would be impossible to cover every imaginable use case.

> What if there is one team, but different developers (one working on the frontend, another on the back)?

Then presumably everyone in that team are full stack?(Otherwise it would be different teams in the same department) so it still makes sense to have a monorepo because you could have a situation (holiday, sickness) where someone would be working on both the front end and back end. Thankfully got is a distributed version control solution and supports feature branches so you can still have multiple people working on the same repo and then merge back into a developement branch.

> What if QA can test the API while the frontend development is ongoing?

Testing isn’t the same as released versions. You can (and should) test code at all stages of development regardless of team structures, git repo structures nor release cycles.

> What if the front and backends have different toolchains, and ultimately separate execution environments (server app backend vs JS running on client machines).

I’d already covered that point when talking about different languages in the same repo. You’re making a distinction about something that version control doesn’t care in the slightest about.

I think it’s fair to say any significant cross-project tooling should be it’s own repo (you wouldn’t include the web browser or JVM with your frontend and backend repos). But if it’s just bootstrapping code that is used specifically by that project then of course you’d want that included. Eg you wouldn’t have Makefiles separate from C++ code. But you wouldn’t include GCC with it because that’s a separate project in itself.

Ultimately though, there is no right answer. It’s just what works best for the release schedule of a product and teams who have to work upon that project.

Re: Monorepos: Please don’t

#388

Post by Vinnl » Fri, Jan 04, 2019, 10:28 AM UTC

erik_seaberg wrote: post #325

Vinnl wrote: post #215

Earlier quoted context omitted.

So now we're at three repo's, one of which is shared by the other two, and changes will have to be coordinated over them. I fail to see how that is an improvement over having both in the same repo. In the end, I think the other comments are right that it mostly depends on who's working on something. If it's different teams, then different repo's probably make sense. But if I'm responsible for both the back-end and th…

> I'm willing to believe Googlers that it works well for Google It doesn't. In my entire career, that was the only environment in which some random would break us and we couldn't do anything about it other than hope for a rollback and then wait for hours for the retest queue to clear before we could deploy anything at all. Maybe not all the time, but you need the escape hatch of pinning healthy deps, because HEAD of…

Well, I'm willing to believe you that it didn't work well for you as well. My point is that company-wide monorepo's are largely irrelevant to my point, as I'm not arguing in favour or against those (I'm leaving that to people who've worked with it).

Re: Monorepos: Please don’t

#389

Post by Too » Fri, Jan 04, 2019, 10:29 AM UTC

malkia wrote: post #69

I do really like mono-repos, but google's other significant new project: fuchsia - is set-up as multi-git repo (and I believe chromium too, maybe android (haven't checked)). For fuchsia, they use a tool called "jiri"[1] to update the repos, previously (and maybe still in use) is the "gclient" sync tool [2] way from depot_tools[3] [1] - https://fuchsia.googlesource.com/jiri/ [2] - https://chromium.googlesource.com/chr…

Android uses a top level repo that behaves as a monorepo with thousands of submodules inside. It's also designed for multiple companies sharing code and working with non shared code at the same time which introduces some constraints and challenges.

Re: Monorepos: Please don’t

#390

Post by gdfasfklshg4 » Fri, Jan 04, 2019, 11:07 AM UTC

skj wrote: post #301

gdfasfklshg4 wrote: post #255

Earlier quoted context omitted.

* Changing graphql schemas. * Any non-backwards compatible change in the interface between the components. Yes this can be solved. But when working in a smaller team on proprietary software why use time solving a problem you don't need to solve? (This is from experience.)

> why use time solving a problem you don't need to solve? Unless they're running on the same computer and deploy literally simultaneously, this is already a problem you need to solve.

A surprising number of companies are prepared to accept an hour of downtime for an internal system if it saves them money.. In my experience the best business practice is to offer the product owner/manager the costed options in such a situation and allow them to choose.