Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monorepo or Multirepo? Role-Based Repositories

blog.7mind.io

Post reply on HN
Print view View original

81–90 of 120 posts

Re: Monorepo or Multirepo? Role-Based Repositories

#81

Post by shhsshs »

nhumrich wrote: post #46

The cons to multi repo are all anti patterns for microservices anyway. If you're doing microservices you shouldn't have build dependencies on other projects. The should only call eachother at a network level.

That’s the part about monorepos I can’t quite wrap my mind around - yes deploying a single large change to many different systems simultaneously is cool in theory, but how does it actually pan out? Deployment is never instant, so any system-to-system breaking changes would cause a short downtime while everything deploys. In the world I operate in, that’s absolutely not acceptable.

Not that you can’t still make your changes backwards compatible with themselves. But if I’m going to have to deploy everything in two steps anyway, what’s the point?

Re: Monorepo or Multirepo? Role-Based Repositories

#82

Post by laurentlb »

pshirshov wrote: post #63
jyounker wrote: post #57

Earlier quoted context omitted.

Monorepo/multirepo and monolith/microservice are orthogonal concepts. When organizations don't understand that then they may end up building a distributed monolith in across multiple repos. (The "Distributed Big Ball of Mud" anti-pattern.) Monorepo advocates are typically advocating for microservices, but within a single code base. The way you provide access control is through code review and build system visibility.…

> Monorepo/multirepo and monolith/microservice are orthogonal concepts. Yes and no. In both the cases it's a story about components and their isolation . > they may end up building a distributed monolith Yup, seen that many times. > Monorepo advocates are typically advocating for microservices, but within a single code base. I'm avocating roles. Everywhere. > If you're using a tool like bazel If only Bazel supports S…

> If only Bazel supports Scala well enough...

Many companies build their Scala code using Bazel[1]. For example, Databricks wrote about their experience using Bazel on a monorepo containing mostly Scala[2]. Can you share the specific concerns or issues you faced? Thanks.

[1] https://github.com/bazelbuild/rules_scala/blob/master/README... [2] https://databricks.com/blog/2019/02/27/speedy-scala-builds-w...

(Disclaimer: I work on Bazel)

Re: Monorepo or Multirepo? Role-Based Repositories

#83

Post by pshirshov »

laurentlb wrote: post #82
pshirshov wrote: post #63

Earlier quoted context omitted.

> Monorepo/multirepo and monolith/microservice are orthogonal concepts. Yes and no. In both the cases it's a story about components and their isolation . > they may end up building a distributed monolith Yup, seen that many times. > Monorepo advocates are typically advocating for microservices, but within a single code base. I'm avocating roles. Everywhere. > If you're using a tool like bazel If only Bazel supports S…

> If only Bazel supports Scala well enough... Many companies build their Scala code using Bazel[1]. For example, Databricks wrote about their experience using Bazel on a monorepo containing mostly Scala[2]. Can you share the specific concerns or issues you faced? Thanks. [1] https://github.com/bazelbuild/rules_scala/blob/master/README... [2] https://databricks.com/blog/2019/02/27/speedy-scala-builds-w... (Disclaimer:…

Thank you, I know. Though I need to build ScalaJS (and I have one small Scala Native) project. This is a total no-go for Bazel. Unfortunately.

Re: Monorepo or Multirepo? Role-Based Repositories

#85

Post by dodobirdlord »

akhilcacharya wrote: post #11

Amazon does multi-repo. I don't see what the problem or debate over this is. We seem to be handling it pretty fine despite a massive-scale SOA architecture.

The multi-repo pattern certainly meshes well with Amazon's team structure, and of course integrates well with the build system and deployment system, given that they were created around it. But "handling it pretty fine" seems like a stretch.

When last I was there things were finally beginning to burst at the seams. Platform architecture migrations were failing or being abandoned over too many untracked dependencies on specific versions of platform-provided libraries. (RHEL5, anyone?) Third-party had become a jungle of unmaintained libraries with dozens of versions that nobody ever upgraded, that may or may not have security vulnerabilities or known bugs, and many teams hadn't released new versions of their clients/libraries into Live for years in fear of breakage. The Builder Tools team was talking about giving up and abandoning both Brazil and Live as unsalvageable. Framework teams (Coral) were throwing their hands up in the air about how Coral-dependent services would not be able to upgrade to Java 11 without fixing a bunch of breaking changes that they would never agree to fix. The solutions being proposed to these problems by the Builder Tools team looked a lot like moving toward a monorepo, at least conceptually.

Re: Monorepo or Multirepo? Role-Based Repositories

#86

Post by peterwwillis »

sthomas1618 wrote: post #84

I'm curious about those who use a monorepo with microservices: how do you solve CD/CI? Is Bazel the only solution?

CI and CD are more workflows than tools. It doesn't really matter what your repo setup is, you just adapt your workflows to it. On one project I work on we use a monorepo for a handful of microservices. We use standard GitHub flow, no special repo consideration for the CI.

For CD, we have scripts that ask what service you want to build, and they specifically package that service using the set of files & processes dedicated to that service. The build generates a versioned artifact. After that, repo doesn't matter at all, we're just moving service artifacts around.

Re: Monorepo or Multirepo? Role-Based Repositories

#87

Post by UncleMeat »

jayd16 wrote: post #28
username90 wrote: post #23

Earlier quoted context omitted.

At Google we check in the source of every library into the monorepo and compile them ourselves with cached builds from a central server, I don't think we use package managers.

How do you track dependencies of dependencies. Do you need to manually add the full dependency tree and re implement the dependency tracking through your internal system? If a project uses maven or gradle, you need to rewrite those files to point to your internal builds instead?

Full dependency tree yep. No build in google's main repo ever retrieves code externally.

Re: Monorepo or Multirepo? Role-Based Repositories

#88

Post by peterwwillis »

ledneb wrote: post #70

So, in a monorepo world, isn't it often that you have to deploy components together, rather than "it's easy to"? How are services deployed only when there has been a change affecting said service? Presumably monorepo orgs aren't redeploying their entire infrastructure each time there's a commit? Are we taking writing scripts which trigger further pipelines if they detect change in a path or its dependencies? How abou…

Each service has its own code directory, and there's one big "shared code" directory. When you build one service, you copy the shared code directory and the service-specific directory, move to the service-specific folder, run your build process. The artifact that results is that one service. Tagging becomes "-" instead of just "". You may start out with deploying all the services every time (actually hugely simplifies building, testing, and deploying), but then later you break out the infra into separate services the same way as the builds.

Re: Monorepo or Multirepo? Role-Based Repositories

#89

Post by fragmede »

CuriousSkeptic wrote: post #59
weberc2 wrote: post #49

Earlier quoted context omitted.

So where do these binaries get built and how does the system know which binaries to rebuild for a given change? If developers are building binaries and committing them directly, doesn’t that open up security or even correctness issues? How does this approach satisfy compliance concerns (how can the CTO or a manager sign off on the changes that went into the binary if it’s just something a random developer committed?)…

Not sure how people do this in practice. But in principle it seems rather straight forward. A compiler is just a program that takes some input and create some output. Both the compiler and the input can have a cryptographically secure hash. Putting both in a sealed box, like a docker image, with its own hash, gives you a program that takes no input and produces some output. If the box changes, run it in a trusted mac…

Docker makes this drastically easier (need the exact same versions of all libraries and the compiler), but there are still compile time things that are unique per-compile. Debian has been working hard to get hashes of binaries to be useful but the work is far from trivial.

(See also: trusting trust)

Re: Monorepo or Multirepo? Role-Based Repositories

#90

Post by blandflakes »

catalogia wrote: post #66
akhilcacharya wrote: post #11

Amazon does multi-repo. I don't see what the problem or debate over this is. We seem to be handling it pretty fine despite a massive-scale SOA architecture.

When I was there, they were migrating away from perforce because they could no longer scale perforce fast enough to meet demand. I've not seen this talked about much outside of Amazon. It was also a huge day-to-day quality of life improvement for the users (the developers.) There are UX problems with git, but they pale in comparison to the UX problems with perforce which is truly unpleasant software.

The Alexa division migrated aggressively to git as soon as it was available and nobody publicly voiced any regret about losing perforce.
Post reply on HN