This isn’t entirely honest, as Uber doesn’t have one monorepo, it has multiple. They never understood what Mono means lol.
[deleted]
Devpod: Remote development environment at Uber
161–170 of 235 posts
Re: Devpod: Remote development environment at Uber
#162I like this workflow, use something like mutagen to sync from your local environment to the remote machine. For me, I prefer jetbrains IDEs so I tried using jetbrains gateway, it’s really buggy and slow. I prefer to use jetbrains ides on my local machine and use VSCode remote for interacting with the remote machine. I wonder though, is it worth it to setup a remote dev machine if you have an M1 Mac? I think you can g…
I think there are still some limits applied even on cutting edge hardware. GitHub previously said they hit network limitations cloning their multi-gb repo for instance.
If I am not mistaken, git clone is an atomic operation (it either succeeds or fails).
Since I see it mentioned in many threads here: for a huge repo one can always use `git clone --depth 1` to get that repo and later do a proper pull to retrieve all history.
Re: Devpod: Remote development environment at Uber
#163Earlier quoted context omitted.
Monorepos and monoliths are orthogonal Monorepos are a tooling nightmare. Monorepos are a bandwidth black hole. Monorepos make some things possible that are just not possible or very very hard when you have multiple independent repos that are built and tested independently. Namely, a) they allow you to test the effects of a change on all the components that depend on you, before you merge your change. This reduces th…
Can you explain how a monorepo for microservices is a tooling nightmare and bandwidth sink? In my experience, a monorepo significantly makes things easier and saves time everyday for all developers involved when compared with multi repo.
>Can you explain how a monorepo for microservices is a tooling nightmare and bandwidth sink?
bandwidth:
I meant "bandwidth" as literal bandwidth. When your codebase becomes huge, your VCS repo size becomes enormous and it becomes harder and harder to keep a full checkout on all the development machines, especially if they are over the WAN (e.g. at home on your laptop).
This has fuelled solutions like sparse checkouts (like MS vfs for git, now scalar) remote development (like TFA; but also Google's cider and srcfs etc).
tooling:
naïve monorepo tooling (which I've seen in various companies I worked for) simply perform a full build of the whole monorepo for each CI execution. At first this is just fine since you can parallelize builds and call it a day; but after a while, the builds just don't scale anymore ; flaky tests become an increasing frustration etc.
The tooling that can help scale large monorepos does exist, but requires buy in and comes with its own learning curve and tradeoffs. One well known such tool is bazel (https://bazel.build) and bazel remote builds and remote caches. These tools are hard to set up, although folks at https://www.buildbuddy.io/ can help smaller startups by offering a managed service.
(Again, I'm talking about really large monorepos. A monorepo which includes a dozen or so modules, for which you can easily perform a full re-build on a single CI worker and on your laptop is not the kind of repo that creates tooling nightmares.)
Re: Devpod: Remote development environment at Uber
#164Earlier quoted context omitted.
Can you explain how a monorepo for microservices is a tooling nightmare and bandwidth sink? In my experience, a monorepo significantly makes things easier and saves time everyday for all developers involved when compared with multi repo.
>Can you explain how a monorepo for microservices is a tooling nightmare and bandwidth sink? bandwidth: I meant "bandwidth" as literal bandwidth. When your codebase becomes huge, your VCS repo size becomes enormous and it becomes harder and harder to keep a full checkout on all the development machines, especially if they are over the WAN (e.g. at home on your laptop). This has fuelled solutions like sparse checkouts…
Thank you. Monorepos of _that_ size is definitely far more complex and comes with certain issues that you mention. Although, I feel like you'll only hit that wall once you are _very_ large as a company and at that point, you'll also have resources to climb that wall.
For small-medium scale, monorepo has been a blessing after dealing with multi-repo systems for years.
Re: Devpod: Remote development environment at Uber
#165Earlier quoted context omitted.
It's much more of a political tool than a technical one. Every benefit (minus atomic commits*) can be had immediately on micro-repos with a for-loop to do the monorepo-thing in each one. If bazel is what you want, great, use bazel! In microrepos! Want consistent dependencies? Enforce them! HEAD must build? Wonderful, nothing's stopping you from doing that! It's all solvable, and quite easily. What you actually want i…
Monorepo lets me have a _single_ Github PR with the changes across 4 projects in front of me. To replicate what you're talking about in a multirepo setup, you have to introduce changesets in 4 different places, then merge them in the right order, make a bunch of tiny releases, update commit hashes to point to the right one, and do a lot of other bookkeeping because in this house we keep our projects separated.
Indeed, the small additional bookkeeping is the price to pay. I'm quite used to it, so I don't even see it anymore.
Is that all? These 5 less minutes of bookkeeping are the killer feature of monorepos?
Because, as far as I see it, there's an insane amount of engineering to make a monorepo work even at small scale. Are these 5 minutes per multi-project PR worth it?
I wouldn't be surprised these 5 minutes are largely offsets solely by the additional hours of CI testing time introduced by having to run tests on the whole monorepo at each commit instead of just the project that changed.
Re: Devpod: Remote development environment at Uber
#166Earlier quoted context omitted.
Can you explain how a monorepo for microservices is a tooling nightmare and bandwidth sink? In my experience, a monorepo significantly makes things easier and saves time everyday for all developers involved when compared with multi repo.
The biggest issue is that every CI provider speaks with "repo=project" language, so your way forward with monorepos is using stuff like Bazel. Bazel is extremely cool! But you end up with, like, a handful of people on the team who can write Bazel and eveyone else cargo culting their way through it. There are a lot of JS "monorepo management" tools but they all seem concerned about the release phase for a lot of libra…
Thank you.
> repo=project
This has gotten better lately. At least, good enough for small-medium scale projects.
Re: Devpod: Remote development environment at Uber
#167Back in the early 2000s, Sun ("the network is the computer") had a similar solution that worked seamlessly for most of their software org-- the Sun Ray. https://en.wikipedia.org/wiki/Sun_Ray It was a network terminal. Your files and entire session were on the server. Your “local” terminal consisted only of a network interface and enough compute power to display your session. The way they had it set up was that you co…
What's old is new again. How soon until we realize the X window system actually had some good ideas again and start running desktop apps on cloud servers for remote work?
>start running desktop apps on cloud servers for remote work
Hopefully never.
Re: Devpod: Remote development environment at Uber
#168Earlier quoted context omitted.
Monorepo lets me have a _single_ Github PR with the changes across 4 projects in front of me. To replicate what you're talking about in a multirepo setup, you have to introduce changesets in 4 different places, then merge them in the right order, make a bunch of tiny releases, update commit hashes to point to the right one, and do a lot of other bookkeeping because in this house we keep our projects separated.
Indeed, the small additional bookkeeping is the price to pay. I'm quite used to it, so I don't even see it anymore. Is that all? These 5 less minutes of bookkeeping are the killer feature of monorepos? Because, as far as I see it, there's an insane amount of engineering to make a monorepo work even at small scale. Are these 5 minutes per multi-project PR worth it? I wouldn't be surprised these 5 minutes are largely o…
> hours of CI testing time introduced by having to run tests on the whole monorepo at each commit instead of just the project that changed.
This is rarely the case. Most monorepos shard tests such that wall time is minimised and have dependency analysis that only runs tests affected by the changed code.
More of a problem is that IDEs and LSPs often don't deal well with having to index and navigate very large codebases.
I've maintained both monorepo and polyrepo environments and there are pros and cons to both, and they vary _wildly_ based on the language being used.
Re: Devpod: Remote development environment at Uber
#169Earlier quoted context omitted.
>Can you explain how a monorepo for microservices is a tooling nightmare and bandwidth sink? bandwidth: I meant "bandwidth" as literal bandwidth. When your codebase becomes huge, your VCS repo size becomes enormous and it becomes harder and harder to keep a full checkout on all the development machines, especially if they are over the WAN (e.g. at home on your laptop). This has fuelled solutions like sparse checkouts…
Thank you. Monorepos of _that_ size is definitely far more complex and comes with certain issues that you mention. Although, I feel like you'll only hit that wall once you are _very_ large as a company and at that point, you'll also have resources to climb that wall. For small-medium scale, monorepo has been a blessing after dealing with multi-repo systems for years.
I'm a fan of monorepos personally.
That said, I cannot pretend I don't see the problems with suboptimal tooling working in a medium size codebase. "big" and "medium" and "small" are quite subjective things.
At $work we have a monorepo whose git repo grew to 1GB in size and where the CI turn-around is so high, and the glitches so often, that it often takes hours or even days to land some code to prod. Developers instinctively react by making bigger and bigger changes because the very thought of going through PR/review/CI/merge cycle once again terrifies them. It's all compounded by a security policy that forces a code review approval every time the source code changed, including when you have to apply fixes to build failures induced by a component you don't own.
All of these things can and should be be fixed. But this is work, and is not urgent work so it's not often done at the same pace other stuff is done. This induces fatigue in the team and, as usually things go, people tend to blame the most easiest thing to blame: the monorepo.
That's why I try to phrase the problem to be a tooling problem and not a monorepo problem. Clearly if you don't have a problem with your monorepo, you either already have good tooling, or you don't need good tooling.
Re: Devpod: Remote development environment at Uber
#170Can someone explain the rationale behind switching to a monorepo? I just don't get it. Does it mean also having a single unified production environment build? I have been managing a stack composed of 500+ repositories, communicating through webservices, files and ABIs for many years now, and never quite hit much of the issues cited as reasons to switch to monorepos. Having multiple small independent environments for…
I maintain the web monorepo at Uber, so I think I can give some context. Monorepos allows us to centralize important dependency upgrades. E.g. fixing log4j vulns is a lot easier when you can patch everything simultaneously. Same for tzdata (2022g gave very little heads up) Auditing for npm supply chain attacks was a lot simpler in monorepo than microrepos. Etc. Monolithic version control doesn't have to mean monolith…
Thanks for your answer.
How does CI work in practice? how do you avoid rebuilding and retesting the whole repo at every change? That would be an insane waste of resource and time.