Stripe's Monorepo Developer Environment • Hacker News Board

Re: Stripe's Monorepo Developer Environment

#221

Post by shepwalker » Tue, Aug 20, 2024, 11:41 AM UTC

spacemonkey92 wrote: post #216

shepwalker wrote: post #106

Earlier quoted context omitted.

Hi! I work at Stripe on this. What're you curious about?

When you are working on a large monolith, how exactly do you managed pull request? Are they split into git-sub modules/ frameworks? Curious to know the development code review flow.

PR's are not split into submodules/frameworks for distinct review purposes. Functionally though, right now we have three distinct monorepos (not very 'mono' but we're working on it!), that represent our three main development stacks.

In our PR tooling, there's nothing that enforces/encourages scoping changes to a specific subset of any given repo. We generally encourage smaller PRs as a best practice, but a huge chunk of the benefit of a moonorepo is folks can make cross-cutting changes if/when necessary.

We have some custom goo on top of Github that manages the review flow. Specifically, we try to make it easy for folks to farm out reviews to the teams that own the code you're editing, so they don't have to hop in Slack and track down reviewers.

Re: Stripe's Monorepo Developer Environment

#222

Post by [deleted] » Tue, Aug 20, 2024, 1:04 PM UTC

oftenwrong wrote: post #208

jrochkind1 wrote: post #183

Earlier quoted context omitted.

The argument would be that for simple organization, dividing things into independently releasable components is less simple than just having one app. I think that's what most simple organizations do, no? Why do you need the complexity of independently releasable components for your simple organization? Now you have to track compatibility between things, ensure what version of what independently releasable thing works…

The monorepo vs many-repos discussion often hits upon so many implied factors, but it's only really about how source code is stored. It doesn't necessarily indicate much about the deployment model. You can have many separately releaseable things in one repo, and you can have one independently releaseable thing based on the sources of many repos. Monorepos enable, but don't require, source-level co-evolution. Or maybe…

[deleted]

Re: Stripe's Monorepo Developer Environment

#223

Post by MetaWhirledPeas » Tue, Aug 20, 2024, 1:30 PM UTC

hamandcheese wrote: post #212

MetaWhirledPeas wrote: post #182

Earlier quoted context omitted.

Wouldn't this be a reasonable alternative? Asking because I don't have experience with this. 1. New shared builds update container images for applications that comprise the environment 2. Rather than a "devbox", devs use something like Docker Compose to utilize the images locally. Presumably this would be configured identically to the proposed devbox, except with something like a volume pointing to local code. I'm in…

At least on a MacBook, docker is still a compromise in many ways since it has to run on a Linux VM (I live in the SF tech bubble where I've only ever been issued a MacBook). Even on my personal Linux desktop, I don't love developing in containers. It is very tedious to context switch between my local environment and the in-container environment, and I don't even consider myself the type with a super personalized setu…

> It is very tedious to context switch between my local environment and the in-container environment

I think, with my proposed setup, you'd still do development on your local machine. The containers would only be there for the dependencies, and as a shell to execute your code. The container hosting the application under development would use a volume to point to the files on your local machine, but the container itself (with all its permissions and configuration) would match or nearly match what you plan for the production environment.

Re: Stripe's Monorepo Developer Environment

#224

Post by crabbone » Tue, Aug 20, 2024, 1:37 PM UTC

aidos wrote: post #148

crabbone wrote: post #94

Earlier quoted context omitted.

Working in a configuration where your development environment isn't on your computer is always a huge downgrade. Work with VM? -- sooner or later you'll have problems with forwarding your keyboard input to the VM. Work with containers? -- no good way to save state, no good way to guarantee all containers are in sync etc. God forbid any sort of Web browser-based solution. The number of times I accidentally closed the…

Sounds like you’re talking about something else (more like the Citrix / virtual desktop type model - I don’t know the name). The idea here is that you use a VM (cloud or local) to run your compute. Most people can run it in the background without explicitly connecting to it.

What you describe isn't a development process in a remote environment. You are testing on some remote compute resource. Testing is a non-essential part of development, so, in a sense it "doesn't count" that you test somewhere else -- you cannot call it "developing in a remote environment".

Otherwise, you could say that, for example, reading documentation on a Web page you are doing "development in a remote environment" because, well... most likely that Web page isn't hosted on your laptop.

The essential and mandatory part of development is that a program is written. If you write the program on your laptop, you aren't doing "remote development", no matter where other tools you use for development are running.

Re: Stripe's Monorepo Developer Environment

#225

Post by oceanplexian » Tue, Aug 20, 2024, 1:41 PM UTC

Lio wrote: post #218

ahtihn wrote: post #181

Earlier quoted context omitted.

How is a mono repo the simple solution compared to one repo per independently releasable component ? All the tooling is much easier to use when each application has its own repo.

> independently releasable component I think this is the key phrase in what you've written. Quite often I've seen teams insist on separating things that cannot be released independently due to some form of coupling. You end up with people talking about a particular "release" but not really knowing 100% what's in it and then discovering later that something is missing or included by mistake. IMHO it's much easier to k…

> You end up with people talking about a particular "release" but not really knowing 100% what's in it and then discovering later that something is missing or included by mistake.

If your devs couldn’t be bothered to pin versions that was never a tooling problem. You don’t need a 500GB Git repository with every vendored component to know what’s in your code.

Re: Stripe's Monorepo Developer Environment

#226

Post by crabbone » Tue, Aug 20, 2024, 1:44 PM UTC

otabdeveloper4 wrote: post #105

crabbone wrote: post #98

Earlier quoted context omitted.

Not even once did I want to share my dev. environment, nor did anyone want to share mine. We are talking about 25-odd years of being a developer. Never in my life did I want to scale my dev. environment vertically or horizontally or in any other direction. Unless you work on a calculator, I don't know why would you need that. I have no problems with my environment stopping when I close my laptop. Why is this a proble…

> For overwhelming majority of programming projects out there they fit on a programmer's laptop just fine. What? No. You live in a very sheltered world, my friend.

Talking form a perspective of someone who worked at Google and one other similar company that shell remain nameless... as well as simply looking at places like Github where people tend to post projects they are working on: I don't know of any Github project that would be even in the size range to cause any discomfort for a laptop user.

Even when it comes to the larger projects: I have multiple checkouts of GCC and Linux kernel on my laptop, and when I run du their existence doesn't even register in the first dozen results... Of course, proprietary projects tend to be on a bigger side due to putting a lot of not-strictly code-related stuff in a repository, but still... it would have to be billions LoC big to be prohibitively big for a typical laptop.

Re: Stripe's Monorepo Developer Environment

#227

Post by bhuga » Tue, Aug 20, 2024, 2:51 PM UTC

codethief wrote: post #201

bhuga wrote: post #166

Earlier quoted context omitted.

> I’d anticipate you would be equally as able to ssh to VMs on dev laptops. That’s definitely a prerequisite for making this work in the same way as you’re currently doing. Our laptops don't receive connections, but even if they could, folks go on leave and turn them off for 9 months at a time, or they don't get updated for whatever reason, or other nutty stuff. It's surprisingly common with a few thousand of them ou…

Maybe I'm missing something here but couldn't you just track the whole VM setup (dependencies, dev tools, telemetry and everything) in your monorepo? That is, the VM config would get pulled from master just like everything else, and then the developer would use something like nixos-shell[0] to quickly fire up a VM based on that config that they pulled. [0]: https://github.com/Mic92/nixos-shell (not to be confused wit…

Yes, but this still "freezes" the VM when the user creates it, and I've got no tools to force the software running in it to be updated. It's important that boxes can be updated, not just reliably created.

As just one reason why, many developers need to set up complex test data. We have tools to help with that, but they take time to run and each team has their own needs, so some of them still have manual steps when creating a new dev server. These devs tend to re-use their servers until our company-wide max age. Others, to be fair, spin up a new machine for every branch, multiple times per day, and spinning up a new VM might not be burdensome for them.

Re: Stripe's Monorepo Developer Environment

#228

Post by crabbone » Tue, Aug 20, 2024, 3:05 PM UTC

marcosdumay wrote: post #149

crabbone wrote: post #98

Earlier quoted context omitted.

Not even once did I want to share my dev. environment, nor did anyone want to share mine. We are talking about 25-odd years of being a developer. Never in my life did I want to scale my dev. environment vertically or horizontally or in any other direction. Unless you work on a calculator, I don't know why would you need that. I have no problems with my environment stopping when I close my laptop. Why is this a proble…

I get a serious feeling that interpreted languages, monorepos, environment orchestration, snapshot ecosystem aggregators, and per-function execution evironments are all pushing software development into the wrong direction. Those things are not bad by themselves. But people tend to do bad things with them, and those bad things spread remarkably well, disrupting every place they infect.

I'm not sure why monorepos are in the list. Care to elaborate?

I've worked on projects that used a single repository for all the code written by different departments and projects where the same department could have multiple repositories. The later added insane amount of busy work, inordinate amount of errors, difficulty to investigate failures, excessive use of resources to house various permutations of systems created at different times with different combinations of components. The day-to-day in such projects could be described by developers waiting for the infra people to sort out the morning problems which mysteriously broke everything all at once so that no progress can be made.

This was in stark contrast to companies working on a single repository, where days when nothing worked would happen maybe once or twice a year.

I also lived through transitions from multiple repositories to a single repository and the other way around. In operational terms, I've never seen any beneficial effects of splitting a repository. Not in the short, nor in the long term. Complexity always went up, productivity went down, general satisfaction with project infrastructure would also go down with such a change. Department would start attacking and blaming the infra people for creating obstacles to their progress (while never explicitly mentioning the split repository because, usually that was a decision made by the same people complaining).

Re: Stripe's Monorepo Developer Environment

#229

Post by Lio » Tue, Aug 20, 2024, 5:02 PM UTC

oceanplexian wrote: post #225

Lio wrote: post #218

Earlier quoted context omitted.

> independently releasable component I think this is the key phrase in what you've written. Quite often I've seen teams insist on separating things that cannot be released independently due to some form of coupling. You end up with people talking about a particular "release" but not really knowing 100% what's in it and then discovering later that something is missing or included by mistake. IMHO it's much easier to k…

> You end up with people talking about a particular "release" but not really knowing 100% what's in it and then discovering later that something is missing or included by mistake. If your devs couldn’t be bothered to pin versions that was never a tooling problem. You don’t need a 500GB Git repository with every vendored component to know what’s in your code.

Equally, if your team is going to store 500GB vendored components it doesn't matter if that's all in one place or smeared many repos. You still have the same issue.

Re: Stripe's Monorepo Developer Environment

#230

Post by marcosdumay » Tue, Aug 20, 2024, 5:18 PM UTC

crabbone wrote: post #228

marcosdumay wrote: post #149

Earlier quoted context omitted.

I get a serious feeling that interpreted languages, monorepos, environment orchestration, snapshot ecosystem aggregators, and per-function execution evironments are all pushing software development into the wrong direction. Those things are not bad by themselves. But people tend to do bad things with them, and those bad things spread remarkably well, disrupting every place they infect.

I'm not sure why monorepos are in the list. Care to elaborate? I've worked on projects that used a single repository for all the code written by different departments and projects where the same department could have multiple repositories. The later added insane amount of busy work, inordinate amount of errors, difficulty to investigate failures, excessive use of resources to house various permutations of systems cre…

Oh, yeah, all of those issues of enforcing transitive dependencies that need busy work to update, fluid APIs that make all the code around it break, lack of semantic boundaries that make it hard to decide if a problem is local, inter-component interference so that you have to select them perfectly well...

All of those are enabled by monorepos. And once people learn to do them, they seem to want to apply everywhere.