Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stripe's Monorepo Developer Environment

blog.nelhage.com

Post reply on HN
Print view View original

121–130 of 249 posts

Re: Stripe's Monorepo Developer Environment

#121

Post by adamdecaf »

We’ve been using a hundred repositories and a hundred Go services in a local docker-compose setup that’s worked fairly well. CI runners can struggle if their disks can’t keep up with Docker.

It comes up that we should make a devprod for front end folks to make the backend abstracted more.

Overall a lot of people prefer local dev because it gives them access to the entire stack, lets them run branch images easier, and has better performance than remote boxes.

https://moov.io/blog/education/moovs-approach-to-setup-and-t...

Re: Stripe's Monorepo Developer Environment

#122

Post by drited »

aidos wrote: post #112
bhuga wrote: post #81

Earlier quoted context omitted.

I work on this at Stripe. There's a lot of reasons: * Local dev has laptop-based state that is hard to keep in sync for everyone. Broken laptops are _really hard_ to debug as opposed to cloud servers I can deploy dev management software to. I can safely say the oldest version of software that's in my cloud; the laptops skew across literally years of versions of dev tools despite a talented corpeng team managing them.…

I want to double check we’re talking about the same thing here. I’m referring to running everything inside a single VM that you would have total access to. It could have telemetry, you’d know versions etc. I wonder if there’s some confusion around what I’m suggesting given your points above. I’m sure there are a bunch of things that make it the right choice for Stripe. Obviously if you just have too many things to ru…

I see in another comment thread you mentioned downloading the VM iso, presumably from a central source. Your comment in this thread didn't mention that so perhaps this answer (incorrectly) assumes the VM you are talking about was locally maintained/created?

Re: Stripe's Monorepo Developer Environment

#123

Post by bhuga »

aidos wrote: post #112
bhuga wrote: post #81

Earlier quoted context omitted.

I work on this at Stripe. There's a lot of reasons: * Local dev has laptop-based state that is hard to keep in sync for everyone. Broken laptops are _really hard_ to debug as opposed to cloud servers I can deploy dev management software to. I can safely say the oldest version of software that's in my cloud; the laptops skew across literally years of versions of dev tools despite a talented corpeng team managing them.…

I want to double check we’re talking about the same thing here. I’m referring to running everything inside a single VM that you would have total access to. It could have telemetry, you’d know versions etc. I wonder if there’s some confusion around what I’m suggesting given your points above. I’m sure there are a bunch of things that make it the right choice for Stripe. Obviously if you just have too many things to ru…

> I’m referring to running everything inside a single VM that you would have total access to. It could have telemetry, you’d know versions etc. I wonder if there’s some confusion around what I’m suggesting given your points above.

I don't think there's confusion. I only have total access when the VM is provisioned, but I need to update the dev machine constantly.

Part of what makes a VM work well is that you can make changes and they're sticky. Folks will edit stuff in /etc, add dotfiles, add little cron jobs, build weird little SSH tunnels, whatever. You say "I can know versions", but with a VM, I can't! Devs will run update stuff locally.

As the person who "deploys" the VM, I'm left in a weird spot after you've made those changes. If I want to update everyone's VM, I blow away your changes (and potentially even the branches you're working on!). I can't update anything on it without destroying it.

In constrast, the dev servers update constantly. There's a dozen moving parts on them and most of them deploy several times a day without downtime. There's a maximum host lifetime and well-documented hooks for how to customize a server when it's created, so it's clear how devs need to work with them for their customizations and what the expectations are.

I guess its possible you could have a policy about when the dev VM is reset and get developers used to it? But I think that would be taking away a lot of the good parts of a VM when looking at the tradeoffs.

> What’s the size of the cloud instances you have to run on?

We have a range of options devs can choose, but I don't think any of them are smaller than a high-end laptop.

Re: Stripe's Monorepo Developer Environment

#124

Post by tmpz22 »

draw_down wrote: post #101
p-o wrote: post #93

It's always so enlightening to have articles like this one shed light on how companies at scale operate. It goes without saying that many of the problems Stripe faced with their monorepo isn't application to smaller businesses, but there are still bits and pieces that are applicable to many of us. I've been working on an ephemeral/preview environment operator for Kubernetes( https://github.com/pier-oliviert/sequencer…

Right, dev boxes do not need to do double duty as a personal computer plus development target, which allows them to more closely resemble the machine your code will actually run on. They also can be replaced easily, which can be helpful if you ever suspect something is wrong with the box itself - if the new one acts the same way, it wasn't the dev box. I don't recall latency being a big problem in practice. In an org…

I feel like we're not getting the right lessons from this. It feels like we're focusing on HOW we can do something versus pausing for a brief moment to consider if we SHOULD in the first place.

To me the root issue is the complexity of production environments has expanded to the point of impacting complexity in developer environments just to deploy or test - this is in conjunction with expanding complexity of developer environments just to develop - i.e. web pack.

For very large well resourced organizations like Stripe that actually operate at scale that complexity may very well be unavoidable. But most organizations are not Stripe. They should consider decreasing complexity instead of investing in complex tooling to wrangle it.

I'd go as far as to suggest both monorepos and dev-boxes are complex toolchains that many organizations should consider avoiding.

Re: Stripe's Monorepo Developer Environment

#125

Post by froydnj »

froydnj wrote: post #73
KolmogorovComp wrote: post #4

> In addition, Stripe’s monorepo was (to our knowledge) the largest Ruby codebase in existence Bigger than shoppify's?

The most recent publicly available numbers (that I know of, maybe there's a talk available somewhere that's more recent) are from https://stripe.com/blog/sorbet-stripes-type-checker-for-ruby > currently amounting to over 15 million lines of code spread across 150,000 files The monorepo has only gotten bigger over the last two years (source: I work at Stripe).

I should also note that number is Ruby files only.

Re: Stripe's Monorepo Developer Environment

#126

Post by epinephrinios »

tmpz22 wrote: post #124
draw_down wrote: post #101

Earlier quoted context omitted.

Right, dev boxes do not need to do double duty as a personal computer plus development target, which allows them to more closely resemble the machine your code will actually run on. They also can be replaced easily, which can be helpful if you ever suspect something is wrong with the box itself - if the new one acts the same way, it wasn't the dev box. I don't recall latency being a big problem in practice. In an org…

I feel like we're not getting the right lessons from this. It feels like we're focusing on HOW we can do something versus pausing for a brief moment to consider if we SHOULD in the first place. To me the root issue is the complexity of production environments has expanded to the point of impacting complexity in developer environments just to deploy or test - this is in conjunction with expanding complexity of develop…

Absolutely, I worked on tech behemoths and smaller companies. The dev experience was significantly better when all development was local. I even worked on initiatives to move development away from the cloud, and although other devs were skeptical, they ended up loving it.

Re: Stripe's Monorepo Developer Environment

#127

Post by aidos »

bhuga wrote: post #123
aidos wrote: post #112

Earlier quoted context omitted.

I want to double check we’re talking about the same thing here. I’m referring to running everything inside a single VM that you would have total access to. It could have telemetry, you’d know versions etc. I wonder if there’s some confusion around what I’m suggesting given your points above. I’m sure there are a bunch of things that make it the right choice for Stripe. Obviously if you just have too many things to ru…

> I’m referring to running everything inside a single VM that you would have total access to. It could have telemetry, you’d know versions etc. I wonder if there’s some confusion around what I’m suggesting given your points above. I don't think there's confusion. I only have total access when the VM is provisioned, but I need to update the dev machine constantly. Part of what makes a VM work well is that you can make…

So the devs don’t have the ability to ssh to your cloud instances and change config? Other than the size issue, I’m still not seeing the difference. Take your point on it needing to start before you have control, but other than that a VM on a dev machine is functionally the same as one in a cloud environment.

In terms of needing to reset, it’s just a matter of git branch, push, reset, merge. In your world that sync complexity happens all the time, in mine just on reset.

Just to be clear, I think it’s interesting to have a healthy discussion about this to see where the tradeoffs are. Feels like the sort of thing where people try to emulate you and buy themselves a bunch of complexity where other options are reasonable.

I have no doubt Stripe does what makes sense for Stripe. I’d also wager than on balance it’s not the best option for most other teams.

PS thanks for chiming in. I appreciate the extra insights and context.

Re: Stripe's Monorepo Developer Environment

#128

Post by draw_down »

tmpz22 wrote: post #124
draw_down wrote: post #101

Earlier quoted context omitted.

Right, dev boxes do not need to do double duty as a personal computer plus development target, which allows them to more closely resemble the machine your code will actually run on. They also can be replaced easily, which can be helpful if you ever suspect something is wrong with the box itself - if the new one acts the same way, it wasn't the dev box. I don't recall latency being a big problem in practice. In an org…

I feel like we're not getting the right lessons from this. It feels like we're focusing on HOW we can do something versus pausing for a brief moment to consider if we SHOULD in the first place. To me the root issue is the complexity of production environments has expanded to the point of impacting complexity in developer environments just to deploy or test - this is in conjunction with expanding complexity of develop…

[dead]

Re: Stripe's Monorepo Developer Environment

#129

Post by JasonSage »

adamdecaf wrote: post #118
recroad wrote: post #100

Earlier quoted context omitted.

If you have 100 services in your org, I don't have to have 100 running at the same time in your local dev machine. I only run the 5 I need for the feature I'm working on.

We have 100 Go services (with redpanda) and a few databases in docker-compose on dev laptops. It works well when and we buy the biggest memory MacBooks available. https://moov.io/blog/education/moovs-approach-to-setup-and-t...

Your success with this strategy correlates more strongly with ‘Go’ than ‘100 services’ so it’s more anecdotal than generally-acceptable that you can run 100 services locally without issues. Of course you can.

Buying the biggest MacBook available as a baseline criteria for being able to run a stack locally with Docker Compose does not exactly inspire confidence.

At my last company we switched our dev environment from Docker Compose to Nix on those same MacBooks and CPU usage when from 300% to <10% overnight.

Re: Stripe's Monorepo Developer Environment

#130

Post by dheera »

aidos wrote: post #59

Maybe a silly question, but why all this engineering effort when you could host the dev environment locally? By running a Linux VM on your local machine you get a consistent environment that you can ssh to, remove the latency issues but you remove all the complexity of syncing that they’ve created. That’s a setup that’s worked well for me for 15 years but maybe I’m missing some other benefit?

> By running a Linux VM

Or just run Linux on your local machine as the OS. I don't get the obsession with Macs as dev workstations for companies whose products run on Linux.

Post reply on HN