Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stripe's Monorepo Developer Environment

blog.nelhage.com

Post reply on HN
Print view View original

181–190 of 249 posts

Re: Stripe's Monorepo Developer Environment

#181

Post by ahtihn »

jrochkind1 wrote: post #161
tmpz22 wrote: post #124

Earlier quoted context omitted.

I feel like we're not getting the right lessons from this. It feels like we're focusing on HOW we can do something versus pausing for a brief moment to consider if we SHOULD in the first place. To me the root issue is the complexity of production environments has expanded to the point of impacting complexity in developer environments just to deploy or test - this is in conjunction with expanding complexity of develop…

> I'd go as far as to suggest both monorepos and dev-boxes are complex toolchains that many organizations should consider avoiding. I'm not sure "monorepo" means the same thing to you as it does to me? To me, it just means "keep all the code in one repo, instead of trying to split things up into different repos." To me, it's the thing that is the simple solution, it just means "a repo" -- the reason it gets a name is…

How is a mono repo the simple solution compared to one repo per independently releasable component ?

All the tooling is much easier to use when each application has its own repo.

Re: Stripe's Monorepo Developer Environment

#182

Post by MetaWhirledPeas »

hamandcheese wrote: post #164
p-o wrote: post #93

It's always so enlightening to have articles like this one shed light on how companies at scale operate. It goes without saying that many of the problems Stripe faced with their monorepo isn't application to smaller businesses, but there are still bits and pieces that are applicable to many of us. I've been working on an ephemeral/preview environment operator for Kubernetes( https://github.com/pier-oliviert/sequencer…

My main gripe with the dev box approach is that a cloud instance with similar compute resources as a developers MacBook is hella expensive. Even ignoring compute, a 1TB ebs volume with equivalent performance to a MacBook will probably cost more than the MacBook every month.

Wouldn't this be a reasonable alternative? Asking because I don't have experience with this.

1. New shared builds update container images for applications that comprise the environment

2. Rather than a "devbox", devs use something like Docker Compose to utilize the images locally. Presumably this would be configured identically to the proposed devbox, except with something like a volume pointing to local code.

I'm interested in learning more about this. It seems like a way to get things done locally without involving too many cloud services. Is this how most people do it?

Re: Stripe's Monorepo Developer Environment

#183

Post by jrochkind1 »

ahtihn wrote: post #181
jrochkind1 wrote: post #161

Earlier quoted context omitted.

> I'd go as far as to suggest both monorepos and dev-boxes are complex toolchains that many organizations should consider avoiding. I'm not sure "monorepo" means the same thing to you as it does to me? To me, it just means "keep all the code in one repo, instead of trying to split things up into different repos." To me, it's the thing that is the simple solution, it just means "a repo" -- the reason it gets a name is…

How is a mono repo the simple solution compared to one repo per independently releasable component ? All the tooling is much easier to use when each application has its own repo.

The argument would be that for simple organization, dividing things into independently releasable components is less simple than just having one app. I think that's what most simple organizations do, no? Why do you need the complexity of independently releasable components for your simple organization? Now you have to track compatibility between things, ensure what version of what independently releasable thing works with what version of what independently releasable other thing, isn't that added complexity? Why not just have one application, isn't that simpler? You don't need to worry about incompatibilities between your separately releasable things -- every commit that passes CI on your single repo means all the parts are compatible (sans untested bugs).

Usually it stops being "simpler" at a level of organizational complexity or code size where it becomes a mess. The "monorepo" is the attempt to do what everyone was just doing anyway for simple orgs with simple codebases, but keep doing it at huge sizes.

Re: Stripe's Monorepo Developer Environment

#184

Post by zten »

ahtihn wrote: post #181
jrochkind1 wrote: post #161

Earlier quoted context omitted.

> I'd go as far as to suggest both monorepos and dev-boxes are complex toolchains that many organizations should consider avoiding. I'm not sure "monorepo" means the same thing to you as it does to me? To me, it just means "keep all the code in one repo, instead of trying to split things up into different repos." To me, it's the thing that is the simple solution, it just means "a repo" -- the reason it gets a name is…

How is a mono repo the simple solution compared to one repo per independently releasable component ? All the tooling is much easier to use when each application has its own repo.

If you're living in the same dysfunctional world I am, then maybe your organization split things into repos that are separately releasable, but are conceptually so strongly coupled that you now need to create changes on 3 repos to make a change.

Re: Stripe's Monorepo Developer Environment

#185

Post by DiggyJohnson »

MetaWhirledPeas wrote: post #182
hamandcheese wrote: post #164

Earlier quoted context omitted.

My main gripe with the dev box approach is that a cloud instance with similar compute resources as a developers MacBook is hella expensive. Even ignoring compute, a 1TB ebs volume with equivalent performance to a MacBook will probably cost more than the MacBook every month.

Wouldn't this be a reasonable alternative? Asking because I don't have experience with this. 1. New shared builds update container images for applications that comprise the environment 2. Rather than a "devbox", devs use something like Docker Compose to utilize the images locally. Presumably this would be configured identically to the proposed devbox, except with something like a volume pointing to local code. I'm in…

I manage a dev environment for a small, inexperienced (but eager) team and I have a similar setup. I’ll do a write up at some point if I have time. It can work, and does for me, but there are some funny consequences can end up mediate the relationship between a developer’s computer and his code, which is a terrible place to be.

Re: Stripe's Monorepo Developer Environment

#186

Post by numbsafari »

ParetoOptimal wrote: post #165
numbsafari wrote: post #150

Earlier quoted context omitted.

This is my biggest complaint with GitHub CodeSpaces. I should be able to launch a local VM using the GitHub Desktop App just as easily as I can an Azure-hosted instance.

But then how will they lock you into paying them monthly?

They’re just forcing me to stop paying them altogether.

Re: Stripe's Monorepo Developer Environment

#187

Post by tmpz22 »

jrochkind1 wrote: post #161
tmpz22 wrote: post #124

Earlier quoted context omitted.

I feel like we're not getting the right lessons from this. It feels like we're focusing on HOW we can do something versus pausing for a brief moment to consider if we SHOULD in the first place. To me the root issue is the complexity of production environments has expanded to the point of impacting complexity in developer environments just to deploy or test - this is in conjunction with expanding complexity of develop…

> I'd go as far as to suggest both monorepos and dev-boxes are complex toolchains that many organizations should consider avoiding. I'm not sure "monorepo" means the same thing to you as it does to me? To me, it just means "keep all the code in one repo, instead of trying to split things up into different repos." To me, it's the thing that is the simple solution, it just means "a repo" -- the reason it gets a name is…

> To me, it just means "keep all the code in one repo, instead of trying to split things up into different repos."

To me, and perhaps more from a Devops-like perspective, mono repo means "one repo many diverse deployment environments and artifacts often across multiple programming languages".

Im advocating against the Google/Stripe situation of a singular massive repo with complex build tools to make it function - like Bazel. I think sometimes small organizations get lured by ego and bad cost/benefit analysis into implementing such an architecture and it can tank entire product orgs in my experience (obviously not for Stripe, Google, etc.).

Re: Stripe's Monorepo Developer Environment

#188

Post by stealthybox »

This is an awesome writeup of the tools and culture issues you run into maintaining dev environments.

From post, the problems that justified central dev boxes are roughly: 1. dependency / config mgmt / env drift on laptops 2. collaboration / debugging between engineers 3. compute scaling + optimization 4. supporting devs with updates and infra changes

The last one is particularly interesting to me, because supporting the dev env is separate engineering role/task that starts small and grows into teams of engineers supporting the environment.

I'm helping build Flox. We're working on these pain points by making environments (deps, vars, services, and builds) workable across all kinds of Mac/Linux laptops and servers. 1) a. Virtualize the pkg manager per-project b. Nix packages can install across OS/arch pretty well 2) Imperative actions like `flox install`/`upgrade` always edit a declarative env manifest.toml -- share it via git 3) less Docker VM's -- get more out of devteam Macbooks 4) reduce toil with a versioned, shareable envs --> less sending ad-hoc config and brew commands to people (as mentioned in the post.) Just `git pull && flox activate`.

I think on problem point #2, collab tools are advancing to where, pairing on features, bugs, and env issues can be done without central SSH. (ex: tmate, vscode liveshare, screensharing, etc) -- however, that does sort of fall apart on laptops for async debugging of env issues (ex: when devprod is in the US, and eng is in London). Having universal telemetry on ephemeral cloud dev-boxes with a registry and all of the other DNS and SSH goodies could be the kind of infra to aspire to as your small teams run into more big-team problems.

In the Stripe anecdote, adopting the centralized infra created new challenges that their devprod teams were dedicated to supporting: - international latency from central, US-based VM's - syncing code to the dev boxes (https://facebook.github.io/watchman/) - linting, formatting, generating configs (run it locally or serverside?) - a dev workflow CLI tool dedicated to dev-box workflows and sync'ing with watchman's clock - IaaS, registry, config, glue for all the servers

This is all very non-trivial work, but maybe there's a future where people can win some portability with Flox when they are small and grow into those new challenges when it's truly needed -- now their laptop environments just get a quick `flox activate` on some new, shiny servers or Cloud IDE's.

I really like the notes from the author on how useing Language Server Protocol across a high latency link has great optimizations that work along side the watchman sync for real-time code editing.

Re: Stripe's Monorepo Developer Environment

#189

Post by vfclists »

How does a payment service wind up with over a 1000 engineers?

I understand that "engineers" may not mean "developers", it could DevOps, site reliability and all the bits and pieces that make up a large service provider, but over a 1000?

Can someone please enlighten me?

Re: Stripe's Monorepo Developer Environment

#190

Post by mvdtnz »

eikenberry wrote: post #135
bhuga wrote: post #110

Earlier quoted context omitted.

Some important differences from 2019: * Code is off of laptops and lives entirely on the dev server in many (but not all) cases. This has opened up a lot of use cases where devs can have multiple branches in flight at once. * Big investments into bazel. * Heavier investment into editor experiences. We find most developers are not as idiosyncratic in their editor choices as is commonly believed, and most want a pre-co…

That last point has long been a red flag when interviewing. A developer who doesn't care about their tooling also tends to not care about the quality of their work.

I'd rather work with developers who are flexible and open minded about the conditions they can work in than those who get notoriously pissy if things aren't set up exactly the way they like it. Especially when that way is ridiculously elaborate and non-standard.
Post reply on HN