Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stripe's Monorepo Developer Environment

blog.nelhage.com

Post reply on HN
Print view View original

81–90 of 249 posts

Re: Stripe's Monorepo Developer Environment

#81

Post by bhuga »

aidos wrote: post #59

Maybe a silly question, but why all this engineering effort when you could host the dev environment locally? By running a Linux VM on your local machine you get a consistent environment that you can ssh to, remove the latency issues but you remove all the complexity of syncing that they’ve created. That’s a setup that’s worked well for me for 15 years but maybe I’m missing some other benefit?

I work on this at Stripe. There's a lot of reasons:

* Local dev has laptop-based state that is hard to keep in sync for everyone. Broken laptops are _really hard_ to debug as opposed to cloud servers I can deploy dev management software to. I can safely say the oldest version of software that's in my cloud; the laptops skew across literally years of versions of dev tools despite a talented corpeng team managing them.

* Our cloud servers have a lot more horsepower than a laptop, which is important if a dev's current task involves multiple services.

* With a server, I can get detailed telemetry out of how devs work and what they actually wait on that help me understand what to work on next; I have to have pretty invasive spyware on laptops to do the same.

* Servers in our QA environment can interact with QA services in a way that is hard for a laptop to do. Some of these are "real services", others are incredibly important to dev itself, such as bazel caches.

There's other things; this is an abbreviated list.

If a linux VM works for you, keep working! But we have not been able to scale a thousands-of-devs experience on laptops.

Re: Stripe's Monorepo Developer Environment

#82

Post by shepherdjerred »

nxicvyvy wrote: post #70
mleo wrote: post #65

I use syncthing to manage the synchronization of files between local laptop and remote development server. The software code base is upwards of 20 years and has dependencies on Windows for runtime. I can run unit tests locally on very fast MacBook Pro or run it much slower on Windows VM. With syncthing I can easily edit files locally or remotely and they are available locally for source control. The worst problem is…

Try unison, it's built for this use case. https://www.cis.upenn.edu/~bcpierce/unison/

I like Unison, though I found Mutagen a bit better.

https://mutagen.io/

Re: Stripe's Monorepo Developer Environment

#83

Post by nkohari »

reillys wrote: post #37
Maxious wrote: post #19

Earlier quoted context omitted.

I'd suggest you revise your competitor analysis. Bazel definitely has a test command that with remote execution and caching absolutely allows you to run entire test suites in seconds* both locally and in CI eg. https://blog.aspect.build/typescript-with-rbe

This blog post says 2 and a half minutes not seconds. I know Bazel is a build system which distributes builds among remote machines. In fact using any computer language you can achieve these goals - you just need to program it. So yes you could probably do all the things with all the things, but Basel does not solve this problem out of the box. I wonder why stripe didn’t “just use Bazel”.

Stripe does use Bazel. It just didn't exist before Stripe built some of its own internal systems, but it's gradually replacing ~everything from a build standpoint.

The one thing to know about Bazel is that it's both incredibly impressive, and also one of the least ergonomic pieces of software ever created. It's very clearly an internal project which was cleaned up and open sourced without any attempt to make it more usable outside of Google.

Bazel's kind of like Kubernetes in a way -- you don't actually get enough benefits to adopt it until you're at a certain point in the company lifecycle, and to get to that point you usually have to build other systems first. Then you have to gradually replace those systems with Bazel.

Re: Stripe's Monorepo Developer Environment

#84

Post by truetraveller »

"I’ve described a lot of fairly-involved custom tooling; we needed enough engineers to build and maintain it, and enough “customer” engineers for that investment to pay off."

This is so important when deciding to re-invent the wheel. I've gotten bitten by this many times.

Re: Stripe's Monorepo Developer Environment

#85

Post by jcmfernandes »

nkohari wrote: post #74
delhanty wrote: post #29

>Some caveats: It’s been nearly five years, and I have no doubt that I have misremembered some of the specific details, even though I’m confident in the overall picture. I’m also certain that Stripe has continued evolving and I make no claim this document represents the developer experience at Stripe as of today. Are there any more recently ex-Stripe folks here willing and able to comment on how Stripe's developer en…

I spent 4.5 years at Stripe, and left in March. The biggest difference not mentioned is the article is that code is no longer kept on developer machines. The sync process described in the article was well-designed, but also was a fairly constant source of headaches. (For example, sometimes the file watcher would miss an update and the code on your remote machine would be broken in strange ways, and you'd have to reco…

Thanks for this. Can you share the experience of those who don't use VS Code?

Re: Stripe's Monorepo Developer Environment

#86

Post by esprehn »

jdtig wrote: post #46
bastawhiz wrote: post #41

Earlier quoted context omitted.

It is Ruby but not rails

Thanks. I wonder what the experience is like working on a very large codebase with or without a framework. E.g. Stripe vs Shopify. Or if the framework is barely noticeable at that scale and doesn't really matter anymore. That's the impression I get for Instagram (which was built with Django).

At that scale there's certainly a framework and many in house libraries with opinions and patterns. It's just not rails.

Re: Stripe's Monorepo Developer Environment

#87

Post by underdeserver »

n0us wrote: post #69
aidos wrote: post #59

Maybe a silly question, but why all this engineering effort when you could host the dev environment locally? By running a Linux VM on your local machine you get a consistent environment that you can ssh to, remove the latency issues but you remove all the complexity of syncing that they’ve created. That’s a setup that’s worked well for me for 15 years but maybe I’m missing some other benefit?

You're limited by the resources available to you on your local laptop and when you close that laptop the dev environment stops running. Remote dev environments are more costly and complicated to maintain but they can be shared, can scale vertically (or horizontally) on demand, can persist when you exit them, and managing access to various internal services from dev environments can in some cases be simpler. It also c…

Most local laptops are much stronger than is needed to run the entire stack of your average startup with no resource issues.

And the dev environment stops running when you close the laptop, but you also don't need it since you're not developing.

Not saying it can work for absolutely all cases but it's definitely good enough for a lot of cases.

Re: Stripe's Monorepo Developer Environment

#88

Post by bastawhiz »

jdtig wrote: post #46
bastawhiz wrote: post #41

Earlier quoted context omitted.

It is Ruby but not rails

Thanks. I wonder what the experience is like working on a very large codebase with or without a framework. E.g. Stripe vs Shopify. Or if the framework is barely noticeable at that scale and doesn't really matter anymore. That's the impression I get for Instagram (which was built with Django).

They had their own ORM, and a web framework built on Sinatra. It wasn't as though you needed to reach far for a tool if you needed one

Re: Stripe's Monorepo Developer Environment

#89

Post by anthonypasq »

underdeserver wrote: post #87
n0us wrote: post #69

Earlier quoted context omitted.

You're limited by the resources available to you on your local laptop and when you close that laptop the dev environment stops running. Remote dev environments are more costly and complicated to maintain but they can be shared, can scale vertically (or horizontally) on demand, can persist when you exit them, and managing access to various internal services from dev environments can in some cases be simpler. It also c…

Most local laptops are much stronger than is needed to run the entire stack of your average startup with no resource issues. And the dev environment stops running when you close the laptop, but you also don't need it since you're not developing. Not saying it can work for absolutely all cases but it's definitely good enough for a lot of cases.

... this is an article about Stripe, not your average startup

Re: Stripe's Monorepo Developer Environment

#90

Post by riwsky »

n_ary wrote: post #34
pavlov wrote: post #15

Earlier quoted context omitted.

Meta also has a massive monorepo accessed primarily through cloud devservers. When several of the world’s most successful software companies use this approach, it’s hard to argue that it’s inherently bad. Of course it’s sensible to discuss what lessons apply to smaller companies who don’t have the luxury of dedicated tooling teams supporting the monorepo and dev environment.

Just because some successful companies use some approach doesn't make it the best practice. I have seen firsthand nuisance of monorepo, which took almost 15minutes to correctly switch branches on intel machines(and decently spiked the CPU by causing windows defender to panic). It has decent benefit of easy code sharing, but build and test are soul sucking experiences and if someone decides to run some updated formatt…

“Someone autoformatted the whole thing under new settings at the same time as introducing a new feature” is hardly a monorepo problem. That could be a pain in the ass to review even in a single file. But the flip-side, of someone cleanly wanting to a do a mass autoformat or autorefactor, is much easier in a monorepo than in split repos.
Post reply on HN