Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Everything as code: How we manage our company in one monorepo

kasava.dev

Post reply on HN
Print view View original

211–220 of 232 posts

Re: Everything as code: How we manage our company in one monorepo

#211

Post by array_key_first »

suralind wrote: post #209
array_key_first wrote: post #207

Earlier quoted context omitted.

They do, it's just instead of it being a library call it's a network call usually, which is even worse. Makes it nigh impossible to refactor your codebase in any meaningful way.

But if you need to rename endpoint for example you need to route service A version Y to compatible version in service B. After changing the endpoint, now you need to route service A version Z to a new version of service B. Am I missing something? Meaning that it doesn’t truly mater whether you have 1 repo, 2 repos or 10 repos. Deployments MUST be done in sequence and there MUST be a backwards compatible commit in bet…

You just deploy all the services at once, A B style. Just flip to the new services once they're all deployed and make the old ones inactive, in one go. Yes you'll probably need a somewhat central router, maybe you do this per-client or per-user or whatever makes sense.

Re: Everything as code: How we manage our company in one monorepo

#212

Post by eddd-ddde »

RaftPeople wrote: post #198
eddd-ddde wrote: post #180

Earlier quoted context omitted.

Yeah, all new commits are merged to main. The complexity comes from releases. Suppose you have a good commit 123 were all your tests pass for some project, you cut a release, and deploy it. Then development continues until commit 234, but your service is still at 123. Some critical bug is found, and fixed in commit 235. You can't just redeploy at 235 since the in-between may include development of new features that a…

> Suppose you have a good commit 123 were all your tests pass for some project, you cut a release, and deploy it. And you've personally done this for a larger project with significant amount of changes and a longer duration (like maybe 6 months to a year)? I'm struggling to understand why you would eliminate branches? It would increase complexity, work and duration of projects to try to shoehorn 2 different system mo…

Can you clarify why it would impact project duration?

In my experience development branches vastly increase complexity by hiding the integration issues until very late when you try to merge.

Re: Everything as code: How we manage our company in one monorepo

#213

Post by eddd-ddde »

ozozozd wrote: post #190
imiric wrote: post #186

Earlier quoted context omitted.

I don't see how you're avoiding development branches. Surely while a change is in development the author doesn't simply push to main. Otherwise concurrent development, and any code review process—assuming you have one—would be too impractical. So you can say that you have short-lived development branches that are always rebased on main. Along with the release branch and cherry-pick process, the workflow you describe…

Their dev branch is _the_ development branch. They don’t do code reviews or any sort of parallel development. They’re under the impression that “releases are complex and this is how they avoid it” but they just moved the complexity and sacrificed things like parallel work, code reviews, reverts of whole features.

I'm not sure where you got that from. There is a single branch, which obviously has code review, and reverts work just the same way.

What there isn't, is long lived feature branches with non-integrated changes.

Re: Everything as code: How we manage our company in one monorepo

#214

Post by RaftPeople »

eddd-ddde wrote: post #212
RaftPeople wrote: post #198

Earlier quoted context omitted.

> Suppose you have a good commit 123 were all your tests pass for some project, you cut a release, and deploy it. And you've personally done this for a larger project with significant amount of changes and a longer duration (like maybe 6 months to a year)? I'm struggling to understand why you would eliminate branches? It would increase complexity, work and duration of projects to try to shoehorn 2 different system mo…

Can you clarify why it would impact project duration? In my experience development branches vastly increase complexity by hiding the integration issues until very late when you try to merge.

The reason I said it would impact duration is the assumption that the previous version and new version of the system are all in the code at one time, managed via feature flags or something. I think I was picturing that due to other comments later in the thread, you may not be handling it that way.

Either way, I still don't understand how you can reasonably manage the complexity, or what value it brings.

Example:

main - current production - always matches exactly what is being executed in production, no differences allowed

production_qa - for testing production changes independent of the big project

production_dev_branches - for developing production changes during big project

big_project_qa_branch - tons of changes, currently being used to qa all of the interactions with this system as well as integrations to multiple other systems internal and external

big_project_dev_branches - as these get finalized and ready for qa they move to qa

Questions:

When production changes and project changes are in direct conflict, how can you possibly handle that if everyone is just committing to one branch?

How do you create a clean QA image for all of the different types of testing and ultimately business training that will need to happen for the project?

Re: Everything as code: How we manage our company in one monorepo

#215

Post by suralind »

array_key_first wrote: post #211
suralind wrote: post #209

Earlier quoted context omitted.

But if you need to rename endpoint for example you need to route service A version Y to compatible version in service B. After changing the endpoint, now you need to route service A version Z to a new version of service B. Am I missing something? Meaning that it doesn’t truly mater whether you have 1 repo, 2 repos or 10 repos. Deployments MUST be done in sequence and there MUST be a backwards compatible commit in bet…

You just deploy all the services at once, A B style. Just flip to the new services once they're all deployed and make the old ones inactive, in one go. Yes you'll probably need a somewhat central router, maybe you do this per-client or per-user or whatever makes sense.

So that's blue green with added version aware routing. What if you need to rollback? Good luck I guess.

Re: Everything as code: How we manage our company in one monorepo

#216

Post by array_key_first »

suralind wrote: post #215
array_key_first wrote: post #211

Earlier quoted context omitted.

You just deploy all the services at once, A B style. Just flip to the new services once they're all deployed and make the old ones inactive, in one go. Yes you'll probably need a somewhat central router, maybe you do this per-client or per-user or whatever makes sense.

So that's blue green with added version aware routing. What if you need to rollback? Good luck I guess.

You can do phased deployments with blue green, that's what we do. It depends on your application but ours has a natural segmentation by client. And when you roll back you just flip the active and passive again.

Re: Everything as code: How we manage our company in one monorepo

#217

Post by eddd-ddde »

RaftPeople wrote: post #214
eddd-ddde wrote: post #212

Earlier quoted context omitted.

Can you clarify why it would impact project duration? In my experience development branches vastly increase complexity by hiding the integration issues until very late when you try to merge.

The reason I said it would impact duration is the assumption that the previous version and new version of the system are all in the code at one time, managed via feature flags or something. I think I was picturing that due to other comments later in the thread, you may not be handling it that way. Either way, I still don't understand how you can reasonably manage the complexity, or what value it brings. Example: main…

It depends a lot on a team by team basis as different teams would like different approaches

In general, all new code gets added to the tip of main, your only development branch. Then, new features can also be behind feature flags optionally. This allows developers to test and develop on the latest commit. They can enable a flag if they are interested in a particular feature. Ideally new code also comes with relevant automated tests just to keep the quality of the branch high.

Once a feature is "sufficiently tested" whatever that may mean for your team it can be enabled by default, but it won't be usable until deployed.

Critically, there is CI that validates every commit, _but_ deployments are not strictly performed from every commit. Release processes can be very varied.

A simple example is we decide to create a release from commit 123, which has some features enabled. You grab the code, build it, run automated tests, and generate artifacts like server binaries or assets. This is a small team with little SLAs so it's okay to trust automated tests and deploy right to production. That's the end, commit 123 is live.

As another example, a more complex service may require more testing. You do the same first steps, grab commit 123, test, build, but now deploy to staging. At this point staging will be fixed to commit 123, even as development continues. A QA team can perform heavy testing, fixes are made to main and cherry picked, or the release dropped if something is very wrong. At some point the release is verified and you just promote it to production.

So development is always driven from the tip of the main branch. Features can optionally be behind flags. And releases allow for as much control as you need.

There's no rule that says you can only have one release or anything like that. You could have 1 automatic release every night if you want to.

Some points that make it work in my experience are:

1. Decent test culture. You really want to have at least some metric for which commits are good release candidates. 2. You'll need some real release management system. The common tools available like to tie together CI and CD which is not the right way to think about it IMO (example your GitHub CI makes a deployment).

TL:Dr:

Multiple releases, use flags or configuration for the different deployments. They could all even be from the same or different commits.

Re: Everything as code: How we manage our company in one monorepo

#218

Post by eptcyka »

mjr00 wrote: post #34
sethammons wrote: post #17

people talk about "one change, everywhere, all at once." That is a great way to break production on any api change. if you have a db and >2 nodes, you will have the old system using the old schema and the new system using the new schema unless you design for forwards-backwards compatible changes. While more obvious with a db schema, it is true for any networked api. At some point, you will have many teams. And one of…

100%, this is all true and something you have to tackle eventually. Companies like this one (Kasava) can get away with it because, well, they likely don't have very many customers and it doesn't really matter. But when you're operating at a scale where you have international customers relying on your SaaS product 24/7, suddenly deploys having a few minutes of downtime matters. This isn't to say monorepo is bad, thoug…

The only way to do a sane migration without downtime is to have tha application handle both schema versions at the same time. This is easily doable with a monorepo.

Re: Everything as code: How we manage our company in one monorepo

#219

Post by RaftPeople »

eddd-ddde wrote: post #217
RaftPeople wrote: post #214

Earlier quoted context omitted.

The reason I said it would impact duration is the assumption that the previous version and new version of the system are all in the code at one time, managed via feature flags or something. I think I was picturing that due to other comments later in the thread, you may not be handling it that way. Either way, I still don't understand how you can reasonably manage the complexity, or what value it brings. Example: main…

It depends a lot on a team by team basis as different teams would like different approaches In general, all new code gets added to the tip of main, your only development branch. Then, new features can also be behind feature flags optionally. This allows developers to test and develop on the latest commit. They can enable a flag if they are interested in a particular feature. Ideally new code also comes with relevant…

> As another example, a more complex service may require more testing. You do the same first steps, grab commit 123, test, build, but now deploy to staging. At this point staging will be fixed to commit 123, even as development continues. A QA team can perform heavy testing, fixes are made to main and cherry picked, or the release dropped if something is very wrong. At some point the release is verified and you just promote it to production.

But how would you create that QA environment when it involves thousands of commits over a 6 month period?

Re: Everything as code: How we manage our company in one monorepo

#220

Post by yencabulator »

array_key_first wrote: post #205
Eridrus wrote: post #59

Earlier quoted context omitted.

Atomic changes are a lie in the sense that there is no atomic deployment of a repo. The moment you have two production services that talk to each other, you end up with one of them being deployed before the other.

If you have a monolith you get atomic deployment, too.

You lose atomic deployment and have a distributed system the moment you ship Javascript to a browser.

Hell, you lose "atomic" assets the moment you serve HTML that has URLs in it.

Consider switching from to . If you for example, delete kitty from the server and upload puppy.jpg, then change html, you can have a client with URL to kitty while kitty is already gone. Generally anything you published needs to stay alive for long enough to "flush out the stragglers".

Same thing applies to RPC contracts.

Same thing applies to SQL schema changes.

Post reply on HN