41–50 of 108 posts
I find it difficult to imagine why you would need 1000 developers in most single projects. The cost of managing so many people and work streams would seem to overwhelm whatever volume you could get out of them
Did I read it correctly that in first iteration they merged "NOT TESTED" code into master ? Who would ever think it's a good idea ?
It was tested. It's simply that between the time you pushed on your branch, and the time you merged, many other commits made it to the master branch, potentially breaking your branch.
Master should be always in a state of release at any moment.
I just cannot imagine it was an unknown practice for some.
This is exactly the kind of workflow that Bors ( https://github.com/bors-ng/bors-ng ) automates. In addition to Bors, there are a number of apps and services that automate this kind of workflow. Here is an incomplete list: https://forum.bors.tech/t/other-apps-that-implement-the-nrsr... Edited to add: Graydon Hoare (creator of Rust) called this the Not Rocket Science Rule Of Software Engineering (NRSROSE): "automatica…
It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc. Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actua…
Shopify always runs CI on branches before merging to master. Everything this article describes is in addition to that, in order to deal with the problems the article talks about at "merge to master" time, like 2 merged PRs failing or a stale PR that passed on branch but fails on master due changes. At this scale you need to be deploying constantly, otherwise deploys are hundreds of commits large and its impossible to…
What's the minimum residency time to reliably detect problems with my PR? Add deployment time, double to account for jitter caused by humans being humans (forgetful, lunch, meetings, etc), and there probably are not enough hours in the day for 1000 people to be deploying the same monolith.
To increase residency time you can deploy separate units (You can have multiple deployment units even in a monorepo), and those also reduce the surface area of merges.
Honestly what are they doing with 1000 developers? Duplicated effort goes up considerably with a team and codebase of that size. If you forced me to hire that many people, I'd have a lot of them working on open source, trying to steward feature enhancements that help our process. Because otherwise they'd be running around writing proprietary versions of a bunch of shit that already exists and in a better more documented form.
Did I read it correctly that in first iteration they merged "NOT TESTED" code into master ? Who would ever think it's a good idea ?
The branches that are being merged are tested, also in the first version. However, different branches can conflict with each other, and break the master builds. This was happening often enough for us to want to prevent this. The simple way would be to rebase your branch (or merge in master). However, with the amount of changes that are being merged, by the time the CI result comes in for your rebased branch, the tip…
So overall you came to conclusion that what you explained even if can fail is pretty much the best way you can do it.. (rebase just custom..)
Overall DevOps principles are proven to simply work, you just have to follow them..
Earlier quoted context omitted.
Well, the problem is that master is a bottleneck. Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what i…
> Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. Google does it with 50 times the developer count. > At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). True, it is impossible to catch all errors like this, but you can catch a…
The linked page explains the speculative execution aspect, used to ensure every change is tested before merge, with the true state of master at the time of merge despite that state being different than it was when the CI run started ;)
It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc. Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actua…
Hi, Author here! Pull requests are our unit of work, and the queue was created to support all pull requests. We do have feature flags as a tool, but we let our developers make the judgment call on how their changes should be rolled out.
How do you handle the scenario that some developer pushes a send_me_all_the_credit_card_details() function to the code base which does something 'evil'? Do you rely on the reviewer "doing their works properly" to handle that?
I'm not saying formal "signing off"-steps in processes handle it, but some companies does them for that reason.
It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc. Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actua…
Well, the problem is that master is a bottleneck. Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what i…
Microservices are the same kind of solution, with the same gains and costs for this specific problem.
Earlier quoted context omitted.
Well, the problem is that master is a bottleneck. Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what i…
Microservices don't really help with this. They just force you to think about your interfaces, but you should do that in a monolith too. If you interfaces are reasonably stable, merging is unlikely to break master if the branch was green before, if your interfaces change rapidly you get problems with microservices too, just one level higher up, where you try to integrate them into a usable product.
In the SCS literature, you would integrate via async mechanisms across SCSes, provide versioned interfaces, and enforce via consumer-driven contract testing like Pact: https://pact.io