Live data from Hacker News

Successfully Merging the Work of 1000 Developers

engineering.shopify.com

31–40 of 108 posts

Re: Successfully Merging the Work of 1000 Developers

#31
post #24

Did I read it correctly that in first iteration they merged "NOT TESTED" code into master ? Who would ever think it's a good idea ?

That’s what trying to get at in a “more polite” way. Many people use a “develop” branch to stage changes before master so they can run through CI. I’m curious as to why they used an untested queue instead. It seems like they wanted to cherry pick what commits made it to master rather than going in chronological order.

What’s odd is that they seem to characterize this as a tools issue rather than a process issue. There are plenty of CI/CD tools that allow for a similar or the same workflow as what they created. It’s also kinda scary that there’s not an emphasis on the overall SDLC and how the specific attributes of a branch or commit should/shouldn’t affect the process. You’d think at 1000 developers it’d be very important to define what is “launchable” as well. Haven’t worked with 1,000 on the same project but even with 20+, the standards and practices around development were always more important than the tooling. Tooling was just meant to represent workflows that were already defined.

Re: Successfully Merging the Work of 1000 Developers

#33

If I read this article right, if anyone breaks any part of the build it breaks for everyone? Doesn't sound very scalable. Shouldn't the main goal be to break up your continuous integration steps so that a person in one end of the company can continue working even if a person in the other end broke his build? That way you can also add tags for flaky tests etc, to make your builds more reliable. Edit: I didn't understa…

There is an explanation about how we handle this case when I talk about the failure-tolerance threshold. I go deeper into this in my GitHub Universe talk where I also talk about an alternative (but costlier) solution through running parallel branches, but unfortunately that talk is not posted up yet.

Re: Successfully Merging the Work of 1000 Developers

#34
post #27
post #7

Earlier quoted context omitted.

Hi, Author here! Pull requests are our unit of work, and the queue was created to support all pull requests. We do have feature flags as a tool, but we let our developers make the judgment call on how their changes should be rolled out.

Interesting. It seems like you have a very flexible process of how to launch code which could contribute to issues with visibility and rollbacks. I’m curious as to why you had a queue instead of a develop branch before moving to CD? Was this to allow arbitrary commits to be launched to production rather than getting them batched by time?

The queue is simply an automated "develop" branch.

Re: Successfully Merging the Work of 1000 Developers

#35

This sounds quite similar to https://bors.tech . If the authors are here, did you see this, and can you compare and contrast it with what you built? https://graydon2.dreamwidth.org/1597.html also has a good overview of the problem and the original bors.

Yes we have seen this before! The main difference is that throughput is extremely important for us, which we would not get worth Bors. Also, compatibility of multiple simultaneously merging PRs is the case that we are optimizing for, vs. compatibility with current master.

Re: Successfully Merging the Work of 1000 Developers

#36
post #24

Did I read it correctly that in first iteration they merged "NOT TESTED" code into master ? Who would ever think it's a good idea ?

The branches that are being merged are tested, also in the first version. However, different branches can conflict with each other, and break the master builds. This was happening often enough for us to want to prevent this.

The simple way would be to rebase your branch (or merge in master). However, with the amount of changes that are being merged, by the time the CI result comes in for your rebased branch, the tip of master is already changed making the CI result obsolete. So we went for the queue approach instead.

Re: Successfully Merging the Work of 1000 Developers

#37
post #24

Did I read it correctly that in first iteration they merged "NOT TESTED" code into master ? Who would ever think it's a good idea ?

It was tested. It's simply that between the time you pushed on your branch, and the time you merged, many other commits made it to the master branch, potentially breaking your branch.

Re: Successfully Merging the Work of 1000 Developers

#38
post #21

Earlier quoted context omitted.

We didn't have a merge queue at Google. You rebased if there was a merge conflict, ran through CI again, and hoped there wasn't another merge conflict. I think I ran into merge conflicts maybe once a year, if that. I think the success of this system breaks down into several parts: 1) Yup, microservices. You could submit your proto change, which would affect all clients, before actually implementing the code that used…

Point 2 is very important and very hard to get right. For unit tests, there is a clear dependency on the code and you can easily just run a subset of the tests. But wouldn't you have to run any system and integration tests of the affected module, as it's not clear what effects the code change can have? This will blow up CI times again. How did Google deal with this?

Your integration test needed the system that you were integrating with, so you'd have to declare that as a dependency.

My philosophy was to always have integration tests run in the normal CI system. This basically meant creating a test binary that happened to link in the systems you were integrating with, and run tests against that. This is easier when everything is written in the same programming language, and for the cases where it wasn't, I was usually happy with "fakes". (https://testing.googleblog.com/2013/06/testing-on-toilet-fak...)

Other teams really loved the sandbox environment with live instances of everything. They would have some machinery outside the standard CI system to inject their code into this sandbox and run some tests, as well as machinery for keeping their sandbox up to date with production. (And adding test data, etc., etc., which all becomes very complex very quickly.)

Both methodologies have their downsides and upsides.

I generally prefer simplicity and speed; people should be able to run the tests on their workstation 100% of the time without having to set up any external resources. If you have an integration test binary that is built from the build system, this is possible. The downside is that config changes in production can break your system; since you are starting up your own instance of some other team's server, they could theoretically make some config change that breaks your integration. Even if you include their configuration in your in-memory version of their service, there was no guarantee that what is running in production is actually checked in yet. (Debugging in production, emergency rollback to an older prebuilt binary, etc.) These were rare and never caused me problems, however, and not having machinery to maintain a shadow environment meant it was easier to work on the code.

Having a sandbox environment was good because you could "check" (not test) big changes before putting them into production. You could try out your flag flip, database migration, mapreduce, or just load up the website in your browser and send your coworkers a link without affecting production data. And you could test your actual production binary in production-like conditions; as long as you sync'd production changes to your sandbox, your automated test probably ran against something that was very much like production. This let you check for more subtle things like performance regressions before deploying. (I worked on a system to do just that.)

The main problem I had with this method was that it was maintenance-intensive (big teams that used this had entire teams just to maintain the sandbox, and that begat sub teams that maintained the sandbox maintenance) and slow. Building and running another test during CI was relatively fast, but starting up a job in production and scaling it up was significantly slower. This meant that you needed a parallel set of tools to run some subset of this environment locally, and it was always painful. Not having your tests in the standard system meant that downstream dependencies wouldn't see test failures in your system when you made a change, so the "buildcop" would have to detect and fix that.

I found this to be too much overhead, but it is probably necessary when you are developing, say, a mobile application. You will have to write some sort of software to make it possible to try your in-progress code on your personal phone. You will probably want to be able to share links with coworkers. I generally like to push changes to production multiple times a day, and make sure that clients can handle a newer server and still work correctly. This way, as soon as a build passes tests, you can start giving it, say 0.1% of production traffic and keep an eye on the error rates, and promote that to production as quickly as possible. The biggest problem I've run into with this strategy is that 0.1% of Google's traffic is way more than enough for a good canary, but at other places I've worked... 0.1% of traffic might be one request over several days. In that case, you have to have staging and manually bug people to try it out. Sometimes I wonder if that kind of software is worth writing at all, to be perfectly honest. If you get one request a day, maybe just make it open a support ticket, and hire 2 support engineers instead of one software engineer. But I digress ;)

Re: Successfully Merging the Work of 1000 Developers

#39
post #21

Earlier quoted context omitted.

We didn't have a merge queue at Google. You rebased if there was a merge conflict, ran through CI again, and hoped there wasn't another merge conflict. I think I ran into merge conflicts maybe once a year, if that. I think the success of this system breaks down into several parts: 1) Yup, microservices. You could submit your proto change, which would affect all clients, before actually implementing the code that used…

Point 2 is very important and very hard to get right. For unit tests, there is a clear dependency on the code and you can easily just run a subset of the tests. But wouldn't you have to run any system and integration tests of the affected module, as it's not clear what effects the code change can have? This will blow up CI times again. How did Google deal with this?

Not sure if this actually answers the question, but - Bazel, the build system used at Google, creates dependency graphs (example: https://blog.bazel.build/2015/06/17/visualize-your-build.htm...), which I believe can be used to run tests on any code affected by a change.

Re: Successfully Merging the Work of 1000 Developers

#40

This is exactly the kind of workflow that Bors ( https://github.com/bors-ng/bors-ng ) automates. In addition to Bors, there are a number of apps and services that automate this kind of workflow. Here is an incomplete list: https://forum.bors.tech/t/other-apps-that-implement-the-nrsr... Edited to add: Graydon Hoare (creator of Rust) called this the Not Rocket Science Rule Of Software Engineering (NRSROSE): "automatica…

The limitation of Bors for us was throughput, we were more interested in testing multiple simultaneous pull requests merging at the same time, rather than testing against the latest master.

With our CI times and PR volume, by the time any CI run completes, master would have drastically changed.

Post reply on HN