111–120 of 120 posts
There's a nice middle ground between this and a one-at-a-time submit queue: have a speculative batch running on the side. This gives nice speedups (approaching N times more commits, where N is the batch size) with minimal complexity. One useful metric is the ratio between test time and the number of commits per day. If your tests run in a minute, you can test submissions one at a time and still have a thousand succes…
Earlier quoted context omitted.
> For multirepo users this is explicit in that this comes for free :-) Only if you spend the time to build tools to detect commits in your dependencies, as well as your dependent repositories, and figure out how to update and check them out on the appropriate builds. So, no, it doesn't come for free.
Package managers solve it quite well. Just depend on the latest version of your dependencies and tag a new version whenever they change.
There's a nice middle ground between this and a one-at-a-time submit queue: have a speculative batch running on the side. This gives nice speedups (approaching N times more commits, where N is the batch size) with minimal complexity. One useful metric is the ratio between test time and the number of commits per day. If your tests run in a minute, you can test submissions one at a time and still have a thousand succes…
> Optimistic execution of changes is another technique being used by production systems (e.g., Zuul [12]). Similar to optimistic concurrency control mechanisms in transactional systems, this approach assumes that every pending change in the system can succeed. Therefore, a pending change starts performing its build steps assuming that all the pending changes that were submitted before it will succeed. If a change fails, then the builds that speculated on the success of the failed change needs to be aborted, and start again with new optimistic speculation. Similar to the previous solutions, this approach does not scale and results in high turnaround time since failure of a change can abort many optimistically executing builds. Moreover, abort rate increases as the probability of conflicting changes increase (Figure 1).
Earlier quoted context omitted.
What you really want to do is first disable a test you know is unhealthy to unblock everybody. Then, you fix it. After you've reintroduced it healthy, you can turn it back on.
I was talking to someone from Google who works on Bazel things, and he brought an interesting point: flaky tests are asymmetric in that they don't provide much value when they fail (since you don't know if the failure was due to flakiness), but they do provide a lot of value when they pass (because they presumable test something non-trivial.) With this in mind, what Bazel does when a test is marked flaky is run it se…
Earlier quoted context omitted.
If there are 1000 commits per day (which wouldn't be that many), that's 10 master breaks per day.
And at least in my limited experience, the impact of master being broken is pretty big, and even bigger when you have multiple teams. Either you block master - leading to a lot less than those 1000 commits making it to master on that day - or continue merging in stuff, which causes the root cause of the master branch to become fuzzy - and if your reporting is not in order, that is, if the person who broke the build i…
Earlier quoted context omitted.
That's not good stewardship. You have a better API? Great, convince us it's worth investing in soon, you can even deprecate the known-good version. There's always a window where both will be in use, because we can't synchronously replace every running process everywhere (not that it's even a good idea without a canary). The shorter you try to make that window, the more needless pain is created and plans disrupted. Wh…
That's not reality for most large companies, even though it's the right mindset for most software libraries. Ex: A new legal requirement comes in, resulting in a mandate that fields X and Y for certain queries, that are being done all over the codebase, now have to be tokenized. This is a breaking and mandatory change, with no room to allow systems to stay behind, and expensive consequences. In this case you'll have…
Earlier quoted context omitted.
You can get most of the benefit on smaller scales by building feature branches and ensuring they pass unit tests, deployment and integration testing before they're allowed to be merged to mainline. It still depends on well written tests, lest your confidence be dashed when a human starts pushing buttons and pulling levers. Also, don't break up tightly coupled code/modules into separate repos for the sake of microserv…
It isn't just that the tests need to be good and humans break things sometimes. At a certain scale, the following happens enough to be a problem: - changeset A is submitted, an integration branch is cut from latest master, and CI begins - changeset B is submitted, an integration branch is cut from latest master, and CI begins - changeset A's integration branch passes CI build/test, so A is merged into master - change…
disclaimer: I am one of Datree.io founders. We provide a visibility and governance solution to R&D organizations on top of GitHub.
Here are some rules and enforcement around Security and Compliance which most of our companies use for multi-repo GitHub orgs. 1. Prevent users from adding outside collaborators to GitHub repos. 2. Enforce branch protection on all current repos and future created ones - prevent master branch deletion and force push. 3. Enforce pull request flow on default branch for all repos (including future created) - prevent direct commits to master without pull-request and checks. 4. Enforce Jira ticket integration - mention ticket number in pull request name / commit message. 5. Enforce proper Git user configuration. 6. Detect and prevent merging of secrets.
Anyone fancy comparing this to bors?
Actually we have compared it in our paper. Bors builds one change at a time. On the other hand, Submit Queue speculatively builds several changes at a time based on the outcomes of other pending changes in the system. Apart from that, Submit Queue uses a conflict analyzer to find independent changes in order to commit changes in parallel as well as trim the speculation graph. We have also evaluated the performance of…
Bors builds multiple changes at once (it creates a merge commit of all available changes and then runs the tests on all of them), and merges if all of them are good.
Possibly you are thinking of the older bors, as opposed to modern bors-ng?
Anyone fancy comparing this to bors?
The main difference is in the conflict-detection system. Whereas bors only has a single queue, this new system can have one queue for each set of changes which doesn't interact with any other set. Eg. if you've got an ios app, a webapp, and a bunch of documentation all in the same repo, then this system will automatically work out that changes to each of those independent projects can be tested and merged in parallel…