Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Difference Between CI and CD

fire.ci

Post reply on HN
Print view View original

51–60 of 60 posts

Re: The Difference Between CI and CD

#51

Post by jpdel »

bump64 wrote: post #49

I am using Azure Devops for the past couple of years and they have kind of nailed it. You have builds that do most of the CI and Releases which can be fine tuned to do complex deployments and complete the CD story. Then you can set them as a requirement for the pull request approval to the main and branch it helps to guarantee healthy trunk. I don't agree that CI is a team problem and CD is an engineering problem. If…

Also using Azure DevOps and it is indeed very well structured.

As for CI/CD differences: how many commits can actually affect code and infrastructure? I think this is part of the engineering problem at the end of the day.

Re: The Difference Between CI and CD

#52

Post by peterwwillis »

Game_Ender wrote: post #44
peterwwillis wrote: post #42

Earlier quoted context omitted.

An alternative system includes optimizations: 1) don't use a monorepo, 2) don't run tests that have nothing to do with the code changed. Both require redesign of code structure, testing, execution, but both remove the inherent limits of integration. Nobody seems to talk about this and I don't know why. It would remove integration complexity and speed up testing. We do the same thing for CD and nobody seems to have a…

The queue and test system Uber and Google use for their monorepos essentially do both of those. The restructuring you mention was to use a build system such as Bazel or Buck universally. 1) Two changes which don’t effect intersecting parts of the repo are landed separately. Similar to having infinite separate repos. 2) Only the tests that your code effects are run. This is all possible because Bazel let’s you look at…

That is good to hear, but I'm interested in finding the patterns that makes this feasible without a build tool designed for massively parallel building, testing, and integration of a single codebase. A lot of the historic reasons for Google's build system come down to "we just like the monorepo but it needs complex tooling to work".

Re: The Difference Between CI and CD

#53

Post by bluGill »

pojzon wrote: post #31
gregdoesit wrote: post #23

Earlier quoted context omitted.

> If you insist on no errors on master ever you will kill throughput. Unless you solve this engineering problem with tooling. At Uber, the full-blown CI mobile test suite takes over 30 minutes to run on a development machine (linting, unit test, UI tests - most of this time being the long-running UI tests, specific to native mobile). So we only do incremental runs locally, and have a submit queue, which parallelises…

How do you handle situations like that: multiple dvelopers added merge requests to queue, the changes they made are mutually exclusive (automatic rebase wont work). What happens when the first branch gets merged to master and next 10 are still in the queue ? How do you mitigate that to decrease development cycle ? Lets just say in my company it also takes 30m to run tests and 4h to run them on merge pipeline with FAT…

What you describe is typically an architecture problem: if you have a good architecture in place the problem won't happen because you have already broken your system up so that those places that 10 completely different developers need to touch do not exist in the first place. You need to hire more senior developers to think about this problem and fix it. You should be able to assign every area of code to a small team of developers who work together and coordinate their changes to that area. (even with common code ownership you quickly specialize just because on a large project you cannot understand everything)

There are exceptions. Sometimes there is a management problem: management has been told some things cannot be done in parallel because you couldn't mitigate the problem in architecture and they failed to apply project management practices to ensure the developers worked serially.

Sometimes there is a team problem: the 10 developers have been placed on the same team to work on the same thing, and despite all that they still failed to coordinate among themselves to ensure that the changes happened in order.

Re: The Difference Between CI and CD

#54

Post by bluGill »

UK-Al05 wrote: post #32
gregdoesit wrote: post #23

Earlier quoted context omitted.

> If you insist on no errors on master ever you will kill throughput. Unless you solve this engineering problem with tooling. At Uber, the full-blown CI mobile test suite takes over 30 minutes to run on a development machine (linting, unit test, UI tests - most of this time being the long-running UI tests, specific to native mobile). So we only do incremental runs locally, and have a submit queue, which parallelises…

This may surprise you, but tests only help you find breaking changes. They don't guarantee it. To guarantee anything costs a lot.

Tests are there to prove everything you thought of works correctly. They do nothing to find things you didn't think of. However with effort ($$$) you can become very creative in thinking about failure cases that you then test.

If you want to find problems you didn't think of formal proofs are the only think I have heard of. However formal proofs only work if you can think of the right constraints (I forget the correct term) which isn't easy.

Note that the two are not substitutes for each other. While there is overlap there are classes of errors that one along will not catch. For most projects though it is more cost effective to live with some bugs for a long time than to spend enough money on either of the above to find it ahead of time. Different projects have different needs (games vs medical devices...)

Re: The Difference Between CI and CD

#55

Post by bluGill »

peterwwillis wrote: post #52
Game_Ender wrote: post #44

Earlier quoted context omitted.

The queue and test system Uber and Google use for their monorepos essentially do both of those. The restructuring you mention was to use a build system such as Bazel or Buck universally. 1) Two changes which don’t effect intersecting parts of the repo are landed separately. Similar to having infinite separate repos. 2) Only the tests that your code effects are run. This is all possible because Bazel let’s you look at…

That is good to hear, but I'm interested in finding the patterns that makes this feasible without a build tool designed for massively parallel building, testing, and integration of a single codebase. A lot of the historic reasons for Google's build system come down to "we just like the monorepo but it needs complex tooling to work".

For a complex project you need complex tooling. A mono-repo and a multi-repo system have different needs, but both need complex tooling to work. Neither is inherently better than the other, there are pros and cons. Sometimes those are compelling (which is why a few projects at google are not part of the mono-repo)

For me I prefer the pros and cons of multi-repo. However sometimes I wish I could do the large cross project refactoring that a mono-repo would make easy.

Re: The Difference Between CI and CD

#56

Post by Game_Ender »

brown9-2 wrote: post #45
pojzon wrote: post #31

Earlier quoted context omitted.

How do you handle situations like that: multiple dvelopers added merge requests to queue, the changes they made are mutually exclusive (automatic rebase wont work). What happens when the first branch gets merged to master and next 10 are still in the queue ? How do you mitigate that to decrease development cycle ? Lets just say in my company it also takes 30m to run tests and 4h to run them on merge pipeline with FAT…

The robot won’t merge a change in the queue if it can’t be merged or tests fail. The changeset would be left open and the developer notified to fix it. The whole process assumes that multiple changes in the queue don’t depend on each other, if they did, it should all be in the same changeset.

It assumes most do not, but it’s entirely possible for someone to change a common library which makes several down stream changes wait. Even if there are no merge conflicts, if they effect the same tests, changes will have to wait.

Re: The Difference Between CI and CD

#57

Post by sytse »

pojzon wrote: post #31
gregdoesit wrote: post #23

Earlier quoted context omitted.

> If you insist on no errors on master ever you will kill throughput. Unless you solve this engineering problem with tooling. At Uber, the full-blown CI mobile test suite takes over 30 minutes to run on a development machine (linting, unit test, UI tests - most of this time being the long-running UI tests, specific to native mobile). So we only do incremental runs locally, and have a submit queue, which parallelises…

How do you handle situations like that: multiple dvelopers added merge requests to queue, the changes they made are mutually exclusive (automatic rebase wont work). What happens when the first branch gets merged to master and next 10 are still in the queue ? How do you mitigate that to decrease development cycle ? Lets just say in my company it also takes 30m to run tests and 4h to run them on merge pipeline with FAT…

I hope it is possible to decompose this in two problems:

1. Dependencies in incompatible Merge Requests that need to be accounted for, see https://docs.gitlab.com/ee/user/project/merge_requests/merge... on how to do that.

2. Most merge requests can merge in previous changes changes, for that you can use merge trains as detailed in my other comment https://news.ycombinator.com/item?id=21679515

Re: The Difference Between CI and CD

#59

Post by pojzon »

gregdoesit wrote: post #50
pojzon wrote: post #31

Earlier quoted context omitted.

How do you handle situations like that: multiple dvelopers added merge requests to queue, the changes they made are mutually exclusive (automatic rebase wont work). What happens when the first branch gets merged to master and next 10 are still in the queue ? How do you mitigate that to decrease development cycle ? Lets just say in my company it also takes 30m to run tests and 4h to run them on merge pipeline with FAT…

A lot of the below comments touched on things we do (verifying that changesets are independent, breaking tests into smaller pieces, prioritising changes that are likely to succeed). They add up and the approach does become more complex. We wrote an ACM white paper with more of the details[1]. It’s the many edge cases and several optimisation problems that turn this into an interesting theoretical and practical proble…

Sorry, but that link points to "not found" page.

Re: The Difference Between CI and CD

#60

Post by dionian »

dantiberian wrote: post #9
hinkley wrote: post #4

> The scenario we want to avoid is that a faulty commit makes it to the main branch. Close. The scenario we want to minimize is faulty code on the main branch. As your team grows, as the number of commits go up, it becomes a game of chance. Sooner or later something will get through. The more new teammates you have, the more often that will happens. This is an inescapable cost of growth. The cost of promoting people…

> If you insist on no errors on master ever you will kill throughput. There are a large number of automated tools which will help you prevent merging code that could break master: https://github.com/chdsbd/kodiak#prior-art--alternatives . The basic approach is to make a new branch from master, apply one or more commits on top of that branch, run the tests, and if tests pass, merge those commits (with fast-forward) ba…

automated tests don't necessarily catch all errors
Post reply on HN