Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Difference Between CI and CD

fire.ci

Post reply on HN
Print view View original

41–50 of 60 posts

Re: The Difference Between CI and CD

#41

Post by SideburnsOfDoom »

cottonseed wrote: post #17
smcleod wrote: post #16

Earlier quoted context omitted.

I would say for me that's a pretty reasonable estimate for microservice architecture applications / services, of course large legacy monoliths take longer but not more than say 15-20 minutes at most.

3m seems aggressive to do builds and spin up infrastructure for anything non-trivial. Reading a bit closer, I see the author describes CI as a sanity check, "ensur[ing] the bare minimum" and doesn't consider deploying on every commit. Maybe 3-7m is more realistic then. However, I'm slightly surprised by this definition of CI. According to Fowler [0], "Continuous Delivery is a software development discipline where you…

> the author describes CI as a sanity check

Which is a nonsense, since CI is the practice of merging to master frequently, in a state that can be released if need be.

We won't understand it unless we distinguish the practice from the supporting tools that help us do it safely:

in this case, the practice is frequent merge to shared trunk, and the supporting tools are as many automated checks before and after that merge as can be done quickly.

A "CI build" of a branch is a tool to help you do CI, but unless you merge that branch when it's green, you're not _doing_ CI.

Misunderstanding this and doing "CI on a branch" means that you are mistaking the tool for the practice, and not doing the practice: by delaying integration, you will be accomplishing the opposite of CI.

Re: The Difference Between CI and CD

#42

Post by peterwwillis »

gregdoesit wrote: post #23
hinkley wrote: post #4

> The scenario we want to avoid is that a faulty commit makes it to the main branch. Close. The scenario we want to minimize is faulty code on the main branch. As your team grows, as the number of commits go up, it becomes a game of chance. Sooner or later something will get through. The more new teammates you have, the more often that will happens. This is an inescapable cost of growth. The cost of promoting people…

> If you insist on no errors on master ever you will kill throughput. Unless you solve this engineering problem with tooling. At Uber, the full-blown CI mobile test suite takes over 30 minutes to run on a development machine (linting, unit test, UI tests - most of this time being the long-running UI tests, specific to native mobile). So we only do incremental runs locally, and have a submit queue, which parallelises…

An alternative system includes optimizations: 1) don't use a monorepo, 2) don't run tests that have nothing to do with the code changed. Both require redesign of code structure, testing, execution, but both remove the inherent limits of integration.

Nobody seems to talk about this and I don't know why. It would remove integration complexity and speed up testing. We do the same thing for CD and nobody seems to have a problem with it...

Re: The Difference Between CI and CD

#43

Post by jpdel »

s_Hogg wrote: post #39

The title reminds me a lot of all those fatuous articles about the difference between statistics and machine learning. This one is alright though - I wonder how we got to lumping CI and CD together as is commonplace now.

I think this is a business trend unfortunately. Tools tackling the CI space wanted a piece of CD and then boom. Things became the same :)

Re: The Difference Between CI and CD

#44

Post by Game_Ender »

peterwwillis wrote: post #42
gregdoesit wrote: post #23

Earlier quoted context omitted.

> If you insist on no errors on master ever you will kill throughput. Unless you solve this engineering problem with tooling. At Uber, the full-blown CI mobile test suite takes over 30 minutes to run on a development machine (linting, unit test, UI tests - most of this time being the long-running UI tests, specific to native mobile). So we only do incremental runs locally, and have a submit queue, which parallelises…

An alternative system includes optimizations: 1) don't use a monorepo, 2) don't run tests that have nothing to do with the code changed. Both require redesign of code structure, testing, execution, but both remove the inherent limits of integration. Nobody seems to talk about this and I don't know why. It would remove integration complexity and speed up testing. We do the same thing for CD and nobody seems to have a…

The queue and test system Uber and Google use for their monorepos essentially do both of those. The restructuring you mention was to use a build system such as Bazel or Buck universally.

1) Two changes which don’t effect intersecting parts of the repo are landed separately. Similar to having infinite separate repos.

2) Only the tests that your code effects are run.

This is all possible because Bazel let’s you look at a commit and determine with certainty which test needs to run a and which targets are effected.

Re: The Difference Between CI and CD

#45

Post by brown9-2 »

pojzon wrote: post #31
gregdoesit wrote: post #23

Earlier quoted context omitted.

> If you insist on no errors on master ever you will kill throughput. Unless you solve this engineering problem with tooling. At Uber, the full-blown CI mobile test suite takes over 30 minutes to run on a development machine (linting, unit test, UI tests - most of this time being the long-running UI tests, specific to native mobile). So we only do incremental runs locally, and have a submit queue, which parallelises…

How do you handle situations like that: multiple dvelopers added merge requests to queue, the changes they made are mutually exclusive (automatic rebase wont work). What happens when the first branch gets merged to master and next 10 are still in the queue ? How do you mitigate that to decrease development cycle ? Lets just say in my company it also takes 30m to run tests and 4h to run them on merge pipeline with FAT…

The robot won’t merge a change in the queue if it can’t be merged or tests fail. The changeset would be left open and the developer notified to fix it.

The whole process assumes that multiple changes in the queue don’t depend on each other, if they did, it should all be in the same changeset.

Re: The Difference Between CI and CD

#46

Post by brown9-2 »

jdlshore wrote: post #5
hinkley wrote: post #4

> The scenario we want to avoid is that a faulty commit makes it to the main branch. Close. The scenario we want to minimize is faulty code on the main branch. As your team grows, as the number of commits go up, it becomes a game of chance. Sooner or later something will get through. The more new teammates you have, the more often that will happens. This is an inescapable cost of growth. The cost of promoting people…

It's possible (and not that hard) to define an integration process that prevents faulty commits from being integrated to the main branch. > If you insist on no errors on master ever you will kill throughout. Not sure why you believe this. It hasn't been my experience; just the opposite, in fact. By using CI in conjunction with a process that prevents errors on master, everything goes more smoothly, because people don…

It depends on what you means by “errors on master”. Tests won’t catch all possible bugs in production.

Re: The Difference Between CI and CD

#47

Post by Game_Ender »

pojzon wrote: post #31
gregdoesit wrote: post #23

Earlier quoted context omitted.

> If you insist on no errors on master ever you will kill throughput. Unless you solve this engineering problem with tooling. At Uber, the full-blown CI mobile test suite takes over 30 minutes to run on a development machine (linting, unit test, UI tests - most of this time being the long-running UI tests, specific to native mobile). So we only do incremental runs locally, and have a submit queue, which parallelises…

How do you handle situations like that: multiple dvelopers added merge requests to queue, the changes they made are mutually exclusive (automatic rebase wont work). What happens when the first branch gets merged to master and next 10 are still in the queue ? How do you mitigate that to decrease development cycle ? Lets just say in my company it also takes 30m to run tests and 4h to run them on merge pipeline with FAT…

Well first step is to optimize, parallelize and refactor so you do not have a single process that takes hours, but many separate ones you can run at once in a cluster.

If those get too expensive to run or you cannot speed them up them you have to do what Chromium does: run them post commit then bisect and revert any changes that break the tests. If things are truly broken you close the tree for a bit while you get the break reverted or fixed.

Also the system that is landing changes tests the optimistically in parallel assuming they will all succeed, so it does land a change only 30 minutes for example.

Re: The Difference Between CI and CD

#48

Post by a_imho »

The process of Continuous Integration is independent of any tool.

This is one of my pet peeves, people using the term CI referring to the tooling. For me this alone invalidates anything they have to say about the subject.

Re: The Difference Between CI and CD

#49

Post by bump64 »

I am using Azure Devops for the past couple of years and they have kind of nailed it. You have builds that do most of the CI and Releases which can be fine tuned to do complex deployments and complete the CD story. Then you can set them as a requirement for the pull request approval to the main and branch it helps to guarantee healthy trunk.

I don't agree that CI is a team problem and CD is an engineering problem. If you are following infrastructure as code principles it is everyone's problem because if you don't add how your new feature should be deployed it will break the CI and CD pipelines and you won't be able to merge it.

Re: The Difference Between CI and CD

#50

Post by gregdoesit »

pojzon wrote: post #31
gregdoesit wrote: post #23

Earlier quoted context omitted.

> If you insist on no errors on master ever you will kill throughput. Unless you solve this engineering problem with tooling. At Uber, the full-blown CI mobile test suite takes over 30 minutes to run on a development machine (linting, unit test, UI tests - most of this time being the long-running UI tests, specific to native mobile). So we only do incremental runs locally, and have a submit queue, which parallelises…

How do you handle situations like that: multiple dvelopers added merge requests to queue, the changes they made are mutually exclusive (automatic rebase wont work). What happens when the first branch gets merged to master and next 10 are still in the queue ? How do you mitigate that to decrease development cycle ? Lets just say in my company it also takes 30m to run tests and 4h to run them on merge pipeline with FAT…

A lot of the below comments touched on things we do (verifying that changesets are independent, breaking tests into smaller pieces, prioritising changes that are likely to succeed). They add up and the approach does become more complex. We wrote an ACM white paper with more of the details[1]. It’s the many edge cases and several optimisation problems that turn this into an interesting theoretical and practical problem.

[1] http://delivery.acm.org/10.1145/3310000/3303970/a29-ananthan...

Post reply on HN