Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The purpose of continuous integration is to fail

blog.nix-ci.com

Post reply on HN
Print view View original

41–50 of 52 posts

Re: The purpose of continuous integration is to fail

#41

Post by bilkow »

1970-01-01 wrote: post #9

Oversimplified click bait. The purpose never changed from catching bad bugs before it was sent to prod. The goal of CI is to prevent the resulting problems from doing damage and requiring emergency repairs.

I don't really understand the point you're trying to make, I don't see anywhere in the post nor the title claiming the purpose changed and the title is directly related to the content. In fact, it seems like you are just agreeing with the post.

I think people can get frustrated at CI when it fails, so they're explaining that that's the whole purpose of it and why it's is a actually good thing.

I would personally actually frame it slightly different than the author. Non-flaky CI errors: your code failed CI. Flaky CI errors: CI failed. Just to be clear, that's more precise but would never catch on because people would simplify "your code failed CI" to "CI failed" over time, but I don't thing that changes it from being an interesting way to frame.

Re: The purpose of continuous integration is to fail

#43

Post by lgunsch »

Some of the other practices of CI are also important. Not explicitly mentioned by the article, but perhaps implied. CI is a lot more than just running tests on pull request. It's a whole suite of practices enabling teams to perform and ship better. Some of which include keeping branches short lived by merging back to main early and often. Keeping code ready for deployment at any time by using strategies like feature switches. This keeps the cost of shipping a feature as low as possible, avoiding issues like spending lots of time rebasing and merging long lived feature branches.

Re: The purpose of continuous integration is to fail

#44

Post by pm215 »

deathanatos wrote: post #35

> One dreaded and very common situation is when a failing CI run can be made to pass by simply re-running it. We call this flaky CI. > Flaky CI is nasty because it means that a CI failure no longer reliably indicates that a mistake was caught. And it is doubly nasty because it is unfixable (in theory); sometimes machines just explode. > Luckily flakiness can be detected: Whenever a CI run fails, we can re-run it. If…

A fairly large category of the flaky CI jobs I see is "dodgy infrastructure". For instance one recurring type for our project is one I just saw fail this afternoon, where a gitlab CI runner tries to clone the git repo from gitlab itself and gets an HTTP 502 error. We've also had issues with "the s390 VM that does CI job running is on an overloaded host, so mostly it's fine but occasionally the VM gets starved of CPU and some of the tests time out".

We do also have some genuinely flaky tests, but it's pretty tempting to hit the big "just retry" button when there's all this flakiness we can't control mixed in there.

Re: The purpose of continuous integration is to fail

#46

Post by stego-tech »

I’m one of today’s lucky 10k, because this judo-threw me with how I (didn’t) understand CI/CD. My experience with it has largely been a cumbersome add-on to existing processes that are often incredibly fragile and impossible to amend; turns out, that’s kind of the point. Understanding that it’s the equivalent of doing rocket tests on kit you expect to fail and using that to build better rockets suddenly makes its value far more recognizable, at least to my eyes.

Solid writeup. Definitely keeping in my personal notes.

Re: The purpose of continuous integration is to fail

#47

Post by Norfair »

jquaint wrote: post #45

https://github.com/srid/nixci Is this the project or is this a completely different Nix based CI/CD tool? I can't find a Github or anything on the website.

Author here: NixCI (https://nix-ci.com) is not open-source. https://github.com/srid/nixci has been replaced by om ci: https://omnix.page/om/ci.html

Re: The purpose of continuous integration is to fail

#48

Post by fcarraldo »

rkangel wrote: post #40
9rx wrote: post #33

Earlier quoted context omitted.

The parent is talking about when the implementation is flaky, not the test. When you go to fix the problem under that scenario there is no reason for you to modify the test. The test is fine.

What you're describing is the every day reality but what you WANT is that if your implementation has a race condition, then you want a test that 100% of the time detects that there is a race condition (rather than 1% of the time).

If your test can deterministically result in a race condition 100% of the time, is that a race condition? Assuming that we're talking about a unit test here, and not a race condition detector (which are not foolproof).

Re: The purpose of continuous integration is to fail

#49

Post by 9rx »

fcarraldo wrote: post #48
rkangel wrote: post #40

Earlier quoted context omitted.

What you're describing is the every day reality but what you WANT is that if your implementation has a race condition, then you want a test that 100% of the time detects that there is a race condition (rather than 1% of the time).

If your test can deterministically result in a race condition 100% of the time, is that a race condition? Assuming that we're talking about a unit test here, and not a race condition detector (which are not foolproof).

You can use deterministic simulation testing to reproduce a real-world race condition 100% of the time while under test.

But that's not the kind of test that will expose a race condition 1% of the time. The kinds of tests that are inadvertently finding race conditions 1% of the time are focused on other concerns.

So it is still not a case of a flaky test, but maybe a case of a missing test.

Re: The purpose of continuous integration is to fail

#50

Post by rkangel »

fcarraldo wrote: post #48
rkangel wrote: post #40

Earlier quoted context omitted.

What you're describing is the every day reality but what you WANT is that if your implementation has a race condition, then you want a test that 100% of the time detects that there is a race condition (rather than 1% of the time).

If your test can deterministically result in a race condition 100% of the time, is that a race condition? Assuming that we're talking about a unit test here, and not a race condition detector (which are not foolproof).

> Assuming that we're talking about a unit test here

I think the categorisation of tests is sometimes counterproductive and moves the discussion away from what's important: What groups of tests do I need in order to be confident that my code works in the real world?

I want to be confident that my code doesn't have race conditions in it. This isn't easy to do, but it's something I want. If that's the case then your unit test might pass sometimes and fail sometimes, but your CI run should always be red because the race test (however it works) is failing.

This is also hints at a limitation of unit tests, and why we shouldn't be over-reliant on them - often unit tests won't show a race. In my experience, it's two independent modules interacting that causes the race. The same can be true with a memory bug caused by a mismatch in passing of ownership and who should be freeing, or any of the other issues caused by interactions between modules.

Post reply on HN