The purpose of continuous integration is to fail • Hacker News Board

Re: The purpose of continuous integration is to fail

#11

Post by vova_hn2 » Thu, Mar 12, 2026, 2:48 PM UTC

cogman10 wrote: post #6

chriswarbo wrote: post #3

I agree. The same can be said for testing too: their main purpose is to find mistakes (with secondary benefits of documenting, etc.). Whenever I see my tests fail, I'm happy that they caught a problem in my understanding (manifested either as a bug in my implementation, or a bug in my test statement).

This ultimately is what shapes my view of what a good test is vs a bad test. An issue I have with a lot of unit tests is they are too strongly coupled to the implementation. What that means is any change to the implementation ultimately means you have to change tests. IMO, good tests are relatively immutable. You should be able to have multiple valid implementations. You should add new tests to describe the new funct…

> IMO, good tests are relatively immutable. You should be able to have multiple valid implementations. You should add new tests to describe the new functionality of that implementation, however, the old tests should remain relatively untouched.

Taken to extreme this would mean getting rid of unit tests altogether in favor of functional and/or end-to-end testing. Which is... a strategy. I don't know if it is a good or bad strategy, but I can see it being viable for some projects.

Re: The purpose of continuous integration is to fail

#12

Post by n2d4 » Thu, Mar 12, 2026, 2:48 PM UTC

chriswarbo wrote: post #4

jgbuddy wrote: post #2

This is of course true as a blanket "gotcha" headline- although I wouldn't call a failed test the CI itself failing. A real failure would be a false positive, a pass where there wasn't coverage, or a failure when there was no breaking change. Covering all of these edge cases can become as tiresome as maintaining the application in the first place (of course this is a generalization)

> a pass where there wasn't coverage I always feel obliged to point out that we can have 100% coverage without making a single assertion (beware Goodhart's law)

True, but you can't have complete tests without 100% coverage. It's a necessary, but not a sufficient condition; as long as it doesn't become the sole goal, it's still a useful metric.

Re: The purpose of continuous integration is to fail

#13

Post by skydhash » Thu, Mar 12, 2026, 2:50 PM UTC

cogman10 wrote: post #6

chriswarbo wrote: post #3

I agree. The same can be said for testing too: their main purpose is to find mistakes (with secondary benefits of documenting, etc.). Whenever I see my tests fail, I'm happy that they caught a problem in my understanding (manifested either as a bug in my implementation, or a bug in my test statement).

This ultimately is what shapes my view of what a good test is vs a bad test. An issue I have with a lot of unit tests is they are too strongly coupled to the implementation. What that means is any change to the implementation ultimately means you have to change tests. IMO, good tests are relatively immutable. You should be able to have multiple valid implementations. You should add new tests to describe the new funct…

It took me a bit of time (and two or three different view) to finally get this. That is mostly why I hardcode my values in the tests. Make them simpler. If something fails, either the values are wrong or the algorithm of the implementation is wrong.

Re: The purpose of continuous integration is to fail

#14

Post by yrjrjjrjjtjjr » Thu, Mar 12, 2026, 2:51 PM UTC

chriswarbo wrote: post #3

I agree. The same can be said for testing too: their main purpose is to find mistakes (with secondary benefits of documenting, etc.). Whenever I see my tests fail, I'm happy that they caught a problem in my understanding (manifested either as a bug in my implementation, or a bug in my test statement).

The purpose of a car's crumple zone is to crumple.

Re: The purpose of continuous integration is to fail

#15

Post by chrisweekly » Thu, Mar 12, 2026, 2:51 PM UTC

ralferoo wrote: post #5

The premise of the article has some weight, but the final conclusion with the suggestion to change the icons seems completely crazy. Green meaning "to the best of our knowledge, everything is good with the software" is well understood. Using green to mean "we know that this doesn't work at all" is incredibly poor UI (EDITED from "beyond idiotic" due to feedback, my bad). And whilst flaky tests are the most problemati…

Good insights but I'd suggest

"beyond idiotic" -> "misleading | poor UX"

(I agree it's a terrible choice, but civility matters, and strengthens your case.)

Re: The purpose of continuous integration is to fail

#16

Post by cogman10 » Thu, Mar 12, 2026, 2:51 PM UTC

vova_hn2 wrote: post #11

cogman10 wrote: post #6

Earlier quoted context omitted.

This ultimately is what shapes my view of what a good test is vs a bad test. An issue I have with a lot of unit tests is they are too strongly coupled to the implementation. What that means is any change to the implementation ultimately means you have to change tests. IMO, good tests are relatively immutable. You should be able to have multiple valid implementations. You should add new tests to describe the new funct…

> IMO, good tests are relatively immutable. You should be able to have multiple valid implementations. You should add new tests to describe the new functionality of that implementation, however, the old tests should remain relatively untouched. Taken to extreme this would mean getting rid of unit tests altogether in favor of functional and/or end-to-end testing. Which is... a strategy. I don't know if it is a good or…

If you can't tell, I actually think functional tests have a lot more value than most unit tests :)

Kent Dodd agrees with me. [1]

This isn't to say I see no value in unit tests, just that they should tend towards describing the function of the code under test, not the implementation.

[1] https://kentcdodds.com/blog/the-testing-trophy-and-testing-c...

Re: The purpose of continuous integration is to fail

#17

Post by 9rx » Thu, Mar 12, 2026, 2:52 PM UTC

vova_hn2 wrote: post #11

cogman10 wrote: post #6

Earlier quoted context omitted.

This ultimately is what shapes my view of what a good test is vs a bad test. An issue I have with a lot of unit tests is they are too strongly coupled to the implementation. What that means is any change to the implementation ultimately means you have to change tests. IMO, good tests are relatively immutable. You should be able to have multiple valid implementations. You should add new tests to describe the new funct…

> IMO, good tests are relatively immutable. You should be able to have multiple valid implementations. You should add new tests to describe the new functionality of that implementation, however, the old tests should remain relatively untouched. Taken to extreme this would mean getting rid of unit tests altogether in favor of functional and/or end-to-end testing. Which is... a strategy. I don't know if it is a good or…

> Taken to extreme this would mean getting rid of unit tests all together in favor of functional and/or end-to-end testing.

The dirty little secret in CS is that unit, functional, and end-to-end tests are all the exact same thing. Watch next time someone tries to come up with definitions to separate them and you'll soon notice that they didn't actually find a difference or they invent some kind of imagined way of testing that serves no purpose and nobody would ever do.

Regardless, even if you want to believe there is a difference, the advice above isn't invalidated by any of them. It is only saying test the visible, public interface. In fact, the good testing frameworks out there even enforce that — producing compiler errors if you try to violate it.

Re: The purpose of continuous integration is to fail

#18

Post by ralferoo » Thu, Mar 12, 2026, 2:53 PM UTC

chrisweekly wrote: post #15

ralferoo wrote: post #5

The premise of the article has some weight, but the final conclusion with the suggestion to change the icons seems completely crazy. Green meaning "to the best of our knowledge, everything is good with the software" is well understood. Using green to mean "we know that this doesn't work at all" is incredibly poor UI (EDITED from "beyond idiotic" due to feedback, my bad). And whilst flaky tests are the most problemati…

Good insights but I'd suggest "beyond idiotic" -> "misleading | poor UX" (I agree it's a terrible choice, but civility matters, and strengthens your case.)

Fair point, updated my wording.

Re: The purpose of continuous integration is to fail

#19

Post by cestith » Thu, Mar 12, 2026, 2:54 PM UTC

zelos wrote: post #10

> Whenever a CI run fails, we can re-run it. If it passes the second time, we are sure it was flaky. Or you have a concurrency issue in your production code?

It's possibly something else nondeterministic, which may be even more subtle from an external look than a race condition. That should be rare, but it’s been known to happen.

Re: The purpose of continuous integration is to fail

#20

Post by cestith » Thu, Mar 12, 2026, 2:56 PM UTC

9rx wrote: post #7

> When it passes, it's just overhead: the same outcome you'd get without CI. The outcome still isn't the same. CI, even when everything passes, enables other developers to build on top of your partially-built work as it becomes available. This is the real purpose of CI. Test automation is necessary, but only to keep things sane amid you continually throwing in fractionally-complete work.

It also allows for much better record keeping than just spinning up new versions in production without the pipeline.