Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using Radicle CI

radicle.xyz

Post reply on HN
Print view View original

11–20 of 49 posts

Re: Using Radicle CI

#11

Post by apwell23 »

NewJazz wrote: post #7
apwell23 wrote: post #6

Earlier quoted context omitted.

> For example, it may run a lot more tests than what's practical in a local system. yes this is what i was taking about. If there are a lots of tests that are not practical to run locally then they are bad tests no matter how useful one might think they are. only good tests are the ones that run fast. It is also a sign that code itself is bad that you are forced to write tests that interact with outside world. For ex…

It is a tradeoff, e.g. running tests with a real database or other supporting service and taking longer vs. mocking things and having a test environment that is less like reality.

do you really need to test postgres api in your own code?

Re: Using Radicle CI

#12

Post by apwell23 »

8n4vidtmkvmk wrote: post #10
apwell23 wrote: post #6

Earlier quoted context omitted.

> For example, it may run a lot more tests than what's practical in a local system. yes this is what i was taking about. If there are a lots of tests that are not practical to run locally then they are bad tests no matter how useful one might think they are. only good tests are the ones that run fast. It is also a sign that code itself is bad that you are forced to write tests that interact with outside world. For ex…

It does suck when those large scale integration tests fail but sometimes that's the only real way to test something. E.g. I have to call a service owned by another team. It has a schema and documentation so I can mock out what I think it will return, but how will I truly know the API is going to do what it says or what I think it says without actually calling the API?

> I truly know the API is going to do what it says or what I think it says without actually calling the API?

what if the API changes all of sudden in production? what about cases where api stays the same but content of response is all wrong? how do tests protect you from that?

edit: they are not hypothetical scenarios. wrong responses are way more common than schema breaking. tooling upsteam is often pretty good at catching schema breakages.

wrong responses often cause way more havoc than schema breakages because you get an alert for schema failures in app anyways.

Re: Using Radicle CI

#13

Post by maratc »

kuehle wrote: post #2

> I find the most frustrating part of using CI to be to wait for a CI run to finish on a server and then try to deduce from the run log what went wrong. I’ve alleviated this by writing an extension to rad to run CI locally: rad-ci. locally running CI should be more common

This can be achieved by running in CI what commonly runs on local.

E.g. if your build process is simply invoking `build.sh`, it should be trivial to run exactly that in any CI.

Re: Using Radicle CI

#14

Post by ambicapter »

maratc wrote: post #13
kuehle wrote: post #2

> I find the most frustrating part of using CI to be to wait for a CI run to finish on a server and then try to deduce from the run log what went wrong. I’ve alleviated this by writing an extension to rad to run CI locally: rad-ci. locally running CI should be more common

This can be achieved by running in CI what commonly runs on local. E.g. if your build process is simply invoking `build.sh`, it should be trivial to run exactly that in any CI.

This is fine until your run into differences between your machine and the CI one (or you're writing code for a different architecture than the one you're using), but I agree, this is definitely the first step.

Re: Using Radicle CI

#16

Post by maratc »

ambicapter wrote: post #14
maratc wrote: post #13

Earlier quoted context omitted.

This can be achieved by running in CI what commonly runs on local. E.g. if your build process is simply invoking `build.sh`, it should be trivial to run exactly that in any CI.

This is fine until your run into differences between your machine and the CI one (or you're writing code for a different architecture than the one you're using), but I agree, this is definitely the first step.

I agree, but if there's an architecture gap then locally running CI is not gonna help you to bridge it either.

Re: Using Radicle CI

#17

Post by esafak »

maratc wrote: post #13
kuehle wrote: post #2

> I find the most frustrating part of using CI to be to wait for a CI run to finish on a server and then try to deduce from the run log what went wrong. I’ve alleviated this by writing an extension to rad to run CI locally: rad-ci. locally running CI should be more common

This can be achieved by running in CI what commonly runs on local. E.g. if your build process is simply invoking `build.sh`, it should be trivial to run exactly that in any CI.

Be sure to run it in a container, so you have a semblance of parity.

Re: Using Radicle CI

#18

Post by esafak »

kuehle wrote: post #2

> I find the most frustrating part of using CI to be to wait for a CI run to finish on a server and then try to deduce from the run log what went wrong. I’ve alleviated this by writing an extension to rad to run CI locally: rad-ci. locally running CI should be more common

dagger does this, at the expense of increased complexity.

Re: Using Radicle CI

#19

Post by __MatrixMan__ »

kuehle wrote: post #2

> I find the most frustrating part of using CI to be to wait for a CI run to finish on a server and then try to deduce from the run log what went wrong. I’ve alleviated this by writing an extension to rad to run CI locally: rad-ci. locally running CI should be more common

Agreed. I've got a crazy idea that I think might help...

Most tests have a step where you collect some data, and another step where you make assertions about that data. Normally, that data only ever lived in a variable, so it is not kept around for later analysis. All you get when you're viewing a failed test is logs with either exception or a failed assertion. It's not enough to tell a full story, and I think this contributes to the frustration you're talking about.

I've been playing with the idea that all of the data generation should happen first (since it's the slow part), it then gets added to a commit (overwriting data from the previous CI run) and then all of the assertions should run afterwards (this is typically very fast).

So when CI fails, you can pull the updated branch and either:

- rerun the assertions without bothering to regenerate the data (faster, and useful if the fix is changing an assertion)

- diff the new data against data from the previous run (often instructive about the nature of the breakage)

- regenerate the data and diff it against whatever caused CI to fail (useful for knowing that your change will indeed make CI happy once more)

Most tools are uncomfortable with using git to transfer state from the failed CI run to your local machine so you can just rerun the relevant parts locally, so there's some hackery involved, but when it works out it feels like a superpower.

Re: Using Radicle CI

#20

Post by girvo »

8n4vidtmkvmk wrote: post #10
apwell23 wrote: post #6

Earlier quoted context omitted.

> For example, it may run a lot more tests than what's practical in a local system. yes this is what i was taking about. If there are a lots of tests that are not practical to run locally then they are bad tests no matter how useful one might think they are. only good tests are the ones that run fast. It is also a sign that code itself is bad that you are forced to write tests that interact with outside world. For ex…

It does suck when those large scale integration tests fail but sometimes that's the only real way to test something. E.g. I have to call a service owned by another team. It has a schema and documentation so I can mock out what I think it will return, but how will I truly know the API is going to do what it says or what I think it says without actually calling the API?

Tbf that’s what post-deployment verification tests sre ideal for, instead of as integration/e2e tests blocking your merges/deployments
Post reply on HN