21–30 of 49 posts
Earlier quoted context omitted.
This can be achieved by running in CI what commonly runs on local. E.g. if your build process is simply invoking `build.sh`, it should be trivial to run exactly that in any CI.
Be sure to run it in a container, so you have a semblance of parity.
However, there are stateless VMs and stateless BMs too.
Earlier quoted context omitted.
It does suck when those large scale integration tests fail but sometimes that's the only real way to test something. E.g. I have to call a service owned by another team. It has a schema and documentation so I can mock out what I think it will return, but how will I truly know the API is going to do what it says or what I think it says without actually calling the API?
> I truly know the API is going to do what it says or what I think it says without actually calling the API? what if the API changes all of sudden in production? what about cases where api stays the same but content of response is all wrong? how do tests protect you from that? edit: they are not hypothetical scenarios. wrong responses are way more common than schema breaking. tooling upsteam is often pretty good at c…
For your example, the best place to invest would be in that API's own test suite (e.g. sending its devs examples of usage that we rely on); but of course we can't rely on others to make our lives easier. Contracts can help with that, to make the API developers responsible for following some particular change notification process.
Still, such situations are hypothetical; whereas the sorts of integration tests that the parent is describing are useful to avoid our deployments from immediately blowing up.
Earlier quoted context omitted.
Be sure to run it in a container, so you have a semblance of parity.
Where possible. (If your build process builds containers and your tests get them up and make them talk, doing that in a container is a challenge.) However, there are stateless VMs and stateless BMs too.
Earlier quoted context omitted.
Where possible. (If your build process builds containers and your tests get them up and make them talk, doing that in a container is a challenge.) However, there are stateless VMs and stateless BMs too.
What is a BM?
> I find the most frustrating part of using CI to be to wait for a CI run to finish on a server and then try to deduce from the run log what went wrong. I’ve alleviated this by writing an extension to rad to run CI locally: rad-ci. locally running CI should be more common
> I find the most frustrating part of using CI to be to wait for a CI run to finish on a server and then try to deduce from the run log what went wrong. I’ve alleviated this by writing an extension to rad to run CI locally: rad-ci. locally running CI should be more common
And yet, that's technically not CI.
The whole point we started using automation servers as an integration point was to avoid the "it works on my machine" drama. (Have watched at least 5 seasons of it - they were all painful!).
+1 on running the test harness locally though (where feasible) before triggering the CI server.
Small scale: a project is almost small enough to run the build and tests locally, but you still want to have a consistent environment and avoid "works on my machine" problems.
Large scale: a project is so large that you need to leverage remote, distributed computing to run everything with a reasonable feedback loop, ideally under 10 minutes.
The opposite ends of the spectrum warrant different solutions. For small scale, actually being able to run the whole CI stack locally is ideal. For large scale, it's not feasible.
> A CI system that’s a joy to use, that sounds like a fantasy. What would it even be like? What would make using a CI system joyful to you?
I spent the past few years building RWX[1] to make a CI system joyful to use for large scale projects.
- Local CLI to read the workflow definitions locally and then run remotely. That way can you test changes to workflow definitions without having to commit and push.
- Remote breakpoints to pause execution at any point and connect via ssh, which is necessary when running on remote infrastructure.
- Automatic content-based caching with sandboxed executions, so that you can skip all of the duplicative steps that large scale CI otherwise would. Sandboxing ensures that the cache never produces false positives.
- Graph-based task definitions, rather than the 1 job : 1 VM model. This results in automatic and maximum parallelization, with no redundancy in setup for each job.
- The graph based model also provides an improved retry experience, and more flexibility in resource allocation. For example, one task in the DAG can crank up the CPU and memory without having to run more resources for downstream tasks (steps in other platforms).
We've made dozens of other improvements to the UX for projects with large build and test workflows. Big engineering teams love the experience.
[1] https://rwx.com
> I find the most frustrating part of using CI to be to wait for a CI run to finish on a server and then try to deduce from the run log what went wrong. I’ve alleviated this by writing an extension to rad to run CI locally: rad-ci. locally running CI should be more common
Earlier quoted context omitted.
This can be achieved by running in CI what commonly runs on local. E.g. if your build process is simply invoking `build.sh`, it should be trivial to run exactly that in any CI.
This is fine until your run into differences between your machine and the CI one (or you're writing code for a different architecture than the one you're using), but I agree, this is definitely the first step.