11–20 of 98 posts
EDIT. In particular the author hasn't acknowledged the prioritization problem, hasn't explained what factors might go into the necessary cost benefit analysis. The author merely points out "developer productivity is important!" You might as well say "invest in your tools!"
this muddying of terms is not useful. No, my IDE breaking down is not a production system failure. It's a development system failure and sure, that can very easily have consequences on the production system but usually not immediately. It should be obvious to everyone how having a functional hammer is going to be important if you want to fix a house. There is no need to start calling the hammer the house.
Shameless plug: To fix all those development pipeline issues, I am working on StableBuild, a tool to easily freeze and pin Docker images, operating system packages, Python packages, and arbitrary build dependencies; in 5 lines of code: https://stablebuild.com .
Once you enumerate what your development pipeline actually depends on, you will see it rely on a lot of stuff. A typical CI run pulls from npm, PyPI, Docker Hub, a distro mirror and a handful of third-party, none under your control, none with an SLA to you. Third-party outages, yanked versions, compromised packages are the things that can waste your time. Some packages may also drop support for whatever distro you're…
Of course, anything deployed for use by the target customer is a production system. And anything that is under development is a dev system. Pipeline itself is not under development and it is in use by the target customer (developer). What's the confusion here?
Acknowledging that developers are in fact customers, is often where it comes undone in my experience.
In a lot of places, only revenue paying people are considered customers, everyone else is a cost centre.
Dev tools are often seen as a tax the business pays to be able to generate revenue. Investing in them, or treating them like the production level systems they are, isn’t intuitive for most, and for some a hard cost to justify.
You should never fix an incident by deploying another fix as it risks making things worse especially while everyone is panicking. Better to roll back - and spread non-backwards compatible work across multiple deploys that each are backwards compatible - always giving yourself a known good state to roll back to.
Rollback should not done be via your CI either! Set up some low dependency thing - a separate tool or script or system.
Once you've rolled back and got the system stable, then produced a fix at your leisure, that's when CI comes back into the picture :)
I guess the problem is visibility. In the same way that many companies don't take security seriously until it becomes a problem. If a problem never manifests, but was prevented, was it really a problem? Of course I know the importance of preventing problems, but how do you properly measure it? A data leak is immediately visible, but preventing them is less so.