11–17 of 17 posts
…otherwise we just throw everything into a scripts directory. Keep it flat until it feels messy, then start organizing. All documentation must live with the code, otherwise it gets out of date too easily.
If they were truly one-off, you wouldn't need to manage them. Categorize and document. SQL queries for analytics go into something like Metabase. Database migrations go into git. Frequent adhoc scripts go into something like Jenkins so you can trigger them easily. So on and so forth. All of those documented somewhere, like the team's Confluence/Notion/whatever. But don't just leave them at that, do work on eliminatin…
I'm curious about your workflow for reusable scripts with Jenkins. What are the advantages to putting it in Jenkins? How do you set up Jenkins to run these individual scripts?
But this isn’t good, we have Airflow and that’s where these things should be. Their stay inside Jenkins is technical debt, but at least is much better than not having it automated. The advantage is that you made an ad-hoc process less unstructured, you can add post-failure alarms, you don’t have to worry about long running scripts being interrupted by shabby notebooks, you don’t need to spread production credentials to every developer and so on.
The classic is the /scripts directory in your codebase (presumably in VCS). The key is also to trawl through occasionally and remove anything that hasn't been used in a while.
We introduce many things that way. First it's just a few commands that do something useful. Then included in the repo. Then tweak. Next thing you know there's an endpoint with that functionality because it's useful and frequently needed.