101–110 of 126 posts
It's basically the minimal version of a standalone workflow automation; it's got crappy state storage and is really only good for a single human execution unit per script (or perhaps multiple swapped sequentially rather than in parallel), but it's an instance of a fairly well-known class of do-something software.
Why do I think that writing a script in Expect ( https://en.wikipedia.org/wiki/Expect ) would take as long as doing this and then you would be done?
An example is approving new users to access a service. Say your list of users is a YAML file. Someone edits the file, fires off a PR, waits for someone to approve it, merges it, and applies the config. You might think it's so easy to do by hand, automating it would be a waste. But if you actually add up the time it takes for humans to do those steps, plus the lead-time from creation to completion, plus the occasional typo, wrong username, or Norway bug, it can take from hours to days to actually complete the task. By automating steps one at a time, you gradually move toward the entire process being a single script that takes a username, and completes the task in about 30 seconds. This is a small win, but added up over all your operations, reduces hundreds of hours of toil and reduces human error. If every engineer automates one step of one runbook per week, over a year you'll have hundreds of reusable automation steps.
To the OP, I'd add the following: 1) use the least-complex way to accomplish the task. If you don't need to write Python, don't. Bourne shell is your friend. Re-use existing tools and automated features. 2) Add documentation to the code that can be auto-generated, so you can use the script as long-lived static documentation. Documenting somewhere else will lead to stale docs. 3) Run all your runbooks from a Docker container. 4) Plan to run these runbooks-in-containers from a CI/CD system using paramaterized builds. 5) Organize your reusable automation steps by technology, but have product-specific runbooks that call the automated features. 6) In a big org, a monorepo of automation features is an efficient way to expose automated features to other teams and ramp up on shared automation quicker.
This is a good idea. I recently wrote down some ideas about "How to Grow a Program," in a symbolic programming environment: https://github.com/theronic/hoist/ I call it Hoist - from "hoisting up" concrete, hard-coded values into abstract symbols for reuse. It's complete vaporware at this point, just a place to collect my thoughts for a future essay.
I get it, but this script looks a lot like a checklist to me. Seems like a checklist is a good place to start, no? If you want more "interest" during the process, make it a point to refine the documentation for your process (clarify a step here, split one command into two steps). Then you have a very clear spec by the time the process needs to be automated.
Earlier quoted context omitted.
What other approach do you suggest, then, in the extremely common scenario where you have an existing, tedious yet reliable multi-step manual process, and you’d like to automate it?
I think you just have to look at the process and consider why is it being tedious. Is there anything specific humans do? Often there are many small decisions hidden in the checklist. The second thing I would do would be to make sure that the automation happens in a "clean" environment, or at least, there is a clear boundary. Humans can adapt to changing conditions and have no issues to adapt the checklists where circ…
This pretty much is the opposite of an employee onboarding environment. Much of the discussion here is about automating things via scripts. A typical large enterprise uses a bunch of HR and related systems that are fairly resistant to automation. If you can automate, it may be a massive saga involving cooperation from the vendor of some arcane software.
e.g there is no concept of a "clean environment" when you are interacting with a benefits system, a payroll system build last century, a manual procedure where Barry downstairs has to cut you an access card, etc. etc.
You can aspire to automate these steps but they are 100 x as hard as running bash scripts on a box. And onboarding new employees does not typically happen often enough to justify the effort.
Earlier quoted context omitted.
I don't understand why you would want to introduce Slack to this. What benefits would that give?
Slack is like the shared OS we never had
The only interpretation of that I'm coming up with is that each step of the process would be kicked off in Slack.
Wouldn't that mean people could easily stomp on each other's toes? Wouldn't there also be tons of needless noise as each of us does our workflows?
I use Slack daily, but we don't have any automation hooked up in it. I realize some teams drive deploys, alerts, and other stuff through it. That makes sense to me - this does not.
Am I missing something?
I start with my "template" script and fill it out.
totally over-engineered for most tasks, but then as the tasks mature they have a firm foundation.
very simplified example:
#!/usr/bin/python
import sys,os,argparse
parser = argparse.ArgumentParser(argument_default=None)
parser.add_argument('-d', '--debug', action='store_true', help='debug flag')
parser.add_argument('files', nargs='*', default=[], help='files to process')
arg = parser.parse_args()
def die(errmsg,rc=1):
print(errmsg, file=sys.stderr)
sys.exit(rc)
if not arg.files:
die('filename required')Whats a good way to generalize such a python script so that windows and nix users can run it?
Last time, I figured I'd start going at it gradually. So what I did was I turned the checklist into a bash script, telling me what to do at each step. I also implemented the ones that are easy to automate. The plan is to knock out two or three items each time I do the backup, until it's fully automated.