Ask HN: Tips on Sysadmin Job

Okay, 3 rules for sysadmin positions:

- The most important rule is that your recovery from backup procedure is working properly. - The second most important rule is that your backups are working properly. - The (distant) third most important rule is to always individually backup any text file you edit before you edit it.

Seriously. These are the most important rules.

Other pro-tips are to test any changes to remote access (SSH most importantly) before you close the session that you made the changes in. Also, schedule restore from system backup and restart before you make any potentially crippling changes to the system configuration. If the changes work then you can cancel the restore / restart. If the changes cripple the system it will (if you've followed rules one and two above) recover itself. Finally, if you can afford it, try to run at least one dev / test instance of each critical system. You can make changes, point a few test clients at the new instance, make sure it's working, then either make the changes to the prod system, or do the dev / prod swap-over.

The actual ways to achieve these rules varies depending on OS mix, hardware, cloud providers (if any), use of certain tech (are you using virtualization? Are you using containers? Are you using some form of orchestration tooling? etc.) and many other things.

The first two rules mentioned above, and the order in which they're mentioned, are by far the most important rules in sysadmin. Ignore them at your own peril ;-)

Good luck. Keep a level head, don't panic, and test your backups!

- What's your ticket system like? If people can just drop by your desk and ask about their printer that's something you'll want to change

- I know with your security background when Bob in accounts demands a domain admin password you won't just give it to him. But who has your back on that? Will he cry to the CEO who in turn will ask you to please do your job and give it to him? Get some relevant policies reviewed and signed off.

A lot of good advice is already thrown around. Let me add another tip:

Always do smoke testing. After setting up a system, and it starts working, reboot once. No preparation, nothing. Just reboot out of the blue.

If the system boots and works as normal, you're done. If not, fix the problems and retry.

Will save a lot of headaches down the line.

Also, document all your procedures. Everything. To something tidy & local. Something like TiddlyWiki or a tool of your choice.

I’m a dev who got sucked into some ops/sysadmin work. Not a whole department, but I think my advice would still apply. The most effective thing I did was a custom script to test everything I was responsible for and send me a nice formatted email w the results. I’m not talking about canned email from Helpdesk or network monitor, but tests like “is the backup file modified date the expected value”, “is disk space on X at least N% free”, “are Z services up”? All in a single email, every morning. I wrote mine in powershell, but you do you.

If something bad happens, add a test for it to that email. It’s not an end all, but helps be proactive and keep pulse of things you are on the hook for. Real-time alerts etc can be useful, but also a lot of noise.

Also a kanban board like Trello or Notion has been useful

There is only one tip for you: if you are not programming, you are going to suck.

I have never met very good sysadmin which wasn't programmer at one point.

I am not talking about hard core programming involving serious patterns, DI, migrations etc. But most of the time, you should be writing good, resilient, scripts in more serious shell language such as PowerShell. If you don't do that, your actions are not reproducible and thus your output is prone to interpretations (i.e. it sucks). Plus, there is no such thing in enterprise as one-time-tasks - if you don't script your task you will have to manually repeat it sooner or later and that is extremely inefficient and error prone.