Ask HN: How do I learn how to become a good sysadmin?

> ... being a good sysadmin ... means striving for simplicity, documenting and standardizing everything and being meticulous.

This. And you learn by not doing this and still having to maintain an infrastructure. After a while you will either start, hang yourself or get fired.

Learn how to make basic autotools project (configure.ac, makefile.am, etc)

Learn how FreeBSD's ports system (building packages from sources) work and why it is so.

Then learn what Redhat Package Manager (RPM) is and how to make a source rpm.

Get enlightened.)

Following random guides on the internet doesn't necesarily have to be harmful if you don't simply copy-paste, but rather make an effort to understand what is the advice being given, why is it being given, and form an opinion about it. For instance, assuming you start from the very beginning, if a guide suggests disabling root login and you do your own research to understand what root accounts are, what can they do, and why would you want to disable it, and after doing your homework you happen to agree it is a good idea, then it's probably okay to follow that specific advice.

That doesn't mean you are going to learn everything there is to server administration by following a random guide on the internet, but for a small startup it's not necesarily a bad way to get started since at the end of the day you are going to learn more from experience than by reading 50 books.

As a long-time sysadmin, my best advice for learning is to give yourself the opportunity to break things. Constantly ask yourself questions about how things work and then see if you can answer them. I wouldn't even begin to start worrying about containers until you're more comfortable with the basics.

Some questions and projects to get you started:

Do you know how to start and stop services, how to tell what's running, how to see what network activity is happening?

Try installing Apache or Nginx on an EC2 instance, then see if you can change how it works. Change where it logs to, or change what port it listens on, or change what directory it pulls files from. Can you change where it reads its configuration from? What happens if you give it a bad configuration? Where does it log errors? Can you break it and then fix it?

Spin up a new EC2 instance and try to see if you can lock yourself out of it or otherwise disable it. What happens if you kill the SSH daemon? Or change the port? Or delete your user's entry in /etc/shadow? What happens if you run "rm -rf /"? What do all the fields in /etc/passwd do? What if you delete your home directory? Or "export PATH="? What happens when you delete files that a running program is using? What happens when you fill up the disk? When you run too many processes? Can you make the thing crash? When something gets screwed up, do you understand why?

Run "ls /bin" and see how many of those commands you know what they do. Pick one you don't know how to use but which you've heard of and try to figure out how to use it. Look at the man page, run the command with "-?" or "--help". Play around with it till you feel comfortable. Then pick another one tomorrow and do the same.

Run "ls /etc" and pick a file and try to find out what it's for. See if you can do something interesting by changing the contents of the file. You might need to reboot or restart a service. Tomorrow pick something else from /etc.

Do the same with /usr/bin, /usr/lib, /var/lib. Figure out where the files in /var/log come from and how you can write to them and how you can change their names and how many there are.

Set up two instances and see if you can get them to talk to each other. On EC2 each host has two network interfaces. Do you know how to find both IPs? Can you set up MySQL on one, and connect to it from the other? Once you get that to work, can you block it? Spin up a third instance and see if you can figure out how to make MySQL accessible to one and not the other. Once you figure out one way to do it, figure out another way. Can you get a service to listen on one IP and not the other? Both?

What's the difference between UDP and TCP? Set up NFS. Set up a RAID. Break a RAID, rebuild it. Do you understand crontabs? syslog? Can you send mail to yourself?

Once you're feeling more comfortable with the environment, and you start actually fulfilling a sysadmin role, your basic philosophy should be:

* Expect anything and everything to fail.

* Trust nothing, even your own software and machines.

* Grant the least possible access to make things work. Developers and vendors will always ask for more, and if you are not pushing back against their requests, you are probably giving them too much.

* Always have a plan for rolling back a change.

* When something breaks and you fix it, don't stop working until you understand what went wrong and you take some steps to avoid it in the future.

* When you have a task that takes more than one step to complete that you do more than once, write a script to do it for you.

That should get you started...

You are right with the following blog posts blindly is a bad idea. And I feel guilty that I didn't invest time to learn why I commented those lines in IP conf (it's bad enough that I don't remember which files I changed). So I've definitely got a place to start working on already.

The mentorship part is a lot more difficult. Good sysadmins are really difficult to find in Sri Lanka. I've worked with a few companies so far. Some examples.

1) One company I worked for didn't even have a policy of hashed passwords and protection against SQL injection. They were developing major enterprise software.

2) I worked as an internal systems developer for a non IT team within another company. This was the one place I could have learnt the most at but the IT team was this very opaque "don't tell people what exactly we do" kind of team.

3) One last example. This other company that I worked at, the sysadmin was pretty good in keeping stuff up and running, but a lot of it was copy paste scripts. I got what I could out of the person but I couldn't pull out much.

Where I'm from, the main cyber security body of the country gave blank looks when asked about heartbleed at a conference held recently after the whole thing exploded.

All that to just sum up why I turned to HN to seek out advice as to what resources I should look at. A lot of threads on the net seem to veer more towards "be a good communicator" and "know your system". While necessary, it's a little too abstract for someone trying to find out what gaps exist and which ones need filling ASAP.

Thanks a lot for the advice. I'll probably start reading up on all those files I had to edit when hardening the server. That should provide a good starting point.