Ask HN: How do I learn how to become a good sysadmin?

Just because Hadoop's install process sucks that doesn't make you a bad sysadmin. Back in the day when we were building our kernels most of us weren't verifying download package sigs either. Containers aren't making us dumber either.

A good sys admin just knows all of the working parts of a server and knows about the latest tools. The only way to get better is hands on practice. Try setting up your own Hadoop cluster from source and run a job through it, then put Hive on it, then attempt to containerize a node ... which not even work but learning why that won't work becomes useful knowledge. Just stumble through it and learn.

I am not a sysadmin, so probably there are better specific advice.

But, from the point of view of a developer, the thing that I appreciate the most on a fellow sysadmins is to be calm and methodic at all times. Organised. Having a plan and know what to do. Being ready for disasters.

Do we need to update a security patch on every single server? Ok, list of servers, start with server one, finish with the last one, don't let any one fall behind.

A server suddenly catches fire? No problem, remove it from the load balancer, get a fire extinguisher, remove it, order another one, recover from backup.

Is there a problem on production? What could be wrong? Check logs, think a little, then try to fix it. Do postmortem and come with improvements. Try not to be bitten again for the same thing.

In mi opinion, the core of good sysadmin is to minimise risks and errors in the stability of the system. Mistakes can (and will) happen, but the aim is to make them only once. I think it has a big component of learn from battle stories.

Processes and servers can fail, but the whole system should hold up.

All of the mentioned points are very good (and applicable) to performing the job properly. But for me it is a matter of curiosity. If I need a function to work, I want to know how it works. If I need an application installed, I want to know its underlying pieces, how it communicates, what dependencies it has, etc. At some point you have to draw the line of course, as it is impossible to know everything, but that line is generally easy to find. For example, I could put you to sleep talking on how TLS handshakes work, but I have no interest or desire in learning the math behind any used algorithms (although I do keep abreast of security issues with ciphers, etc.). Learning new options (say the proliferation of systemd in linux land) is always good, even if you ultimately decide not to implement it at this point in time. So just keep learning, always find new ways to do things, audit your own systems, etc.

I used to believe this. I was aggravated by the pager, and would happily acknowledge problems in Nagios just to make the damned thing shut up. More and more pages would come in, the volume increasing each day as "non-issues" piled up.

The third time this behavior caused downtime for my clients, I wised up and took my own advice of "uptime is king". I took the "every alert is actionable" to heart, and took a few moments to realize that the action can also be against the monitoring.

Alert for disk at 80% on a 2TB volume? Action: Verify growth in Graphite isn't out of the ordinary, and increase the warning threshold to give you about 3-4 weeks of notice that you might need to get bigger disks.

Alert for an excessive number of 404 responses? Browse the nginx logs and identify someone trying to hack your corporate-mandated WordPress install. Verify they aren't making any traction, and add exceptions to your 404's so you don't alert on known (and non-whitelisted) endpoints they're hitting.

Alert for memory at 80%? What's consuming it; do I have a memory leak and need to restart something? If all is well, and MySQL is just being greedy, up the alert to 90%.

Disk capacity warning at 3am? Put some hours around the warning notification, and add a separate "things are growing out of control" alert which doesn't have hours.

API endpoint is not responding again, but it's not the system at fault? Add the API developer to the notifications for that alert and remove yourself for a week or two.

After an admittedly harrying week of this, you gain two things. One: an operational understanding of what is going on in your system. Two: an alerting system which is tuned to your use case, and which lets you know when you have real problems. Remember - uptime is king.

I'd recommend My Philosophy on Alerting by Rob Ewaschuk - https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa... as a good approach to take.

We've fleshed this out a bit more in the Prometheus best practices at http://prometheus.io/docs/practices/alerting/

Taking this approach at our company greatly reduced the alert count and improved responsiveness with no degradation in service.