Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do I learn how to become a good sysadmin?

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 124 posts

Re: Ask HN: How do I learn how to become a good sysadmin?

#81

Post by lighthawk »

TheCowboy wrote: post #51
protomyth wrote: post #30

Start with two mantras: 1. I will know exactly what every command or script I run on a system I control is supposed to do - no exceptions. If I don't and are just following instructions, I really need to learn what it means and why. If you need to setup a test system and snapshot before and after to see how things work. 2. I will document a lot. Imagine some poor person showing up after you have won the lottery (thin…

This is good, and I'd add on to the part about the successes. Learn to document your successes and be able to verbally communicate why anything you do is important or useful to less technical users. If you don't have a good boss who can see that you are good at what you do, you will have to be able to speak up if you want to be paid what you are worth. You want to be a step up from a computer janitor who needs to be…

> Learn to document your successes and be able to verbally communicate why anything you do is important or useful to less technical users.

And on the software engineer/developer side of things the same applies. This is why whenever I am given a self-assessment or asked to help with a review of myself, I go back through my git log, email, etc. looking for what I've done instead of just attempting to summarize based on memory. Then I keep a personal copy of my self-assessment. That way, I have a record of what I did, and so does my company. Wikis, file servers, and other document repositories change, and when you switch jobs, you have that available to look at to update your resume. If your company doesn't make you do at least annual and hopefully quarterly self-assessments, you should do it on your own.

Re: Ask HN: How do I learn how to become a good sysadmin?

#82

Post by emodendroket »

It's a different discipline and you don't really need to be an expert in that and development; you should pick one. That's not to say you should be totally ignorant of administration details, but, really, you could devote all of your time to it if you wanted.

Re: Ask HN: How do I learn how to become a good sysadmin?

#83

Post by marcosdumay »

rdudek wrote: post #24
mvanvoorden wrote: post #23

Earlier quoted context omitted.

I'm a sysadmin for 16 years, I have worked in big and small companies and I don't agree with a great part of this post. I'm not a developer, I have never needed to disassemble or fix programs built by others, never needed to understand kernel subsystems or anything else kernel related (except may be how to replace a broken driver/module). I know nothing of C, I know just the basics of system calls and I've never hear…

I agree. When we hire sysadmins, we look for folks with experience using various products and weigh in extra bonus points for folks who are also SecOps. We could care less if a person knows how to code. That's what we have the developers for.

Lots of people agree.

Those end up creating environments where everything just sorta-works enough to run, for as long as you put a constant stream of sysadmin time into it.

And then as see they doubting people that claim that a person can administer hundreds of servers, or that an admin can go away for a while and nothing will happen.

(Also, exactly what do you look for in SecOps? Most people that use that term are selecting for exactly the wrong things.)

Re: Ask HN: How do I learn how to become a good sysadmin?

#84

Post by thaumaturgy »

kpcyrd wrote: post #75
Mahn wrote: post #35

Earlier quoted context omitted.

Following random guides on the internet doesn't necesarily have to be harmful if you don't simply copy-paste, but rather make an effort to understand what is the advice being given, why is it being given, and form an opinion about it. For instance, assuming you start from the very beginning, if a guide suggests disabling root login and you do your own research to understand what root accounts are, what can they do, a…

There are some surprises if you go down that hole. The best argument I could find for "disable root login" was "the attacker has to guess the username, too", which doesn't align with Kerckhoffs's principle and isn't the way security should be done, imo. Also, fail2ban is a protection against bruteforce. If bruteforce is an issue for you, you're doing something wrong. Please correct me if you know more.

Some of the brute force attempts against servers are so relentless now that they can consume a significant amount of server resources just causing the server to say, "no. no. no. no. no. no. no. no. no...." They also fill up your log files, needlessly consuming disk space and making it a pain to crawl through logs later on to troubleshoot legitimate issues. Plus, you can hook Fail2Ban so that other services can use it to buff up their filters. For instance, if someone's spamming your mail server, your mail server can trigger Fail2Ban and then Fail2Ban can tell your web server to also block the IP (or network) to help reduce common sources of WordPress spam.

There are good reasons to use Fail2Ban, and the counterarguments that it doesn't actually improve security miss all the other benefits it brings.

And, I've read all of Theo de Raadt's arguments against these approaches. I understand and mostly agree with them. I get that with ssh key only authentication and sane services configuration and so on that people can hammer away at your server all day and never accomplish anything. But that still doesn't mean I want to provide a test bed for every dumb script kiddie on the internet (and there are many).

Re: Ask HN: How do I learn how to become a good sysadmin?

#85

Post by microtonal »

thaumaturgy wrote: post #84
kpcyrd wrote: post #75

Earlier quoted context omitted.

There are some surprises if you go down that hole. The best argument I could find for "disable root login" was "the attacker has to guess the username, too", which doesn't align with Kerckhoffs's principle and isn't the way security should be done, imo. Also, fail2ban is a protection against bruteforce. If bruteforce is an issue for you, you're doing something wrong. Please correct me if you know more.

Some of the brute force attempts against servers are so relentless now that they can consume a significant amount of server resources just causing the server to say, "no. no. no. no. no. no. no. no. no...." They also fill up your log files, needlessly consuming disk space and making it a pain to crawl through logs later on to troubleshoot legitimate issues. Plus, you can hook Fail2Ban so that other services can use i…

Use the simplest tool possible. Fail2ban relies on log parsing, which is a possible attack vector.

The thing is that you can reach pretty much the same effect with a smaller attack service and better efficiency using rate limiting in your packet filter.

E.g. in iptables the 'recent' module can do this, see man man iptables-extensions and search for 'recent'. E.g. you can set up a rule that any IP address making more than 5 connection attempts to port 22 in one minute gets put on a list that is DROPped.

Edit: BTW, if you think the fail2ban attack vector is purely theoretical, you might want to check the CVEs:

http://www.cvedetails.com/vulnerability-list/vendor_id-5567/...

Re: Ask HN: How do I learn how to become a good sysadmin?

#86

Post by Arubis »

There's a lot of good advice already in this thread, and I'd rather not just repeat it all in new phrasing. So, read it and weigh it and take what you like.

And then, when you're done, fire up a console window and unplug your mouse and put it somewhere that's really annoying to get to. Live with this for a week.

This will force you to live your life in a terminal, which means that all those little tasks become commands and scripts and configuration files. You will not have a GUI and you will have to understand how to make stuff work anyway.

Trust me, you will learn fast once you don't have a choice about it :)

Re: Ask HN: How do I learn how to become a good sysadmin?

#87

Post by protomyth »

hacknat wrote: post #60
protomyth wrote: post #30

Start with two mantras: 1. I will know exactly what every command or script I run on a system I control is supposed to do - no exceptions. If I don't and are just following instructions, I really need to learn what it means and why. If you need to setup a test system and snapshot before and after to see how things work. 2. I will document a lot. Imagine some poor person showing up after you have won the lottery (thin…

I would like to add that you should be patient and kind to the developers with whom you are working. When they have product successes, they may call attention to your efforts as being tied to its success. This is a good thing; having your job tied to profit-center activity is very good. Even if attention isn't drawn to your work, smart developers know how incredibly valuable smart, communicative sysadmins are, they w…

You should be patient and kind with everyone. Although, patient and kind does not included adding untested code to production to "fix" something late on a Sunday night. Insisting on proper deployment almost always makes developers[1] irritable. It also prevents http://dougseven.com/2014/04/17/knightmare-a-devops-cautiona...

1) I have more years as a developer than system admin (11 vs 7 and 5 as something I'm still not sure).

Re: Ask HN: How do I learn how to become a good sysadmin?

#88

Post by mtalantikite »

ocdtrekkie wrote: post #45
mtalantikite wrote: post #42

Sysadmin is (unfortunately) a role that is on the decline, so from a pure employability perspective I'd suggest you focus more on the dev side. As for the skills, I'd suggest running a Linux distro as your personal, everyday machine, not just a server you log into on AWS or DO every so often to configure (which, also -- don't do that. You don't want snowflakes in your environment). It'll force you to learn a lot abou…

Why would you say that the need for Sysadmins is on the decline? If anything, I should think it would be greater than ever. I'm very curious here.

It's not that sysadmins are going to disappear, it's just that with IaaS and the automation tooling that's been developed in the past decade teams don't need to be nearly as large. The role has also changed.

A few people can manage a deployment of a thousand server instances now fairly easily (I've been on teams like that). A decade ago you'd be renting colo space, racking/stacking yourself, managing your networks, swapping dead hardware, and managing all the software that goes on top (I've also been on a team like that). You'd need a large team dedicated to just ops and sysadmining.

Hiring today is different. A sysadmin didn't necessarily need to know how to code beyond some scripting with bash or perl. These days in order to manage the complexity of large cloud systems you probably should be a solid developer in addition to having a deep knowledge of systems. Or if you're a small startup you'll probably have your devs work additionally on your infrastructure or use a PaaS.

Re: Ask HN: How do I learn how to become a good sysadmin?

#89

Post by thaumaturgy »

microtonal wrote: post #85
thaumaturgy wrote: post #84

Earlier quoted context omitted.

Some of the brute force attempts against servers are so relentless now that they can consume a significant amount of server resources just causing the server to say, "no. no. no. no. no. no. no. no. no...." They also fill up your log files, needlessly consuming disk space and making it a pain to crawl through logs later on to troubleshoot legitimate issues. Plus, you can hook Fail2Ban so that other services can use i…

Use the simplest tool possible. Fail2ban relies on log parsing, which is a possible attack vector. The thing is that you can reach pretty much the same effect with a smaller attack service and better efficiency using rate limiting in your packet filter. E.g. in iptables the 'recent' module can do this, see man man iptables-extensions and search for 'recent'. E.g. you can set up a rule that any IP address making more…

iptables rate limiting still doesn't solve the problem of identifying attacks against one service so that they can be preemptively blocked by other services on other servers.

Re: Ask HN: How do I learn how to become a good sysadmin?

#90

Post by exelius »

ocdtrekkie wrote: post #46
mvanvoorden wrote: post #23

Earlier quoted context omitted.

I'm a sysadmin for 16 years, I have worked in big and small companies and I don't agree with a great part of this post. I'm not a developer, I have never needed to disassemble or fix programs built by others, never needed to understand kernel subsystems or anything else kernel related (except may be how to replace a broken driver/module). I know nothing of C, I know just the basics of system calls and I've never hear…

Absolutely agree. Sysadmins' interface with code is basically to be a compliance-checker. Software, be it from in-house developers or a third party solution need to meet that compliance bar, or it doesn't go to production. If your in-house developers are unable to meet those compliance requirements, that's a them issue and not a you issue. It's good for devs to know some ops, and ops to know some dev from a familiari…

The entire problem here is the division of "us" and "them". From the customer's perspective, you are both "them". If the product doesn't work, you're both to blame no matter whose fault it was.

Work with your developers to make the application more supportable. Show them the problems you are having and ask for their help in fixing them. Too many sysadmins just throw it over the fence and say "not my problem, you fix it" and that's honestly not acceptable.

This is the entire reason "DevOps" is a thing. They are not fundamentally different roles; you're both involved in building a system that does things for customers. You bring different skills to the table, but that's often the case with every developer: you probably have a dev who is a whiz with databases, another who knows some other library really well... a sysadmin skill set is no different.

Post reply on HN