Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do I learn how to become a good sysadmin?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 124 posts

Re: Ask HN: How do I learn how to become a good sysadmin?

#61

Post by lamontcg »

jpgvm wrote: post #54
mvanvoorden wrote: post #23

Earlier quoted context omitted.

I'm a sysadmin for 16 years, I have worked in big and small companies and I don't agree with a great part of this post. I'm not a developer, I have never needed to disassemble or fix programs built by others, never needed to understand kernel subsystems or anything else kernel related (except may be how to replace a broken driver/module). I know nothing of C, I know just the basics of system calls and I've never hear…

I guess I come from a different background, work for different companies and have dealt with different problems. I am aware there is plenty of room for the traditional sysadmin in enterprise still but I am not quite sure how long those days will be around. Google and facebook where first to discover the "Site Reliability Engineer" but they were far from the last. SRE is becoming the new sysadmin and you can bet your…

I was doing "SRE" level system engineering at Amazon back in 2001 long before facebook ever existed.

Also when I grew up as a junior system engineer in the mid-90s I looked up to "system administrators" like Wietse Venema who coded in C and wrote tcp_wrappers and postfix, and Larry Wall who wrote his own programming language.

The System Administration / Software Developer is not a new invention of Facebook/Google. What is "new" is the "System Administrator" that started getting hired in the late 90s who couldn't do anything other than install programs and configure them and maybe knew how to 'ps' and 'kill' and that's about it. Even the old school Enterprise-class system admins knew more about debugging crash dumps and using their tools even if they did call up IBM when they needed a patch.

Re: Ask HN: How do I learn how to become a good sysadmin?

#62

Post by ABNWZ »

lamontcg wrote: post #55
nstart wrote: post #41

Earlier quoted context omitted.

Would this list of books be a good starting point? http://everythingsysadmin.com/books.html

That is insufficent, try going through TCP/IP Illustrated Vol 1 at least and really learning TCP/IP and use tcpdump to rip apart packets. You should learn in detail the tcp state diagram and how to read netstat output. Know what the TCP 3-way handshake is and PTMU discovery and 4-way connection teardown and be able to apply that knowledge to debugging. Learn how to program in C at least well enough to write simple cl…

This is very extensive. Great wealth of content and knowledge in here. Is this all strictly necessary? Or is this how to MASTER your craft?

Re: Ask HN: How do I learn how to become a good sysadmin?

#64

Post by lamontcg »

mvanvoorden wrote: post #23
jpgvm wrote: post #8

At the end of the day it's important to find a mentor or join a bigger company where you can study under a more senior sysadmin. These days things are getting more complicated, to be a good sysadmin you also have to be a good developer. Usually as good and sometimes more than the people that write the apps that you will run, maintain and love long after they have decided they have something more shiny they could be w…

I'm a sysadmin for 16 years, I have worked in big and small companies and I don't agree with a great part of this post. I'm not a developer, I have never needed to disassemble or fix programs built by others, never needed to understand kernel subsystems or anything else kernel related (except may be how to replace a broken driver/module). I know nothing of C, I know just the basics of system calls and I've never hear…

You are literally the problem with system administration.

The attitude that you don't need to know how to program or understand the architecture of the systems that you run on is a highly privileged attitude that you are practitioner who can remain ignorant of their tools.

The good news is that kids these days are being raised on SREs and there's a billion people in India who will take your job and won't have an attitude about learning how to program. Ultimately, you are going to be a curious dinosaur. You are a unique product of the late 90s Internet bubble where the need for system administrators expanded so much faster than the available supply and anyone halfway technically minded got hired for really good salaries and put in charge of servers.

That is going to change and evolve.

And I spend 5-10 years learning all that knowledge back in the 90s and its 15 years later and I can state with confidence that there's no regrets. If anything my only regret is that I didn't learn how to really practice "software development" as opposed to "programming" even earlier. Going forwards, I expect that more an more the lines between systems and software is going to blur and MY advice to kids these days would be that they will only be able to be ignorant of software development practices at their own peril.

And I stated doing PC Tech work in the late 80s, became a Unix admin in the mid-90s, managed the configuration management system and base configuration of a site that grew to 30,000 linux servers in the 200Xs, and then after 15 years switched to being a Software Developer. I'm fully confident based on experience that you are offering absolutely terrible advice to someone who is just learning, and you are out of touch with the direction of your own field.

Re: Ask HN: How do I learn how to become a good sysadmin?

#65

Post by neonfreon »

ocdtrekkie wrote: post #45
mtalantikite wrote: post #42

Sysadmin is (unfortunately) a role that is on the decline, so from a pure employability perspective I'd suggest you focus more on the dev side. As for the skills, I'd suggest running a Linux distro as your personal, everyday machine, not just a server you log into on AWS or DO every so often to configure (which, also -- don't do that. You don't want snowflakes in your environment). It'll force you to learn a lot abou…

Why would you say that the need for Sysadmins is on the decline? If anything, I should think it would be greater than ever. I'm very curious here.

According to the US Bureau of Labor Statistics in 2012, the number Sys and Network Admin jobs will increase by 12% between 2012 and 2022. (http://l.md/gr4)

That's a few years out of date now though. I wonder if there are newer statistics somewhere.

Re: Ask HN: How do I learn how to become a good sysadmin?

#66

Post by lamontcg »

ABNWZ wrote: post #62
lamontcg wrote: post #55

Earlier quoted context omitted.

That is insufficent, try going through TCP/IP Illustrated Vol 1 at least and really learning TCP/IP and use tcpdump to rip apart packets. You should learn in detail the tcp state diagram and how to read netstat output. Know what the TCP 3-way handshake is and PTMU discovery and 4-way connection teardown and be able to apply that knowledge to debugging. Learn how to program in C at least well enough to write simple cl…

This is very extensive. Great wealth of content and knowledge in here. Is this all strictly necessary? Or is this how to MASTER your craft?

I certainly threw in things there which I would not consider a requirement for hiring. I DO expect basic knowledge of the TCP state diagram and ability to parse and explain netstat output in actual hiring interviews. I would not expect someone to have read Schimmel's book and know how kernel locking works in great detail.

My advice is to really learn how to write some toy C programs at minimum an learn the sockets API and really have a copy of APUE and use it occasionally. I highly recommend a project where you do some kind of trivial 'hello world' patch to the actual kernel sources and things of that nature.

I have actually, on my job, seen issues that require this kind of knowledge -- like one incident with massive amounts of TCP RSTs and I've cracked open the kernel sources and tracked down where the TCP stack can issue RSTs, and found a problem with TCP Timestamps and the PAWS algorithm and the way that our layer 4 hardware loadbalancers at the time were doing packet rewriting and preserving all the random TCP timestamps from the clients which was hitting very high velocity servers and when TIME_WAIT state sockets were hit with a 'backwards in time' TCP timestamp from a different client it would issue a RST and tear down the connection because of PAWS. You might be able to find that kind of problem by studying the TCP RFCs and books like TCP/IP Illustrated instead of opening the kernel sources, but its the kind of problem that you aren't going to be able to Google. And I don't really have the ability to write a [modern, multithreaded] TCP/IP stack (although I could certainly acquire it), but I have demonstrated the ability to read the sources and get real-life work done by doing so.

For all the candidates that I interview I want to see some indication that they can work without the Google safety net and show some ability to track down problems on their own.

Re: Ask HN: How do I learn how to become a good sysadmin?

#67

Post by drzaiusapelord »

crypt1d wrote: post #22

Sorry to disagree with a lot of posts here. >At the most basic level, when firing up a new server I follow the how to harden your server guides and install fail2ban, disable root login, enable ssh only login etc. This is simply wrong. There are two things wrong with this approach: 1. If I understood correctly, you are essentially following random guides on the internet about setting up your security rig. Not too far…

> It shows that you do not understand the core issue and what the main purpose of such tools is, how they function , etc.

The people who write those guides don't either. I almost never see mention of things like OSSEC or running a WAF or even a basic IDS/IPS. Just lots of naive "hackers get in through passwords, dont use passwords in ssh and enable fail2ban." That's on top of security via obscurity like changing port numbers. Uh, what?

Security needs to have a layered onion approach:

Network> IDS/IPS, Firewall

Server> OSSEC, SELinux, smart permissions, WAF, AV

Storage/Filesystem> Backups, tripwire

DB> Hashed passwords, etc

Software> Patching schedule, writing secure software from day 1, not giving into to team/client demands for insecurity

Monitoring> OSSEC alerts, nagios/zabbix alerts, tripwire alerts, etc

Once you get used to a security stack, this stuff comes easy. The problem is cheap VPS's have turned devs into sysadmins who just plant a naked linux box on the web, turn on fail2ban, and call it a day.

Re: Ask HN: How do I learn how to become a good sysadmin?

#69

Post by fsniper »

mvanvoorden wrote: post #23
jpgvm wrote: post #8

At the end of the day it's important to find a mentor or join a bigger company where you can study under a more senior sysadmin. These days things are getting more complicated, to be a good sysadmin you also have to be a good developer. Usually as good and sometimes more than the people that write the apps that you will run, maintain and love long after they have decided they have something more shiny they could be w…

I'm a sysadmin for 16 years, I have worked in big and small companies and I don't agree with a great part of this post. I'm not a developer, I have never needed to disassemble or fix programs built by others, never needed to understand kernel subsystems or anything else kernel related (except may be how to replace a broken driver/module). I know nothing of C, I know just the basics of system calls and I've never hear…

I'm sorry for you, my fellow sysadmin who has more years on his/her shoulders than me. I'm sorry because you lack the the ultimate knowledge needed to be a good system administator. You did not ever get hold of it.

The sysadmin practice is not only installing software and OS, doing basic troubleshooting. It's more than that. It's knowing systems inside out, from kernel internals to application internals. It's knowing and handling more area than anyone else in the field.

If you do not know about these, you do not automate, yo do not code, you do not debug, you become an IT helpdesk employee.

Re: Ask HN: How do I learn how to become a good sysadmin?

#70

Post by nstart »

daxfohl wrote: post #17

Not on Hacker News! Most of the people here (including myself) are developer hacks that think they can do sysadmin, which is probably worse than not knowing it at all.

I dunno. The replies I've got have so far pointed me in a fairly consistent direction that can't possibly leave me worse than where I am right now. What I do from there will be up to me (and anyone else who benefits from this thread) I guess.
Post reply on HN