Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do I learn how to become a good sysadmin?

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 124 posts

Re: Ask HN: How do I learn how to become a good sysadmin?

#91

Post by exelius »

lamontcg wrote: post #72
ocdtrekkie wrote: post #46

Earlier quoted context omitted.

Absolutely agree. Sysadmins' interface with code is basically to be a compliance-checker. Software, be it from in-house developers or a third party solution need to meet that compliance bar, or it doesn't go to production. If your in-house developers are unable to meet those compliance requirements, that's a them issue and not a you issue. It's good for devs to know some ops, and ops to know some dev from a familiari…

Continuous Delivery is going to eliminate that from the equation. The software developer will write software and tests and if they pass the tests throughout the pipeline then it will ship to production automatically and potentially multiple times per day. Much faster than the heavy ITIL based change control process that requires you to be inserted in the middle of it will manage. The companies that do this will have…

Agreed, but for different reasons.

You still need a sysadmin to set up your CI systems in a continuous delivery world. You still need people to debug performance issues -- most developers don't know enough about system I/O to do that effectively.

You're right in that there's no longer some heavy-handed change control process, but wrong that sysadmins will go away. We just call them DevOps instead, but it's the same skill set, just embedded inside a dev team.

Re: Ask HN: How do I learn how to become a good sysadmin?

#92

Post by pjungwir »

If you want to learn to "think like a sysadmin", the book Time Management for System Administrators by Limoncelli is short, funny, and excellent---and covers far more than just time management. That will give you some sense of what to care about and why. It's very useful even as a dev or devops person.

If you are really serious about being a professional sysadmin, Limoncelli's other book is considered outstanding---but it's huge and I haven't read it yet. Again, it's a mix of non-technical goals and technical solutions. The Nemeth book is how I started (though I'm not a sysadmin) and it's also quite good, but more purely technical.

I wouldn't say to a beginner, "Know what every command does," but try to learn a new thing every day. Keep notes. I like to write private man pages (https://github.com/pjungwir/manpj), but do what works for you.

I think learning sysadmin skills mostly just takes time, so try to make each task double as a learning opportunity. Sometimes this requires a lot of digging, or reading, or "debugging". Here I probably agree with @protomyth: the holy grail is understanding. If you strive to achieve that (and don't just go with the first thing that appears to work) you will become better and better.

While you're learning the nitty-gritty tech stuff, also try to keep in mind the different priorities of a sysadmin over a developer. Both care about reducing their own effort and annoyances. A sysadmin wants stability (no 3 a.m. downtime), automation (easy to deploy/update/scale), transparency (monitoring, logging), auditability (logging), recoverability (failover, backups), controllability (runbooks), security. Maybe a real sysadmin could chime in with what I'm leaving out. :-)

Re: Ask HN: How do I learn how to become a good sysadmin?

#93

Post by jpgvm »

lamontcg wrote: post #61
jpgvm wrote: post #54

Earlier quoted context omitted.

I guess I come from a different background, work for different companies and have dealt with different problems. I am aware there is plenty of room for the traditional sysadmin in enterprise still but I am not quite sure how long those days will be around. Google and facebook where first to discover the "Site Reliability Engineer" but they were far from the last. SRE is becoming the new sysadmin and you can bet your…

I was doing "SRE" level system engineering at Amazon back in 2001 long before facebook ever existed. Also when I grew up as a junior system engineer in the mid-90s I looked up to "system administrators" like Wietse Venema who coded in C and wrote tcp_wrappers and postfix, and Larry Wall who wrote his own programming language. The System Administration / Software Developer is not a new invention of Facebook/Google. Wh…

Very true, I should have been more specific when I say "created". Popularised is probably a better term.

I worked with some old and crusty Perl and C hackers that called themselves sysadmins that definitely would have qualified as "SRE"s during the 90s, it's just they were taken for granted.

All the term SRE has done is create a role that is more appreciated and better compensated with a better understood set of responsibilities and required skills.

Re: Ask HN: How do I learn how to become a good sysadmin?

#94

Post by rdl »

kpcyrd wrote: post #75
Mahn wrote: post #35

Earlier quoted context omitted.

Following random guides on the internet doesn't necesarily have to be harmful if you don't simply copy-paste, but rather make an effort to understand what is the advice being given, why is it being given, and form an opinion about it. For instance, assuming you start from the very beginning, if a guide suggests disabling root login and you do your own research to understand what root accounts are, what can they do, a…

There are some surprises if you go down that hole. The best argument I could find for "disable root login" was "the attacker has to guess the username, too", which doesn't align with Kerckhoffs's principle and isn't the way security should be done, imo. Also, fail2ban is a protection against bruteforce. If bruteforce is an issue for you, you're doing something wrong. Please correct me if you know more.

Disable root login is part of the principle of making all access accountable to individuals, not to role accounts. Imagine how much more challenging things are forensically if you see a bunch of actions in the logs taken by "root" vs. by "joeg, the sysadmin who was fired a week later.

fail2ban helps with a lot of things. It keeps spam out of the logs. Some systems have high CPU cost per login (bcrypt), so similar systems can help prevent brute force attempts turning into (or being actually intended as) DoS.

Re: Ask HN: How do I learn how to become a good sysadmin?

#95

Post by joshbaptiste »

The best sysadmins I have seen understand the Operating system from top to bottom. The easiest way to understand an OS without being a OS developer is using tracing tools to solve problems such as Dtrace under FreeBSD or Solaris (Linux also has many tracing tools). I would also recommend watching CS 162 series from Berkeley on Youtube.

Re: Ask HN: How do I learn how to become a good sysadmin?

#96

Post by lamontcg »

exelius wrote: post #91
lamontcg wrote: post #72

Earlier quoted context omitted.

Continuous Delivery is going to eliminate that from the equation. The software developer will write software and tests and if they pass the tests throughout the pipeline then it will ship to production automatically and potentially multiple times per day. Much faster than the heavy ITIL based change control process that requires you to be inserted in the middle of it will manage. The companies that do this will have…

Agreed, but for different reasons. You still need a sysadmin to set up your CI systems in a continuous delivery world. You still need people to debug performance issues -- most developers don't know enough about system I/O to do that effectively. You're right in that there's no longer some heavy-handed change control process, but wrong that sysadmins will go away. We just call them DevOps instead, but it's the same s…

In order to be able to usefully debug perf issues you need to have a fairly deep understanding of systems architecture though or you're no better off than the software devs that you're slamming. There are a lot of "I don't need to learn programming" system admins whose knowledge of how to debug performance issues I could teach to a decent software developer in an afternoon. There are even more SAs who cargo cult completely incorrect assumptions and tend to throw money semi-randomly at problems because the last time they had a wonky server they upgraded the RAM so this time if they spend the money or more RAM that'll fix it too.

And yes, I've had to painfully explain why Linux servers report nearly all their memory as being fully used and what the VM free is was to a Principle Architect at a major internet firm with a string of PhDs and who had built the software architecture of the company from its founding. Controlling a huge chunk of the technical direction of the company and doesn't know the first thing about the VM or how to tell if a server is really out of RAM or not. There's still going to be a skillset which is closer to the hardware, but people need to really have that skillset. Most people who call themselves SAs do not actually have that skillset, they're just semi-technical people who wield root power and throw their weight around the company.

Re: Ask HN: How do I learn how to become a good sysadmin?

#97

Post by lamontcg »

angersock wrote: post #74
lamontcg wrote: post #55

Earlier quoted context omitted.

That is insufficent, try going through TCP/IP Illustrated Vol 1 at least and really learning TCP/IP and use tcpdump to rip apart packets. You should learn in detail the tcp state diagram and how to read netstat output. Know what the TCP 3-way handshake is and PTMU discovery and 4-way connection teardown and be able to apply that knowledge to debugging. Learn how to program in C at least well enough to write simple cl…

Much better than the "ignore blogposts" meme.

I read blogposts all the time, I do use google first if I don't know the answer myself offhand. I have the background, though, to be able to determine what smells correct and don't just type in all the commands from the first random web page that google kicks back at me...

Re: Ask HN: How do I learn how to become a good sysadmin?

#99

Post by exelius »

lamontcg wrote: post #96
exelius wrote: post #91

Earlier quoted context omitted.

Agreed, but for different reasons. You still need a sysadmin to set up your CI systems in a continuous delivery world. You still need people to debug performance issues -- most developers don't know enough about system I/O to do that effectively. You're right in that there's no longer some heavy-handed change control process, but wrong that sysadmins will go away. We just call them DevOps instead, but it's the same s…

In order to be able to usefully debug perf issues you need to have a fairly deep understanding of systems architecture though or you're no better off than the software devs that you're slamming. There are a lot of "I don't need to learn programming" system admins whose knowledge of how to debug performance issues I could teach to a decent software developer in an afternoon. There are even more SAs who cargo cult comp…

Yeah, a system administrator who doesn't know how to program is just as useless. Why avoid a supremely useful tool because you have some personal aversion to it? You describe people who really don't have a technical mindset to problem solving and were never taught that you need data to back up your hunches. Unfortunately, a lot of those people tend to get stuck in the "system administrator" role because many sysadmin tasks can be easy and repetitive, and putting them on those tasks keeps them away from anything they could break.

Architecture is a "10,000 foot view" type job. I do a good bit of architecture work, and I don't give a flying fuck about how much RAM is in a server or how many CPUs it has. I care about the function the server performs, whether that performance is adequate relative to current demand, and whether the architecture can scale to handle 10-100x that demand. When I was a sysadmin, I used to care about RAM/CPU tuning, but it's not relevant to my current work so I have forgotten a lot of technical details. I do know that if I think a system isn't adequately tuned to its use case, I can go talk to my performance testing team and they will investigate, generate a hypothesis and test that hypothesis.

Re: Ask HN: How do I learn how to become a good sysadmin?

#100

Post by ocdtrekkie »

exelius wrote: post #90
ocdtrekkie wrote: post #46

Earlier quoted context omitted.

Absolutely agree. Sysadmins' interface with code is basically to be a compliance-checker. Software, be it from in-house developers or a third party solution need to meet that compliance bar, or it doesn't go to production. If your in-house developers are unable to meet those compliance requirements, that's a them issue and not a you issue. It's good for devs to know some ops, and ops to know some dev from a familiari…

The entire problem here is the division of "us" and "them". From the customer's perspective, you are both "them". If the product doesn't work, you're both to blame no matter whose fault it was. Work with your developers to make the application more supportable. Show them the problems you are having and ask for their help in fixing them. Too many sysadmins just throw it over the fence and say "not my problem, you fix…

"Us and Them" makes it sounds adversarial, but it's not meant to be. It's about who is best equipped to fix the problem. More often than not, a problem with an in-house developed piece of code will be best fixed by the person who wrote that code.
Post reply on HN