Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Lost Art of System Administration

matt-rickard.com

Post reply on HN
Print view View original

71–80 of 98 posts

Re: The Lost Art of System Administration

#71

Post by bsagdiyev »

tasuki wrote: post #62
throwaway892238 wrote: post #35

Most of those things listed are just technical details that change over time. Before all the Linux trivia there was the Solaris trivia, AIX trivia, Novell, HP/UX, Windows NT, OS/2, VMS, AS/400. Alpha, Sparc, VAX, tape robots, IPX/SPX, Token ring, VT100 dumb terminals. Old heads have forgotten more technical crap than exists today. (I can barely remember how to set up a Cisco router, and I used to work there.. I do, h…

> No, I'm not giving you root on all the servers, tell me what you want to do. You just want to copy a file? You don't need root for that. I'm a developer, so I'm very biased. My interactions with system administrators were more along the lines of "I can't give you access to this system, you have to [come to my desk / hop on a zoom call] and tell me what to type into the shell." and me "Please type tail dash f, no, n…

Anecdotally I've had the opposite experience. Majority of developers I've interacted with didn't understand how anything systems admin side worked. This isn't to say all developers are like that, but that you may be more of an outlier in recent times. Also your admin team sounds awful to work with.

Re: The Lost Art of System Administration

#72

Post by arcbyte »

tasuki wrote: post #62
throwaway892238 wrote: post #35

Most of those things listed are just technical details that change over time. Before all the Linux trivia there was the Solaris trivia, AIX trivia, Novell, HP/UX, Windows NT, OS/2, VMS, AS/400. Alpha, Sparc, VAX, tape robots, IPX/SPX, Token ring, VT100 dumb terminals. Old heads have forgotten more technical crap than exists today. (I can barely remember how to set up a Cisco router, and I used to work there.. I do, h…

> No, I'm not giving you root on all the servers, tell me what you want to do. You just want to copy a file? You don't need root for that. I'm a developer, so I'm very biased. My interactions with system administrators were more along the lines of "I can't give you access to this system, you have to [come to my desk / hop on a zoom call] and tell me what to type into the shell." and me "Please type tail dash f, no, n…

A metaphor that just occurred to me.

When it comes to debugging running systems, nobody has it better than doctors. They knock the "system administrator" out and go right in with root access via scalpel to do what they know needs to be done.

SysAdmins can be a huge help in devops culture, but sometimes things go wrong and the surgeon needs to get in there.

Re: The Lost Art of System Administration

#73

Post by mfuzzey »

b112 wrote: post #45
Syonyk wrote: post #26

Earlier quoted context omitted.

> Have we reached that level of the civilisation that goes extinct, for forgetting how its own inner machines work? Probably. Almost nobody under the age of 40 works in the "deep weeds" anymore (firmware, kernel, casually "Ring 0 and below" spaces). A decade ago, it was about 30. I'm the last wave, it seems, of people in these spaces, and that's quite terrifying. But I get it. The hardware is maddeningly complex, rap…

I often wonder the same. I grew up at a time, where I had to learn assembly on the C64, to get my machine to handle > 300 baud with uploads/downloads, on a BBS I wrote. On the hardware side, it was very easy to learn how to fix said machines, with schematics provided in the back of the user manual. This level of understanding is not as easily possible today, and my understanding grew outward from that. I think two th…

Actually this was the much of the reason for the creation of the Raspberry Pi.

Its creator, Eben Upton was a director of studies at Cambridge university and realised that there were fewer CS applicants than in the past and that many of those that did apply knew far much less about computers and programming through there own "tinkering" than previously.

He saw that this was due to modern computers being far more opaque than the systems he had used as a kid (like the BBC Micro). So he decided to build something to address that issue that was open enough to understand the workings and cheap enough that people wouldn't be discouraged from tinkering due to the risk of breaking it.

It seems to have worked quite well...

Re: The Lost Art of System Administration

#74

Post by Gordonjcp »

hsn915 wrote: post #25

The fact that linux systems need adminstration is one of the reasons why they suck. Imagine after buying a macbook that you also need to hire a sysadmin to keep it running smoothly? The idea is laughable. > Learning how to effectively use the command line. Unix philosophy. Pipes, scripting, and tooling. So much of programming is stitching things together. This is really bad. Much of the complexity in contemporary sof…

Just out of interest, I wonder what the largest and "most important" thing you've worked on is?

Re: The Lost Art of System Administration

#75

Post by smitty1e »

blacklion wrote: post #57
smitty1e wrote: post #12

Storage management is a real point here. When everything is a blob in a cloud, who really cares about the underlying details?

Owners of the cloud (and, by transition, "your" data)? And you should too, BTW. Or one day you learn, that cloud you choose to use don't have BOTH backups and redundancy, for example. Learn in a bad way. Or, maybe, you'll learn, that you need 10x more performance now (more throughput, less latency), and this cloud can not provide it at any cost.

You're absolutely correct.

I was just moving an EBS volume between regions and doing the partitioning and thinking "wow, you just don't do this much anymore".

Re: The Lost Art of System Administration

#76

Post by tasuki »

amtamt wrote: post #70
tasuki wrote: post #62

Earlier quoted context omitted.

> No, I'm not giving you root on all the servers, tell me what you want to do. You just want to copy a file? You don't need root for that. I'm a developer, so I'm very biased. My interactions with system administrators were more along the lines of "I can't give you access to this system, you have to [come to my desk / hop on a zoom call] and tell me what to type into the shell." and me "Please type tail dash f, no, n…

In a properly designed and deployed systems, no one (including sysadmin) should have a need to run tail/ less/ grep or whatever.... Logs should be in what ever centralized logging system is in use.. Metrics should be in what ever centralized metrics system is in use.. Config and secrets update should be possible from CI/ CD/ build system, without a full compile + test + deploy pipeline. Need of anyone having access t…

You're of course right.

But also, are you shipping all the logs? Of course you're shipping application logs somewhere searchable, what about gc logs, syslog, kernel logs, etc? Is it never necessary to run ad hoc commands on your production systems?

Re: The Lost Art of System Administration

#77

Post by nibbleshifter »

Syonyk wrote: post #26
keyle wrote: post #8

I grew up with spectrums and ms-dos with a turbo and a reset button. I feel sad for my kids who will grow up with iPads and Windows 14 PCs with not the slightest clue on how things are running and the OS shielding them from crashing the thing or running viruses for fun. Same things for young lads that come out of university today with their degrees. They're well spoken and can recite algo theory like nothing but stil…

> Have we reached that level of the civilisation that goes extinct, for forgetting how its own inner machines work? Probably. Almost nobody under the age of 40 works in the "deep weeds" anymore (firmware, kernel, casually "Ring 0 and below" spaces). A decade ago, it was about 30. I'm the last wave, it seems, of people in these spaces, and that's quite terrifying. But I get it. The hardware is maddeningly complex, rap…

> Almost nobody under the age of 40 works in the "deep weeds" anymore (firmware, kernel, casually "Ring 0 and below" spaces)

A lot of the best people I know in that field (low level stuff) are younger than me.

Re: The Lost Art of System Administration

#78

Post by gmuslera »

Working with diverse and complex infrastructures the system that you administer becomes less the individual machine operating system and more the whole interconnected companywide system. You still have to go down one particular server and tune, configure, orchestrate or whatever, but the big vision should be present.

Re: The Lost Art of System Administration

#79

Post by pxc »

Test0129 wrote: post #24
q845712 wrote: post #10

Isn't it knowing kubernetes now? Which does kinda tend to require you to also know a fair bit of "single machine" admin tools along with a whole slew of new complexity.

I have built a lot of stuff on kubernetes. Outside of certain edge cases I've had to use almost none of the actual system administration knowledge I have. The vast majority of things are relatively simple, or can be handled via a nice UI provided by . In fact, I'd argue that kubernetes was really made to remove sysadmin from being needed. For large kubes obviously some skill is needed but in the average case a develo…

> I'd argue that kubernetes was really made to remove sysadmin from being needed. For large kubes obviously some skill is needed but in the average case a developer can do 99.9% of all the work needed. It was a huge boon for businesses that would otherwise have to pay both developers and sysadmins. Now they can pay one of them half the value!

> The majority of other sysadmin tasks have been offloaded to cloud providers

Yes. There is a word for this process: deskilling — https://en.wikipedia.org/wiki/Deskilling

It is typically followed or accompanied by a loss of autonomy and/or other privileges in the workplace

Re: The Lost Art of System Administration

#80

Post by amtamt »

tasuki wrote: post #76
amtamt wrote: post #70

Earlier quoted context omitted.

In a properly designed and deployed systems, no one (including sysadmin) should have a need to run tail/ less/ grep or whatever.... Logs should be in what ever centralized logging system is in use.. Metrics should be in what ever centralized metrics system is in use.. Config and secrets update should be possible from CI/ CD/ build system, without a full compile + test + deploy pipeline. Need of anyone having access t…

You're of course right. But also, are you shipping all the logs ? Of course you're shipping application logs somewhere searchable, what about gc logs, syslog, kernel logs, etc? Is it never necessary to run ad hoc commands on your production systems?

How are gc/ syslog/ kernel logs are special and not sent to central logging? Ad-hoc logs (such as thread dumps etc.) can also be triggered via safe application end points from restricted locations.

> "Is it never necessary to run ad hoc commands on your production systems?"

Ad-hoc commands in production AS ROOT should be exception, not norm, and reserved for on-call members, not every developer working on team.

Even the most competent people, sysadmin or developer, can make a mistake.

I did not say no commands should ever be run on production, there could be non-root read-only accounts. I am opposed to everyone having root on production. I have seen enough data loss incidents or full blown outages because of mistakes.

Post reply on HN