"The network is down, why are we paying the sys admins?"
"The network is running perfectly, why are we paying the sys admins?"
The Lost Art of System Administration
61–70 of 98 posts
Re: The Lost Art of System Administration
#62Most of those things listed are just technical details that change over time. Before all the Linux trivia there was the Solaris trivia, AIX trivia, Novell, HP/UX, Windows NT, OS/2, VMS, AS/400. Alpha, Sparc, VAX, tape robots, IPX/SPX, Token ring, VT100 dumb terminals. Old heads have forgotten more technical crap than exists today. (I can barely remember how to set up a Cisco router, and I used to work there.. I do, h…
> No, I'm not giving you root on all the servers, tell me what you want to do. You just want to copy a file? You don't need root for that.
I'm a developer, so I'm very biased.
My interactions with system administrators were more along the lines of "I can't give you access to this system, you have to [come to my desk / hop on a zoom call] and tell me what to type into the shell." and me "Please type tail dash f, no, not slash, dash, the horizontal line thing" and a whole lot of back and forth like this, after which I thank them politely and they feel very accomplished while I'm left guessing why that entire department is necessary...
Re: The Lost Art of System Administration
#63Isn't it knowing kubernetes now? Which does kinda tend to require you to also know a fair bit of "single machine" admin tools along with a whole slew of new complexity.
I have built a lot of stuff on kubernetes. Outside of certain edge cases I've had to use almost none of the actual system administration knowledge I have. The vast majority of things are relatively simple, or can be handled via a nice UI provided by . In fact, I'd argue that kubernetes was really made to remove sysadmin from being needed. For large kubes obviously some skill is needed but in the average case a develo…
As someone who gravitated to sysadmin side (it was easier to get work than AI, especially if you grew banging together components to get a working desktop since 1990s)...
k8s is a great lever for sysadmin. It's the kind of tool we spoke of in fevered tones when we shared legends of "automating our job so we would only need to pick they paycheck". Does it mean there's less need for absolute Ops-side numbers? Sure. Did we have the numbers before? hahaha, fat chance, nope. So in my experience, moving onto k8s often means my job involves less understaffing now, less panic, and less cursing. Most importantly, less reasons to visit Scary Devil Monastery (alt.sysadmin.recovery).
And if the low amount of work starts to get depressing... you can always take another job[1] or start engineering the shit out of the infrastructure. Bonus points if you make it without making it problem for future you (so documentation and the like) and without raising costs.
[1] A polish joke about sysadmins: "Hey, Joe lost a job, you heard?" "How can he deal with having only 5 full time jobs?"
Re: The Lost Art of System Administration
#64Earlier quoted context omitted.
Another part of the problem is that developers are often discouraged or outright not allowed to work with machines, due to "it's not your job" kind of arguments, or corporate security enforced by auditors.
You don't need corporate security or auditors; even in a small web-shop, administering production servers is a power reserved for wizards. Sure, you can administer your own developer workstation; perhaps you get a Linux VM to yourself, that you can tinker with. It would be nuts to let every developer tinker with the production server, or the source-code repo, or the fileserver. The private VM gives them a playground…
> It would be nuts to let every developer tinker with the production server, or the source-code repo, or the fileserver.
I think that it might be a good idea to have most of the configuration for servers be based on Git repos, with something like Ansible or another such solution (Chef, Puppet, Salt), so that server configuration wouldn't be treated that differently from versioned code (which also serves as a historical record).
Don't give developers access to just push changes for production servers, but absolutely let them create merge/pull requests with changes to be applied once proper review is done: ideally with a description of what these changes accomplish, references to other merge/pull requests for development/test/staging environments where they were tested beforehand (perhaps after prior tests against local VMs) and a link back to whatever issue management system is used.
Then, have an escape hatch that can be used by whoever is actually responsible for keeping the servers up and running, in cases an update goes bad or other manual changes are absolutely necessary, but other than that generally disallow anyone to access the server directly for whatever reason (or at least remove any kinds of write permissions).
Personally, I'd also argue that containers should be used to have a clear separation between infrastructure and business applications, but that's mostly the approach to use when dealing with web development or similar domains. Regardless, I find that it's one of the more sane approaches nowadays, the Ops people can primarily worry about the servers, updates, security, whereas the developers can worry about the applications and updates/security for those.
Re: The Lost Art of System Administration
#65I grew up with spectrums and ms-dos with a turbo and a reset button. I feel sad for my kids who will grow up with iPads and Windows 14 PCs with not the slightest clue on how things are running and the OS shielding them from crashing the thing or running viruses for fun. Same things for young lads that come out of university today with their degrees. They're well spoken and can recite algo theory like nothing but stil…
> Have we reached that level of the civilisation that goes extinct, for forgetting how its own inner machines work?
This thought was expressed in Isaac Asimov's short story "The Last Question" [PDF]: https://physics.princeton.edu/ph115/LQ.pdf
> Alexander Adell and Bertram Lupov were two of the faithful attendants of Multivac. As well as any human beings could, they knew what lay behind the cold, clicking, flashing face -- miles and miles of face -- of that giant computer. They had at least a vague notion of the general plan of relays and circuits that had long since grown past the point where any single human could possibly have a firm grasp of the whole. Multivac was self-adjusting and self-correcting. It had to be, for nothing human could adjust and correct it quickly enough or even adequately enough. So Adell and Lupov attended the monstrous giant only lightly and superficially, yet as well as any men could. They fed it data, adjusted questions to its needs and translated the answers that were issued. Certainly they, and all others like them, were fully entitled to share in the glory that was Multivac's
I wonder whether they'll have to deal with CrashLoopBackoff errors in 2061.
Re: The Lost Art of System Administration
#66Earlier quoted context omitted.
I'm a dude who uses Kubernetes to make pizzas. Kubeternetes is absolutely a commercial bakery class machine, but much of it's adoption is due to the fact that for just a bit more in price and effort, you can have that class of machine in your home and run real things on it. Seriously: I run clusters from a few dozen nodes (down from a few hundred at my peak, sigh) down to a trio of Raspberry Pis in my living room. Th…
I'd say Kubernetes was more like having a lathe in your shed. Almost no-one needs one but it sure does make some projects a lot easier.
Of topic but having a lathe in my shed is a definite life goal.
Re: The Lost Art of System Administration
#67Earlier quoted context omitted.
Wrong comparison, sorry. K8s is enterprise thing, while LAMP is good for SOHO. So cookies factory vs small bakery.
You do realize some of the internet's biggest sites run on the lamp stack, right?
With Apache httpd inside? They don’t have devops at all?
Re: The Lost Art of System Administration
#68Earlier quoted context omitted.
I'm a dude who uses Kubernetes to make pizzas. Kubeternetes is absolutely a commercial bakery class machine, but much of it's adoption is due to the fact that for just a bit more in price and effort, you can have that class of machine in your home and run real things on it. Seriously: I run clusters from a few dozen nodes (down from a few hundred at my peak, sigh) down to a trio of Raspberry Pis in my living room. Th…
I'd say Kubernetes was more like having a lathe in your shed. Almost no-one needs one but it sure does make some projects a lot easier.
I just got one and not sure I need it. The lathe always seemed so dangerous to me.
Re: The Lost Art of System Administration
#69Earlier quoted context omitted.
> Have we reached that level of the civilisation that goes extinct, for forgetting how its own inner machines work? Probably. Almost nobody under the age of 40 works in the "deep weeds" anymore (firmware, kernel, casually "Ring 0 and below" spaces). A decade ago, it was about 30. I'm the last wave, it seems, of people in these spaces, and that's quite terrifying. But I get it. The hardware is maddeningly complex, rap…
I often wonder the same. I grew up at a time, where I had to learn assembly on the C64, to get my machine to handle > 300 baud with uploads/downloads, on a BBS I wrote. On the hardware side, it was very easy to learn how to fix said machines, with schematics provided in the back of the user manual. This level of understanding is not as easily possible today, and my understanding grew outward from that. I think two th…
As someone said, kids this day build computers with arduinos and stm32.
Re: The Lost Art of System Administration
#70Most of those things listed are just technical details that change over time. Before all the Linux trivia there was the Solaris trivia, AIX trivia, Novell, HP/UX, Windows NT, OS/2, VMS, AS/400. Alpha, Sparc, VAX, tape robots, IPX/SPX, Token ring, VT100 dumb terminals. Old heads have forgotten more technical crap than exists today. (I can barely remember how to set up a Cisco router, and I used to work there.. I do, h…
> No, I'm not giving you root on all the servers, tell me what you want to do. You just want to copy a file? You don't need root for that. I'm a developer, so I'm very biased. My interactions with system administrators were more along the lines of "I can't give you access to this system, you have to [come to my desk / hop on a zoom call] and tell me what to type into the shell." and me "Please type tail dash f, no, n…
In a properly designed and deployed systems, no one (including sysadmin) should have a need to run tail/ less/ grep or whatever....
Logs should be in what ever centralized logging system is in use..
Metrics should be in what ever centralized metrics system is in use..
Config and secrets update should be possible from CI/ CD/ build system, without a full compile + test + deploy pipeline.
Need of anyone having access to production to be able to run tail (or anything of that nature) is a sign that system is not well designed, and compensated by... "Give me root access everywhere".
Let's not forget user privilege separation came in for some serious reasons.