Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Lost Art of System Administration

matt-rickard.com

Post reply on HN
Print view View original

81–90 of 98 posts

Re: The Lost Art of System Administration

#81

Post by sgarland »

amtamt wrote: post #70
tasuki wrote: post #62

Earlier quoted context omitted.

> No, I'm not giving you root on all the servers, tell me what you want to do. You just want to copy a file? You don't need root for that. I'm a developer, so I'm very biased. My interactions with system administrators were more along the lines of "I can't give you access to this system, you have to [come to my desk / hop on a zoom call] and tell me what to type into the shell." and me "Please type tail dash f, no, n…

In a properly designed and deployed systems, no one (including sysadmin) should have a need to run tail/ less/ grep or whatever.... Logs should be in what ever centralized logging system is in use.. Metrics should be in what ever centralized metrics system is in use.. Config and secrets update should be possible from CI/ CD/ build system, without a full compile + test + deploy pipeline. Need of anyone having access t…

As the other child comment said, are you shipping _all_ logs? For a K8s cluster of a few nodes, sure, you might send `/var/log/*` out, but that doesn't scale. If you have weird Kubernetes problems like mysteriously dying containers that don't trip 137 (hello, non-init processes [0]), you will need to dive into kernel messages. For this particular example, there is to my knowledge one poorly-maintained package [1] that fixes this, but it also requires at the very least some Linux capabilities that might raise eyebrows with security, and as written it requires privileged containers. Thankfully this particular example will hopefully be fixed with cgroupsv2, but still.

SaaS DB like RDS is another example. I've seen mysterious growth on a read replica, which was fixed by restarting it. Temporary intrinsic tables were not the cause, if you were wondering. No noted errors were logged in its error log it shipped out, and of course the general log wasn't enabled because it generally gets absurdly large. Had this been a self-hosted installation, there may well have been clues that would have been available only by logging into the host.

"Cattle, not pets" is an admirable goal, but sometimes you have to play veterinarian on a few of your herd to figure out what illness is spreading, and how to inoculate against it.

[0] https://github.com/kubernetes/kubernetes/issues/50632

[1] https://github.com/transferwise/oomie

Re: The Lost Art of System Administration

#82

Post by sgarland »

b112 wrote: post #44
glintik wrote: post #33

Earlier quoted context omitted.

Wrong comparison, sorry. K8s is enterprise thing, while LAMP is good for SOHO. So cookies factory vs small bakery.

This is a poor analogy overall, but I think to would be better to think of it thusly... K8 is where you don't own the kitchen, and you lease it when you wish to cook. You aren't aware of how to maintain the kitchen, or buy the ingredients you choose to cook with. In fact, you aren't even able to tell if a mango is ripe or not when shopping, because you don't shop and don't know how to. That's what K8s are. Meanwhile,…

A couple of examples I've found:

Celery spawns n workers, defaulting to the number of logical processors available. As anyone familiar with cgroups can tell you, this is fraught with problems when containerized, since nearly all mechanisms to detect processor count (or available memory) lead back to `/proc`, which will dutifully report the host's information out to the container. This leads to questions like, "I requested 4 vCPUs; why do I have 4+n threads?"

ORM in general. The worst example I've seen was querying a massive table, and to get n results from the end, was using OFFSET + LIMIT. It was also deliberately not using the PKEY, leading to a full table scan every time the query ran. If you aren't familiar with DBs, it may seem perfectly reasonable that querying a ~100 million row DB would take a long time, when in fact it could and should be extremely fast with a properly written query.

Re: The Lost Art of System Administration

#83

Post by sgarland »

Test0129 wrote: post #24
q845712 wrote: post #10

Isn't it knowing kubernetes now? Which does kinda tend to require you to also know a fair bit of "single machine" admin tools along with a whole slew of new complexity.

I have built a lot of stuff on kubernetes. Outside of certain edge cases I've had to use almost none of the actual system administration knowledge I have. The vast majority of things are relatively simple, or can be handled via a nice UI provided by . In fact, I'd argue that kubernetes was really made to remove sysadmin from being needed. For large kubes obviously some skill is needed but in the average case a develo…

> a developer can do 99.9% of all the work needed

I've yet to meet a dev who knows how or wants to create and maintain the IaC to manage the infra.

> Terraform really brought everything together in an absolutely brainless mechanism

Understanding Terraform's limitations and being able to work around it in a way that is both readable and scalable is most definitely not brainless. Sure, I can write some .tf files to spin up an EKS cluster pretty easily. Load balancer? I can also do that in Terraform, or I can let the cluster manage its own. Uh-oh, a fork. Which one is better? What if I want to have multiple AWS accounts; how do I properly delegate IAM permissions across them? Etc.

Re: The Lost Art of System Administration

#84

Post by sgarland »

hsn915 wrote: post #25

The fact that linux systems need adminstration is one of the reasons why they suck. Imagine after buying a macbook that you also need to hire a sysadmin to keep it running smoothly? The idea is laughable. > Learning how to effectively use the command line. Unix philosophy. Pipes, scripting, and tooling. So much of programming is stitching things together. This is really bad. Much of the complexity in contemporary sof…

> "Do things by stitching programs together" only works for relatively simple tasks. For everything else you need to program the thing properly.

What do you think functions are?

> I'm really grateful we have things like Go and SQLite that make it possible to create fully functional websites as self-contained programs instead of the hot mess that everything else uses.

Yes, because as someone who will be called to troubleshoot it, I definitely want to trudge through whatever logging you may or may not have implemented in this monolith to solve a cache miss or 5xx error. /s

Re: The Lost Art of System Administration

#85

Post by amtamt »

sgarland wrote: post #81
amtamt wrote: post #70

Earlier quoted context omitted.

In a properly designed and deployed systems, no one (including sysadmin) should have a need to run tail/ less/ grep or whatever.... Logs should be in what ever centralized logging system is in use.. Metrics should be in what ever centralized metrics system is in use.. Config and secrets update should be possible from CI/ CD/ build system, without a full compile + test + deploy pipeline. Need of anyone having access t…

As the other child comment said, are you shipping _all_ logs? For a K8s cluster of a few nodes, sure, you might send `/var/log/*` out, but that doesn't scale. If you have weird Kubernetes problems like mysteriously dying containers that don't trip 137 (hello, non-init processes [0]), you will need to dive into kernel messages. For this particular example, there is to my knowledge one poorly-maintained package [1] tha…

Seems "not well deployed system" or "system not well understood by developer", what stops one reducing retention to few hours for node kernel logs or having read only access to certain parts, if at all there are so many kernel/ sys logs. I have never come across a system where application logs were not multiple orders of magnitude more in volume than syslogs. Not sure what RDS vs in-house DB example was meant to convey.

Ironically, these examples are only proving the point made in first comment.

Re: The Lost Art of System Administration

#86

Post by surajrmal »

Syonyk wrote: post #26
keyle wrote: post #8

I grew up with spectrums and ms-dos with a turbo and a reset button. I feel sad for my kids who will grow up with iPads and Windows 14 PCs with not the slightest clue on how things are running and the OS shielding them from crashing the thing or running viruses for fun. Same things for young lads that come out of university today with their degrees. They're well spoken and can recite algo theory like nothing but stil…

> Have we reached that level of the civilisation that goes extinct, for forgetting how its own inner machines work? Probably. Almost nobody under the age of 40 works in the "deep weeds" anymore (firmware, kernel, casually "Ring 0 and below" spaces). A decade ago, it was about 30. I'm the last wave, it seems, of people in these spaces, and that's quite terrifying. But I get it. The hardware is maddeningly complex, rap…

Do we work in the same field? I'm 32 and am the oldest person on my team of 10. We work on drivers and other low level software. Adjacent teams have similar mixes. At previous roles I saw similarly young demographics in my coworkers. I might agree with the premise that the overall number of engineers down in the weeds has increased at a snails pace compared to the rest of the industry which may give the impression folks are older by comparison.

Re: The Lost Art of System Administration

#87

Post by Test0129 »

sgarland wrote: post #83
Test0129 wrote: post #24

Earlier quoted context omitted.

I have built a lot of stuff on kubernetes. Outside of certain edge cases I've had to use almost none of the actual system administration knowledge I have. The vast majority of things are relatively simple, or can be handled via a nice UI provided by . In fact, I'd argue that kubernetes was really made to remove sysadmin from being needed. For large kubes obviously some skill is needed but in the average case a develo…

> a developer can do 99.9% of all the work needed I've yet to meet a dev who knows how or wants to create and maintain the IaC to manage the infra. > Terraform really brought everything together in an absolutely brainless mechanism Understanding Terraform's limitations and being able to work around it in a way that is both readable and scalable is most definitely not brainless. Sure, I can write some .tf files to spi…

In my experience once terraform moves beyond "brainless script" it becomes a massive footgun. Completely unexpected, unintuitive behavior can happen that destroys an entire cluster. You're often better at that point legitimately paying for an SRE team to manage things but "move fast and break things" tends to prefer detonating a cluster over paying professionals.

Re: The Lost Art of System Administration

#88

Post by Maursault »

throwaway892238 wrote: post #35

Most of those things listed are just technical details that change over time. Before all the Linux trivia there was the Solaris trivia, AIX trivia, Novell, HP/UX, Windows NT, OS/2, VMS, AS/400. Alpha, Sparc, VAX, tape robots, IPX/SPX, Token ring, VT100 dumb terminals. Old heads have forgotten more technical crap than exists today. (I can barely remember how to set up a Cisco router, and I used to work there.. I do, h…

I like the list of OS you give, but System Administration is not the configuration of a system but instead thousands, with a mind towards performance and allocation of resources, but also includes stressing systems to see what will break. What is being administered is actually people, not machines, and perhaps ironically, relationships become of fundamental importance. Generally, sysadmins are not software developers; they are technology generalists and experts at any operating system and every application, even those they have never used.

Re: The Lost Art of System Administration

#89

Post by throwaway892238 »

tasuki wrote: post #62
throwaway892238 wrote: post #35

Most of those things listed are just technical details that change over time. Before all the Linux trivia there was the Solaris trivia, AIX trivia, Novell, HP/UX, Windows NT, OS/2, VMS, AS/400. Alpha, Sparc, VAX, tape robots, IPX/SPX, Token ring, VT100 dumb terminals. Old heads have forgotten more technical crap than exists today. (I can barely remember how to set up a Cisco router, and I used to work there.. I do, h…

> No, I'm not giving you root on all the servers, tell me what you want to do. You just want to copy a file? You don't need root for that. I'm a developer, so I'm very biased. My interactions with system administrators were more along the lines of "I can't give you access to this system, you have to [come to my desk / hop on a zoom call] and tell me what to type into the shell." and me "Please type tail dash f, no, n…

Admittedly, that experience sucks. But there's a couple reasons they did that:

1. They could give you root access. But then you might start making other changes, and over time that makes everything unmaintainable. By them just typing in what you want this one time, they limit the scope of your access to just one change, they are aware of what changed, and can push back on it. It's basically a crappy change management process.

2. This person has a lot on their plate (a way larger backlog than the average dev) and doesn't have time to research how to do the thing you want them to do. They are asked to be experts at everything, but they can't be. So they ask you to show them exactly what to do. Gets rid of ambiguity, lets you both troubleshoot any unexpected problems in real time.

3. A lot of roles where one person/group has "power" over another, often takes it for granted and unnecessarily gate-keeps and adds red tape and blockers. DevOps is supposed to flip this situation on its head by giving you, the dev, the tools needed to do your job, without you making changes to the system that would affect system stability. This keeps the system stable while giving you agency. Unfortunately, doing all that is much more difficult than most people realize, so it rarely happens.

4. Like every role, sometimes there's idiots. I've met my fair share of both idiot devs and idiot sysadmins.

Re: The Lost Art of System Administration

#90

Post by mbrodersen »

throwaway892238 wrote: post #35

Most of those things listed are just technical details that change over time. Before all the Linux trivia there was the Solaris trivia, AIX trivia, Novell, HP/UX, Windows NT, OS/2, VMS, AS/400. Alpha, Sparc, VAX, tape robots, IPX/SPX, Token ring, VT100 dumb terminals. Old heads have forgotten more technical crap than exists today. (I can barely remember how to set up a Cisco router, and I used to work there.. I do, h…

I am a developer. Give me a VM I have root access to and walk away. That’s all I need from you.
Post reply on HN