Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reclaiming the lost art of Linux server administration

pietrorea.com

Post reply on HN
Print view View original

41–50 of 485 posts

Re: Reclaiming the lost art of Linux server administration

#41

Post by bblb »

candiddevmike wrote: post #13

Blame the folks demonizing/shaming having "pet" servers and pushing immutable infrastructure. Linux server administration is quite enjoyable, and with how well apps these days can scale vertically, it really takes a special kind of workload to need (and actually saturate) fleets of servers.

I work in IT Operations of a big IT house. 100% local gov customers. We fully manage around 5000 pet servers. ~30 sysadmins and some of us do architecture designing also. There's also a separate networking team of about 10 network specialists.

Senior sysadmins are really hard to come by today, not to mention someone who wants to do architecture also.

My hunch is that the 5000 onprem pet servers are not going away any day soon, because a massive amount of it is legacy systems that take a long time to migrate to cloud, if ever. Also the work stress is just ridiculous. So much stuff to do, even with automation. Only reason I still do this is that I like the "old school" tech stack vs. cloud IaaS/PaaS alternatives.

Re: Reclaiming the lost art of Linux server administration

#42

Post by encryptluks2 »

chasil wrote: post #3

-"As for scripting, commit to getting good at Bash." That advice can cause substantial headache on Ubuntu/Debian, where the Almquist shell is /bin/sh. This does not implement much of bash and will fail spectacularly on the simplest of scripts. This is also an issue on systems using Busybox. A useful approach to scripting is to grasp the POSIX shell first, then facets of bash and Korn as they are needed. -"As a practi…

> -"As a practical goal, you should be able to recreate your host with a single Bash script."

I disagree with this. A single bash script configuring an entire hosts can be overly complex and very difficult to follow. As someone who has created complex bash scripts, this will become very time consuming and prevent you from making many changes without significant efforts. I'd suggest familiarizing yourself with tools like cloud-init and Ansible.

Re: Reclaiming the lost art of Linux server administration

#43

Post by Karrot_Kream »

candiddevmike wrote: post #13

Blame the folks demonizing/shaming having "pet" servers and pushing immutable infrastructure. Linux server administration is quite enjoyable, and with how well apps these days can scale vertically, it really takes a special kind of workload to need (and actually saturate) fleets of servers.

I've never seen anyone demonize or shame having pet servers, if anything people on tech news sites write about their pet servers constantly (understandably, it's fun!) Just like you're not going to make all your furniture by hand when you start a business but instead just buy whatever works from Ikea, likewise you make a more conscious decision to build or buy (as TFA touched on) based on the constraints of the business. And sometimes a business, for compliance reasons for example, may choose to keep their servers in-house, in which case you could potentially be a sysadmin.

Re: Reclaiming the lost art of Linux server administration

#44

Post by rank0 »

warent wrote: post #34

When my SaaS app started scaling, I saw how badly cloud can be priced if you have even slightly unusual use-cases. It occurred to me that instead of spending ~$600/mo on GCP, I can invest in a $3000 PowerEdge server with much better hardware, run it out of my home office, and it pays for itself in less than a year. Running your own server is an investment that doesn't make sense for everyone. If you can get it, it is…

Do you have a static IP?

I have a homelab too but getting “enterprise grade” service from comcast seems to be my biggest barrier to scaling without leaning on aws.

Re: Reclaiming the lost art of Linux server administration

#45

Post by dymax78 »

trabant00 wrote: post #19
VWWHFSfQ wrote: post #12

Regular SA and DBA jobs will be almost completely gone within a decade or so. Same as there are hardly any auto mechanics anymore because nobody can fix any of the new cars but the manufacturer. You'll only find those jobs at one of the handful of cloud companies. Nobody will know how to do anything for themselves anymore and all this experience and knowledge will be lost. There are no more actual administrators. Jus…

I've been hearing this for the past 20 years. And now my sysadmin skills are more and more in demand. For the past 5 years or so I started making more money than a dev because of supply and demand. Rent to AWS actually drives demand up quite a lot since the bills are huge and very few people understand what is under the hood and how it can be optimized. I doubt very much things will change in the near future. In the…

> Rent to AWS actually drives demand up quite a lot since the bills are huge and very few people understand what is under the hood and how it can be optimized.

A few years ago I participated in a Splunk deployment and the cloud solution utterly dwarfed an in-house enterprise solution, in regards to cost. Even in the event that cost was irrelevant, certain sectors (financial institution(s)) are going to have a difficult time pivoting to a cloud-based solution and relinquishing control over the underlying infrastructure.

Re: Reclaiming the lost art of Linux server administration

#46

Post by pmarreck »

warent wrote: post #34

When my SaaS app started scaling, I saw how badly cloud can be priced if you have even slightly unusual use-cases. It occurred to me that instead of spending ~$600/mo on GCP, I can invest in a $3000 PowerEdge server with much better hardware, run it out of my home office, and it pays for itself in less than a year. Running your own server is an investment that doesn't make sense for everyone. If you can get it, it is…

Does it have a backup schedule (and did you prove your restore process works)? Is it replicated to another physically-offsite location? Do you have to manage your own security keys? Load balancing? Multi region availability? How do you admin it remotely? Firewalled? Notifications of emergency situations like low disk space, downages, over-utilization of bandwidth, memory leakage, SMART warnings, etc.? What's your version upgrade strategy? What's your OS upgrade strategy? Failover? IPv6? VPN access? DMZ?

Basically, I think cloud provides a loooooot of details that you have to now take on yourself if you self-host (at least if you want to do it "legitimately and professionally" as a reliable service). It's not clearly a win-win.

That all said, I recently canceled my cloud9 dev account at amazon because the resources I needed were getting too expensive, and am self-hosting my new dev env in a VM and accessing it from anywhere via Tailscale, so that's been nice.

Re: Reclaiming the lost art of Linux server administration

#47

Post by qmarchi »

rank0 wrote: post #44
warent wrote: post #34

When my SaaS app started scaling, I saw how badly cloud can be priced if you have even slightly unusual use-cases. It occurred to me that instead of spending ~$600/mo on GCP, I can invest in a $3000 PowerEdge server with much better hardware, run it out of my home office, and it pays for itself in less than a year. Running your own server is an investment that doesn't make sense for everyone. If you can get it, it is…

Do you have a static IP? I have a homelab too but getting “enterprise grade” service from comcast seems to be my biggest barrier to scaling without leaning on aws.

Hey, you actually have a few options, notably, doing nothing!

Comcast doesn't actually change your public IP address between DHCP renewals and thus it's effectively static. The only time that it'll change is when the modem is powered off for an amount of time, or the upstream DOCSIS concentrator is powered off for maintenance or otherwise.

Re: Reclaiming the lost art of Linux server administration

#48

Post by MattGaiser »

LAC-Tech wrote: post #36

Full disclaimer, I'm very much not a sysadmin or devops guy. However, every team I've been on recently has spent a lot of time struggling with gluing their AWS stuff together, diagnosing bugs etc. It didn't seem to save a heck of a lot of time at all. I couldn't figure out AWS. But I could figure out how to host sites on a linux VPS. So what's the story here - is serverless something that only makes sense at a certai…

A lot of it is lack of awareness of things like Caddy or any other tools that simplify the process.

I did not know about it until I googled it right now. I have spent days/even two weeks figuring out how to set up Nginx and for all I know I did it terribly wrong. I paired it with other tools that I do not even remember. But I would be starting from scratch again if I needed to set another one up.

So a lot might come down to that. I was on a team that transitioned from a owned server to cloud as one day one of the test servers went down and after a week of trying, nobody knew how to fix it. We realized at that point that if a server caused a production error, we were utterly screwed as someone who had left set it up and nobody had a clue where to begin fixing it beyond reading endless tutorials and whatever came up in Google searches.

The server infrastructure was cobbled together in the first place and for a period was theoretically maintained by people who didn't even know the names of all the parts.

At least with cloud, there is an answer of sorts that can be had from the support team.

Re: Reclaiming the lost art of Linux server administration

#49

Post by jhickok »

invokestatic wrote: post #8

I used to reach for shell scripts to configure servers, then Puppet, then Salt, and then finally to Ansible. Configuring servers declaratively is such a massive improvement over shell scripts. The fact that Ansible is agentless is also very nice and works very well for when you only have a handful of servers. Only thing I dislike is YML, which I think is yucky!

Out of curiosity, have you tried tools like Pulumi? I've never used it but as a longtime Ansible user it's something that has my interest.

Re: Reclaiming the lost art of Linux server administration

#50

Post by netr0ute »

cure wrote: post #17
netr0ute wrote: post #9

Earlier quoted context omitted.

Why would anyone want to target bash specifically which doesn't exist in all systems instead of just sticking to what's implemented in /bin/sh?

Because you're not going to have a great time with /bin/sh (i.e. dash or the like) if you want to do anything more than very, very basic scripts.

Relying on bash is a recipe for non-portability.
Post reply on HN